Your Agent Aced the Task. Will It Do It Again?

Your Agent Aced the Task. Will It Do It Again?

The first run passes. The second run passes. The third run quietly refunds the wrong order, and nobody notices until a customer does. That gap — between an agent that can complete a task and an agent that reliably completes it — is where most production agent work actually lives.

This article is about measuring that gap, and then closing it. It is a practical engineering guide: a small harness, a handful of metrics, and a set of experiments you can run this afternoon to find out whether your agent's success was a capability or a coincidence.

The starting point is a question posed by IBM Research on the Hugging Face blog: Your Agent Aced the Task. Will It Do It Again? (source). That post frames the problem of agent consistency. Everything below — the harness, the commands, the metric definitions — is standard reliability engineering applied to agents; it is not a summary of that post's methods, and you should read the source directly for its own framing.

The demo trap

Agent demos are optimized for a single successful trajectory. You pick a task, you run it, it works, you ship it. The demo is a sample of size one, and sample size one has no error bars.

Three properties make this worse for agents than for most software:

Agents are stochastic by default. Unless you are running a local model with a fixed seed on fixed hardware, the same prompt can produce different tool calls on different runs. Sampling, batching, and provider-side changes all nudge the distribution.

Agents depend on the world. A tool call hits an API that returns different data, a search index that has been reindexed, a database whose rows have changed. Two runs of the "same" task may not be the same task at all.

Agents fail silently. A pipeline that crashes is easy to debug. An agent that returns a plausible-sounding wrong answer looks identical to success at the logging layer.

None of this means agents are unusable. It means the unit of evidence is not one run. It is k runs, and the metric that matters is not "did it work" but "how often does it work, and does it work the same way."

What "again" actually means

Before installing anything, be precise about the claim you want to test. There are at least three distinct properties people lump together under "reliable":

• Availability — the agent completes without an unhandled exception.
• Correctness — the output satisfies an automated checker.
• Consistency — repeated executions of the same task produce equivalent outcomes.

These fail independently. An agent can be perfectly consistent and consistently wrong. It can be correct on average and unusable in practice because 20% of runs fail. It can be available 100% of the time while producing different answers each run.

The metric that captures the production-relevant question is what I will call pass^k: run the same task k times, and count the task as passing only if all k attempts pass. This is deliberately harsher than pass@1 (the average success rate), because in most agent deployments a task is not "done" if it works three times out of five. Pass^k penalizes flakiness directly, and it degrades fast: an agent with a 90% per-run success rate passes a 10-run check only about 35% of the time.

That arithmetic is the whole argument for this article. A 90% agent sounds good and behaves badly.

Requirements

You need a working agent you can call as a function, and a checker that can decide whether an output is correct. Everything else is standard tooling.

• Python 3.10 or newer.
• pip (or uv, if you prefer faster installs).
• A task set: ideally 10–50 representative tasks with known-good outcomes.
• A programmatic checker per task. String matching, schema validation, unit tests on the output, or a small assertion script. If your only checker is a human reading the output, start there — but automate the easy cases first.
• Optional: Docker, if you want the run environment to be identical across machines.
• Git, so every run is tied to a commit.

The checker is the hard part. Budget most of your effort there, not in the harness.

Step-by-step installation

Create an isolated environment so your harness dependencies do not collide with your agent's.

python -m venv .venv && source .venv/bin/activate

On Windows, activate with .venv\Scripts\activate instead. If you prefer uv, the equivalent is:

uv venv && source .venv/bin/activate

Install the test and analysis tooling. pytest runs the checks, pytest-repeat re-runs a single test N times, and pandas/numpy handle the aggregation.

pip install "pytest>=8" pytest-repeat pandas numpy

Freeze the exact environment so a future run can be reproduced. This file is an artifact you should commit.

pip freeze > requirements.lock

Record the code version alongside every result set. Without this, a change in consistency is unattributable.

git rev-parse HEAD > .run-commit && cat .run-commit

If you want the runtime isolated as well, build an image once and run all attempts inside it.

docker build -t agent-under-test:1.0 .

Finally, pin the hash seed. Python's hash randomization changes dictionary iteration order in some code paths, which is a genuine source of run-to-run variation in tool routing.

export PYTHONHASHSEED=0

That is the entire toolchain. No agent framework required.

Building the repeatability harness

The harness has one job: call the agent k times per task, record everything, and never throw away a failure. Put this in consistency/harness.py.

# consistency/harness.py
from __future__ import annotations

import json
import time
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Callable


@dataclass
class RunRecord:
    task_id: str
    attempt: int
    success: bool
    latency_s: float
    output: str
    error: str | None = None


def run_attempts(
    agent: Callable[[str], str],
    task_id: str,
    task_input: str,
    checker: Callable[[str], bool],
    attempts: int = 10,
    out_path: Path = Path("runs.jsonl"),
) -> list[RunRecord]:
    """Execute one task `attempts` times and append every result to disk."""
    records: list[RunRecord] = []
    for i in range(attempts):
        t0 = time.perf_counter()
        output, error = "", None
        try:
            output = agent(task_input)
        except Exception as exc:  # record, never swallow silently
            error = f"{type(exc).__name__}: {exc}"
        latency = time.perf_counter() - t0
        records.append(
            RunRecord(
                task_id=task_id,
                attempt=i,
                success=bool(error is None and checker(output)),
                latency_s=round(latency, 3),
                output=output,
                error=error,
            )
        )

    with out_path.open("a", encoding="utf-8") as fh:
        for rec in records:
            fh.write(json.dumps(asdict(rec)) + "\n")
    return records

Three design choices matter here. Outputs are stored in full, because you cannot debug a flake you did not capture. Failures are recorded rather than raised, because a crash on attempt 3 should not hide the data from attempts 1 and 2. Results are appended to JSONL, so a crashed process loses only the current task.

Wire it into a test that asserts on the distribution, not a single run.

# tests/test_consistency.py
from consistency.harness import run_attempts


def test_refund_agent_is_stable(agent, refund_checker):
    records = run_attempts(
        agent,
        task_id="refund-order-42",
        task_input="Refund order 42 in full.",
        checker=refund_checker,
        attempts=10,
    )
    failures = [r for r in records if not r.success]
    assert not failures, (
        f"{len(failures)}/10 attempts failed. "
        f"First error: {failures[0].error or failures[0].output[:200]}"
    )

This test passes only when the agent passes 10 out of 10. On a real agent, it will fail the first time you run it, and that failure is the useful output.

Usage examples

Example 1: Gate a prompt change

You rewrote the system prompt. Does it help? Compute pass^k before and after, on the same task set.

import json
import pandas as pd

rows = [json.loads(line) for line in open("runs.jsonl", encoding="utf-8")]
df = pd.DataFrame(rows)

summary = (
    df.groupby("task_id")["success"]
      .agg(attempts="size", passes="sum")
      .assign(pass_at_1=lambda d: d["passes"] / d["attempts"])
      .assign(stable=lambda d: d["passes"] == d["attempts"])
)
print(summary)
print("pass^k (all attempts passed):", summary["stable"].mean().round(3))

Run the block once against the old prompt's runs.jsonl and once against the new one. If pass@1 improves from 0.82 to 0.85 but pass^k drops from 0.60 to 0.45, the new prompt is buying average performance with variance — usually the wrong trade.

Example 2: Canary a dependency or model upgrade

Run the harness against the candidate and the incumbent in the same session, then compare per-task rather than in aggregate. Aggregate numbers hide offsetting changes: five tasks fixed, five tasks broken, indistinguishable average.

pivot = (
    df.pivot_table(index="task_id", columns="variant",
                   values="success", aggfunc="mean")
)
pivot["delta"] = pivot["candidate"] - pivot["baseline"]
print(pivot.sort_values("delta").head(10))  # regressions first

Sorting ascending surfaces regressions first, which is what you want to look at.

Example 3: Reproduce a flaky failure

When a task fails 3 times in 10, the failure output is in runs.jsonl but the cause is usually in the trace. Wrap your tool calls in a record/replay layer so a failed attempt can be re-executed without hitting the live world.

import hashlib
import json
from pathlib import Path

CASSETTE = Path("cassettes/tools.json")


def tool_key(name: str, args: dict) -> str:
    payload = json.dumps({"name": name, "args": args}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]


def load_cassette() -> dict:
    return json.loads(CASSETTE.read_text()) if CASSETTE.exists() else {}

In replay mode, look up tool_key(name, args) in the cassette and return the recorded response. In record mode, call the real tool and write the response back. This turns an unreproducible symptom into a deterministic unit test, and it is the single highest-leverage step in the whole workflow.

Where the variance comes from

Once failures are reproducible, attribute them. In practice, agent inconsistency clusters in five places.

Sampling. Setting temperature to zero reduces sampling variance but does not guarantee identical outputs across runs; provider-side batching, hardware, and version changes can still move results.

Time and environment. Prompts that embed the current date, the user's locale, or a session ID will differ across runs by construction. Freeze these explicitly.

Tool nondeterminism. Live APIs return different data. Record/replay removes this as a confound — it does not fix it in production, but it tells you whether your flakiness is yours or the world's.

Retrieval drift. If the agent queries an index that is rebuilt between runs, the retrieved context changes. Snapshot the index for a benchmark run.

Control flow. Multi-step agents have many decision points. Small per-step error rates compound: ten steps at 98% per-step reliability gives roughly 82% end-to-end. Measure per-step, not just end-to-end, or you will not know where to spend the effort.

Reading the numbers honestly

Small k produces noisy estimates, and it is easy to over-interpret them. Use an interval.

import math


def wilson(passes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% Wilson score interval for a binomial proportion."""
    if n == 0:
        return (0.0, 1.0)
    p = passes / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = (z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))) / denom
    return (round(max(0.0, centre - half), 3), round(min(1.0, centre + half), 3))


print(wilson(8, 10))  # e.g. (0.49, 0.94)

A "80% success rate" measured over 10 runs is compatible with a true rate anywhere from roughly 50% to 95%. That is not enough evidence to gate a release on. Ten attempts per task across 20 tasks — 200 runs — gives a much tighter picture, at 20× the cost. Choose the sample size deliberately, and state it whenever you report a rate.

Two other reporting habits:

Separate verified facts from interpretation. "The agent failed 4 of 50 runs on task X, with error Y" is a fact. "The retriever is the bottleneck" is an interpretation until you test it.

Report the worst task, not only the mean. A pass@1 of 0.95 with one task at 0.30 is a different system from a uniform 0.95. The mean hides the tail that users will find.

A practical checklist

• Every task has an automated checker, even if crude.
• Every run is logged in full, including the failing output.
• Every result set is tagged with a commit hash and a locked dependency file.
• Pass^k is reported alongside pass@1, never instead of it.
• Tool calls are recorded and replayable.
• Time, locale, and session identifiers are frozen or injected.
• Regressions are reviewed per task, sorted by delta, before any aggregate.

Conclusion

"It worked" is a hypothesis, not a result. The question the IBM Research post raises — will it do it again? — is the right one to ask of any agent before it touches production, and answering it does not require a new framework. It requires a loop, a checker, a log, and a willingness to report the pass^k number rather than the best run.

Run your best task ten times this week. If it passes all ten, run it twenty. If it does not, you have found the work — and you have found it in a test suite instead of in a customer's account.