Agent Accuracy Is Not Enough: Measuring and Improving Consistency

Agent Accuracy Is Not Enough: Measuring and Improving Consistency

An AI agent may complete a task successfully during testing, then take a different path and fail when asked to perform the same task again. In production, that variability is a reliability problem. It can be especially serious in workflows such as reconciling a financial transaction or checking a contract for an obligation.

Many benchmarks conceal this behavior behind an average score. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. However, it succeeded in all five runs for only 53.0% of tasks, producing a 24.4 percentage-point consistency gap.

Most evaluations report the first figure. A more useful reliability measure is the second, which asks whether the agent can repeat the same result consistently.

IBM Research's earlier ALTK-Evolve system turns an agent's previous trajectories into reusable guidelines. These guidelines are distilled automatically and added during inference. The system improves task success, but its initial evaluations focused on average performance. A new guideline type, called consistency guidelines, addresses repeatability directly. It is built on a diagnostic tool called the Consistency Analyzer, included in the altk-evolve toolkit.

Key findings

• Average accuracy can hide unreliability. A ReAct agent using GPT-4.1 on AppWorld test_normal succeeded on 77.4% of runs on average, but passed all five repeated runs for only 53.0% of tasks. The gap reached 30 percentage points on hard tasks.
• The Consistency Analyzer identifies unstable decisions. It resamples the agent's recorded trajectory to find decision points where a different token choice could change the outcome. It requires one trace, no ground truth, and no end-to-end task replay. By default, it requests five completions at each decision point.
• Consistency guidelines reduce the gap. The gap fell from 24.4 percentage points to 12.0 percentage points. Pass⁵ improved by 16.0 percentage points on the same tasks and by 13.0 percentage points on related tasks, without reducing average accuracy.
• The complete methodology and evaluation results are available in the technical report on arXiv.

The metric most evaluations omit

Standard agent evaluations commonly report Mean@k: run a benchmark k times and average the pass rate. The value of k may be three or one, and the resulting figure is often summarized as an agent's accuracy.

Mean@k answers the question, “How well does the agent perform on average?” It does not show whether the agent will succeed when a user repeats the exact same request. For that, Pass^k measures the fraction of tasks that succeed on all k runs.

Pass^k is different from Pass@k. Pass@k asks whether at least one of k attempts succeeds, which is useful when a result can be verified and retried. Pass^k requires every attempt to succeed. In general, Pass^k ≤ Mean@k ≤ Pass@k.

A GPT-4.1 ReAct agent achieved a Mean@5 score of 77.4% on AppWorld test_normal, but its Pass⁵ score was only 53.0%. This means that a substantial portion of the benchmark contained tasks the agent could solve sometimes but not consistently, even though the task itself did not change.

The difference between Mean@k and Pass^k is called the consistency gap. It represents a dimension separate from capability: an agent can be capable of solving a task while still behaving inconsistently.

Why agent decisions change

Each agent decision, such as selecting an API, choosing an argument, or deciding whether to retry, is generated from a probability distribution over possible next tokens.

A sharp distribution concentrates most of its probability on one token. The alternatives are far behind, so the same decision is likely to appear across runs. A flat distribution assigns similar probabilities to several alternatives. Small changes can then determine which option is selected.

Those small changes can arise from GPU floating-point behavior, request batching, or other platform effects. They may not affect a sharp distribution enough to change the winner, but they can reorder nearly tied alternatives in a flat distribution. Since an agent trajectory contains many decisions, the probability that at least one decision changes increases across the entire task.

The issue can remain even with deterministic-looking decoding settings. Greedy decoding and a fixed seed control how the model converts its probability distribution into a token, but they do not change the distribution itself. On a hosted endpoint, slight shifts in probabilities can cause a near-tie to resolve differently from one run to the next, even at temperature zero.

In the reported experiments, the ReAct agent used temperature 0.0, so the observed variation was not ordinary sampling variance.

Diagnose first, then generate guidelines

The approach uses a two-stage pipeline connected to ALTK-Evolve's existing storage and retrieval system.

1. Detect unstable decisions

The Consistency Analyzer receives one recorded trajectory and examines each decision step through controlled resampling. It measures how much the model's output varies at each point.

The process adds one model call per decision step and runs offline. Each call requests k completions at once, with k=5 by default, using the context already recorded in the trajectory. It does not make new tool calls, interact with the environment again, or rerun the task end to end.

The analyzer records a consistency score for each decision step in a scorecard. This identifies decisions that may flip during a subsequent run. The method is black-box: it requires no logits, model internals, or additional instrumentation beyond the existing trace.

2. Generate targeted guidelines

Each flagged decision becomes a candidate consistency guideline in the standard ALTK-Evolve format. For example, GPT-4.1 generated the following guidelines from a trajectory for the AppWorld task, “How many activities are done in my bucket list as per my SimpleNote note?”

Guideline 1: When counting checkbox-style markers in note content, use a line-anchored regular expression rather than a plain substring count, because note titles may repeat the marker symbol in a legend line.

Guideline 2: Verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding.

These guidelines address recurring decision patterns rather than isolated task details. Counting strings incorrectly and failing to verify search results can occur across many AppWorld tasks. The analyzer therefore targets instability, including decisions the agent got right in the recorded run but might get wrong in a later run.

Results on AppWorld

The evaluation used 168 tasks from AppWorld test_normal. A ReAct agent running on GPT-4.1 generated consistency guidelines from one baseline trajectory per task, then tested those guidelines on five fresh runs.

The aggregate results were:

• Pass⁵: 53.0% to 69.0%
• Mean@5: 77.4% to 81.0%
• Consistency gap: 24.4 percentage points to 12.0 percentage points

The gap was reduced by roughly half. Nearly one-third of tasks that had previously been inconsistent became tasks the agent passed on every run.

Results by difficulty

The medium and hard tiers showed the largest gains:

• Medium: +22.9 percentage points, or +44% relative
• Hard: +14.3 percentage points, or +45% relative
• Easy: +12.2 percentage points

The results indicate that the guidelines stabilized decision points where the agent's uncertainty affected the final outcome.

Mean@5 did not decline. Preserving average accuracy was a requirement, since increasing Pass⁵ by lowering Mean@5 would move unreliability rather than resolve it. Mean accuracy held or improved at every difficulty level.

Generalization to related tasks

When applied to a different but related task in the same AppWorld scenario, the consistency guidelines increased Pass⁵ by 13.0 percentage points. This was only three points below the same-task improvement, indicating that the guidelines were not limited to the exact trajectory from which they were generated.

The results also included the weaker gpt-oss-120b model. Same-task Pass⁵ increased by 6.0 percentage points, from 10.1% to 16.1%. On similar tasks, the gain was 8.7 percentage points, exceeding the same-task improvement. This suggests that the guidelines captured reusable failure patterns rather than memorizing details from one trajectory.

Implications for agent evaluation

Teams deploying agents can use several practices to assess repeatability:

• Report Pass^k alongside Mean@k. Average scores cannot distinguish a reliable agent from one that succeeds inconsistently. Even k=3 can expose a meaningful gap.
• Expect the gap to increase with task difficulty. A single average is often least informative for the hardest tasks.
• Do not assume a larger model will solve consistency problems. Consistency is separate from capability. A stronger model may improve Mean@k without reducing the consistency gap.
• Use offline diagnosis when live replay is impractical. The analyzer requires one additional LLM call per decision step, with five completions by default. It needs neither a grader nor a second interaction with the environment.

Understanding the metrics

• Mean@k: Run a task k times and report the average pass rate.
• Pass^k: The fraction of tasks that succeed on all k independent runs. It is always less than or equal to Mean@k.
• Pass@k: The fraction of tasks for which at least one of k runs succeeds.
• Consistency gap: Mean@k minus Pass^k, measured in percentage points.

References

• ALTK-Evolve repository: github.com/AgentToolkit/altk-evolve
• Technical report: arXiv:2609.08832
• Consistency guideline demonstration: 2-minute video