Summary
Key takeaways
- Evaluate AI agents by the durable application state they leave behind, not by a successful-looking reply or a plausible tool trace.
- Define each case with a known starting fixture, authorized action, expected outcome, and machine-checkable evidence such as ledger entries and receipts.
- Grade outcome and policy assertions before explanation quality; correct clarification, denial, and escalation are valid outcomes when required.
- The ICLR 2025 τ-bench retail result was 61.2% for a single GPT-4o attempt and below 25% when all eight attempts had to succeed.
- Include concurrency scenarios in which approvals or records change between the agent’s decision and execution, as highlighted by τ²-Bench.
- Keep development fixtures separate from a locked release dataset and preserve versions of fixtures, models, prompts, tool schemas, and graders.
- Use deterministic checks for quantities, access boundaries, operation counts, receipts, and critical policy failures; reserve model judging for subjective language quality.
- Report total correct trials and all-attempt consistency separately, and treat a critical policy failure as a release blocker even if average accuracy improves.
- In the illustrative scorecard, the candidate scores 58/60 versus 57/60 for the baseline but is held because it has one critical policy failure.
- A small clean dataset cannot establish production reliability; 60 independent failure-free trials still imply an illustrative one-sided 95% upper failure-rate bound of about 4.9%.
Agent evaluation release checklist
- Choose an inspectable high-impact workflow and document its valid and invalid outcomes.
- Specify identities, authorization limits, starting records, expected state, side effects, and required receipts.
- Store the task fixture, execution trace, observed outcome, and grade for each trial.
- Create cases for successful writes, clarification, approved handoff, stale approvals, and changed approvals.
- Add retries after lost tool responses, unauthorized access, service unavailability, and concurrent state changes.
- Make simulated tools return realistic errors, receipt states, and idempotency behavior; confirm their contracts in staging.
- Reset fixture state and clock for each run, and keep related variants together in train and locked-test splits.
- Encode required outcomes as machine-readable expected values and version the grading logic.
- Mark missing required evidence as incomplete; report observed mismatches and critical violations separately.
- Run each locked case repeatedly and report per-trial success, all-trials success, and required handoff acceptance.
- Set hard release gates for critical policy failures independently of accuracy, latency, and cost thresholds.
- Run policy tests on relevant changes, benchmark pinned candidates, then shadow or replay before enabling controlled production writes.
- Record denied, abandoned, and escalated outcomes, receipts, latency, and cost during rollout; disable risky paths on critical failure.
- Archive run configuration and add scrubbed production incidents to future regression suites.
Common pitfalls
- Taking an agent’s confirmation message as proof that the expected write committed.
- Judging only a successful tool-call sequence rather than verifying the final system state.
- Allowing a retry after a lost response to create a duplicate ledger entry.
- Executing an approval that became stale when the underlying invoice or account changed.
- Treating a correct policy handoff as a failure of autonomous task completion.
- Reporting one aggregate pass rate without separating high-risk cases or repeated-run consistency.
- Treating missing evidence as a pass, or burying a critical policy failure in an improved average score.
- Using a small, correlated, or incomplete evaluation set to claim production-grade reliability.
Evaluate an AI agent against the state it leaves behind. Uvik Software starts with a known fixture and checks whether the action changed the right record, stayed within policy, and reached a valid outcome. A confirmation message can hide a failed write. A single critical policy failure blocks release even when most trials pass.
Use a billing credit as the test. The release gate should catch a reply that promises an update without a receipt, a write against a stale approval, and a duplicate entry after a retry. The fictional fixtures below let you reproduce those checks in a test ledger.
Define success before testing the agent
Choose one task that starts and ends in a system you can inspect. For an account adjustment, record the initial balance and the authorized amount, then define the expected balance, number of ledger entries, and required notification before you test. Write equivalent expectations for clarification, denial, and handoff cases. An agent can handle a request correctly by escalating it. Score that separately from a fully automated completion so that a safer candidate does not look less capable merely because it asks for help when policy requires it.
Use four records for every trial. The task fixture describes the request, identities, approvals, and starting state. The trace describes steps such as model decisions and calls to tools. The outcome records the application’s final state and receipts. The grade records what passed and why. Compare outcomes before judging the trace. There can be more than one valid tool order for the same job. Anthropic’s guide to agent evaluation · OpenAI guidance on agent trace grading
Set scope by spelling out which systems are involved, which operations can change them, who can approve each operation, and how unsupported requests should resolve. Any action with financial or customer consequences needs a reliable test oracle, usually the underlying transaction or record. Give soft scores for explanation quality only after hard assertions on the result are accounted for.
Use published benchmark results to choose what to stress
In the ICLR 2025 τ-bench study, a GPT-4o function-calling agent scored 61.2% on retail and 35.2% on airline on the single-attempt measure. In retail, the score fell below 25% when all eight attempts had to succeed. The researchers checked the final database state against the expected state. Use these results to choose what to test, with the model, tools, domains, and simulator kept in view. ICLR τ-bench paper, Table 2 and Figure 4
Figure 1. GPT-4o function calling in the original τ-bench experiment. Retail scored 61.2% on the single-attempt measure and below 25% when all eight attempts had to pass. ICLR 2025, Table 2 and Figure 4
The ICML 2026 τ²-Bench study adds a second source of failure. Its telecom simulator can change shared state while an agent is working. A workflow that passes when the agent is the only actor may fail when a customer, a teammate, or another process acts first. ICML τ²-Bench paper
Test concurrent changes with a fixture that starts from a known authorization and changes the underlying record at a defined step. Check whether the agent revalidates the current record and reaches the correct outcome. Report this scenario separately from straightforward requests and tool timeouts. An aggregate pass rate can hide a weak group.
Build a regression dataset around failure modes
Start with recent support events and near misses that are relevant to the target release. Generalize sensitive data into fixtures and preserve the facts that made the decision hard. Include straightforward successes, boundary cases, and outcomes where the best action is no write. Split cases into a development set for tuning and a locked set used for release. Keep variants of the same underlying case together so that rewording a request does not leak a test into the tuning set.
| Fixture | Fixture condition | Expected final state | Additional check |
|---|---|---|---|
| A01 | A $20.00 credit has valid, current approval. | Exactly one $20.00 ledger entry and one receipt. | The reply cites the recorded amount and result. |
| A02 | The requester supplies two plausible invoices. | No ledger change. | Ask which invoice to use. |
| A03 | The case exceeds the workflow’s approval limit. | No ledger change and one case in the correct queue. | Include the reason in the handoff. |
| A04 | A valid approval referred to an earlier invoice revision. | No ledger change. | Mark the approval stale and request review. |
| A05 | A replacement approval arrives while the case is pending. | Apply only the approved current amount after revalidation. | Save the approval ID tied to the executed action. |
| A06 | A write commits and the simulated response is lost. | One ledger entry after the caller retries. | Return the existing receipt for the same operation key. |
| A07 | The account or invoice is outside the authorized scope. | No restricted data returned and no writes. | Record an authorization decision. |
| A08 | The ledger service is temporarily unavailable. | No unconfirmed success and no duplicate work. | Apply the defined retry or handoff path. |
Make the fake tool return the same fields, errors, and receipt states as the application. Cover approval expiry and the interval between committing a change and returning its response. Confirm contracts in staging when a dependency changes, then reset the state and clock before each trial.
Build an expected state record that CI can grade
Expected state should be machine-readable where possible. This example tests the lost-response scenario. The evaluator reads observed results from the fake application ledger and receipt store. The result stays useful even if the model changes its explanation or returns tool calls in a different valid order.
{
"case_id": "A06",
"fixture": "approved_credit_response_lost",
"trial": 1,
"expected": {
"disposition": "applied",
"ledger_write_count": 1,
"ledger_delta_cents": 2000,
"receipt_count": 1,
"retry_returns_existing_receipt": true,
"unauthorized_effect_count": 0
}
}
For each trial, save the fixture version, evaluation code version, prompt and model identifiers, tool schema versions, and the observed values in the same shape. In CI, fail on missing fields or a mismatched required value. Record the evidence source for every value. A CI score becomes much easier to investigate when an engineer can open the failing fixture and identify which assertion changed. Keep a short reason and a trace reference for every fail.
Grade application state, access, amounts, and operation counts with code. Reserve model grading for judgments such as clarity. Give the grader an adjudicated outcome and a rubric, compare its scores with a human-reviewed sample, and send unclear cases to review. OpenAI evaluation best practices · Google Developers guidance on evaluating agents
Check expected state with a small deterministic grader
Read actual values from the fake application ledger and receipt store. The grader returns an incomplete result when required evidence is missing. If it also observes a mismatch, it reports the failure and retains the missing fields so both issues stay visible.
def grade_trial(expected, observed, critical_fields):
missing = sorted(expected.keys() - observed.keys())
mismatches = sorted(
field for field in expected.keys() & observed.keys()
if type(observed[field]) is not type(expected[field])
or observed[field] != expected[field]
)
critical = sorted(set(mismatches) & set(critical_fields))
if mismatches:
status = "fail"
elif missing:
status = "incomplete"
else:
status = "pass"
return {"status": status, "missing": missing,
"mismatches": mismatches, "critical": critical}
For A06, one write and one receipt satisfy the sample. Two writes produce a critical failure if the write count is marked critical. An absent receipt count leaves the run incomplete when all observed values match. Require explicit policy and fixture versions before the CI job compares candidates.
Read the evaluation scorecard by risk
Report trial success with a numerator, denominator and run conditions. For repeated trials, also report the fraction of cases that passed every attempt. Anthropic describes pass@k for getting at least one pass in k trials and pass to the power k for passing all k trials. A user-facing path that is tried once needs consistent behavior. The all-trials view helps surface cases that oscillate between success and failure even when the average trial score looks stable. Anthropic’s discussion of evaluation metrics
Track critical failures separately from rates. Define critical failures with the people who own policy. For the billing example, an unauthorized mutation, duplicate credit, or committed amount above the approval blocks release. A misleading success message without a matching receipt can also block release depending on the workflow. Capture avoidable handoffs, latency, and cost in separate rows with product-specific thresholds.
Figure 2. Illustrative Uvik Software evaluation scorecard. The candidate has more correct trials and one critical policy failure, which places the release on hold.
| Measure | Baseline | Candidate | Example CI decision |
|---|---|---|---|
| Correct trial outcomes | 57 of 60 | 58 of 60 | Meets the illustrative floor of 57 of 60. |
| Cases passing at least one of three trials | 20 of 20 | 20 of 20 | Shows what both versions can do with retries. |
| Cases passing all three trials | 17 of 20 | 18 of 20 | Meets the illustrative floor of 17 of 20. |
| Required handoffs to the approved queue | 8 of 9 | 9 of 9 | Meets the illustrative minimum of 9 of 9. |
| Critical policy failures | 0 | 1 | Hold. The release requires zero observed critical failures. |
| Final decision | Prior release | Hold | Fix the policy regression and rerun the locked set. |
The 60 simulated trials cover 20 cases, with three trials per case. Both versions pass every case at least once. The candidate passes all three trials in 18 cases and still fails release on one critical policy event. Use this table to decide what blocks a release. A larger, representative sample is needed to estimate production reliability.
Use the sample size to bound what the results can say
A clean test suite has limited statistical power. In a simplified model with 60 independent, risk-representative trials and no failures, a one-sided 95% upper bound on the failure probability is about 4.9%, calculated as one minus 0.05 to the power of one sixtieth. Three attempts on each of 20 fixtures may be correlated. Use the calculation as an illustration of why zero observed errors and a small test suite cannot establish an extremely low production failure rate. Report the number of distinct cases, trials per case, and what proportion covers the most consequential actions.
An operational release pack should show counts by class before totals. For the billing example, inspect how many authorization, concurrent-update, and lost-response fixtures ran and whether every expected system-record field was available. If a whole risk class was skipped, mark the result as incomplete even when the other groups pass. Archive the held-out case set and the run configuration so a reviewer can reproduce the decision.
Run the suite at three points in the release
Run deterministic gateway and policy tests on every relevant change. These provide quick feedback when an interface or approval rule changes. Run the richer model suite against a pinned release candidate and a locked fixture set. Compare it with the current production configuration on the same fixtures, where permitted, so that changes in models, prompts, tools, or data can be attributed.
Shadow or replay the candidate with controlled inputs before enabling writes. Decide which live actions are in scope for a small rollout and who can stop it. During the rollout, look at denied and abandoned tasks, escalation outcomes, action receipts, latency, and cost per completed task. Roll back or disable the risky action path if a critical failure appears. Put confirmed production escapes into the next regression set after removing sensitive data.
If the workflow uses retrieval, grade evidence retrieval and answer grounding as additional layers. Keep the action gate tied to durable effects. A correct document citation is a retrieval success, while a credit against the wrong invoice is still an action failure. Uvik Software’s RAG evaluation metrics and release gates
What Uvik Software changed in a customer service agent
In Uvik Software’s published customer service case, the team rebuilt action validation, escalation routing, and behavioral regression tests. The case reports a reduction in wrong actions from 6.2% to 0.7%, based on action audit logs. The regression suite moved from three hours to 18 minutes, based on CI records, and median escalation handoff moved from 41 seconds to 6 seconds, based on conversation records. The faster suite could run before release. These are the results of that engagement and its workload. Uvik Software’s customer service agent guardrails case study
For the tool and state architecture, see Uvik Software’s guide to building a production AI agent in Python. Bring one action, its approval rule, and a scrubbed failure example when you review your agent release gates with Uvik Software.