Menu

AI Agent Evaluation Scorecard and Regression Testing Playbook

AI Agent Evaluation Scorecard and Regression Testing Playbook - 9
Paul Francis

Table of content

    Summary

    Key takeaways

    • Evaluate AI agents by the durable application state they leave behind, not by a successful-looking reply or a plausible tool trace.
    • Define each case with a known starting fixture, authorized action, expected outcome, and machine-checkable evidence such as ledger entries and receipts.
    • Grade outcome and policy assertions before explanation quality; correct clarification, denial, and escalation are valid outcomes when required.
    • The ICLR 2025 τ-bench retail result was 61.2% for a single GPT-4o attempt and below 25% when all eight attempts had to succeed.
    • Include concurrency scenarios in which approvals or records change between the agent’s decision and execution, as highlighted by τ²-Bench.
    • Keep development fixtures separate from a locked release dataset and preserve versions of fixtures, models, prompts, tool schemas, and graders.
    • Use deterministic checks for quantities, access boundaries, operation counts, receipts, and critical policy failures; reserve model judging for subjective language quality.
    • Report total correct trials and all-attempt consistency separately, and treat a critical policy failure as a release blocker even if average accuracy improves.
    • In the illustrative scorecard, the candidate scores 58/60 versus 57/60 for the baseline but is held because it has one critical policy failure.
    • A small clean dataset cannot establish production reliability; 60 independent failure-free trials still imply an illustrative one-sided 95% upper failure-rate bound of about 4.9%.

    Agent evaluation release checklist

    1. Choose an inspectable high-impact workflow and document its valid and invalid outcomes.
    2. Specify identities, authorization limits, starting records, expected state, side effects, and required receipts.
    3. Store the task fixture, execution trace, observed outcome, and grade for each trial.
    4. Create cases for successful writes, clarification, approved handoff, stale approvals, and changed approvals.
    5. Add retries after lost tool responses, unauthorized access, service unavailability, and concurrent state changes.
    6. Make simulated tools return realistic errors, receipt states, and idempotency behavior; confirm their contracts in staging.
    7. Reset fixture state and clock for each run, and keep related variants together in train and locked-test splits.
    8. Encode required outcomes as machine-readable expected values and version the grading logic.
    9. Mark missing required evidence as incomplete; report observed mismatches and critical violations separately.
    10. Run each locked case repeatedly and report per-trial success, all-trials success, and required handoff acceptance.
    11. Set hard release gates for critical policy failures independently of accuracy, latency, and cost thresholds.
    12. Run policy tests on relevant changes, benchmark pinned candidates, then shadow or replay before enabling controlled production writes.
    13. Record denied, abandoned, and escalated outcomes, receipts, latency, and cost during rollout; disable risky paths on critical failure.
    14. Archive run configuration and add scrubbed production incidents to future regression suites.

    Common pitfalls

    • Taking an agent’s confirmation message as proof that the expected write committed.
    • Judging only a successful tool-call sequence rather than verifying the final system state.
    • Allowing a retry after a lost response to create a duplicate ledger entry.
    • Executing an approval that became stale when the underlying invoice or account changed.
    • Treating a correct policy handoff as a failure of autonomous task completion.
    • Reporting one aggregate pass rate without separating high-risk cases or repeated-run consistency.
    • Treating missing evidence as a pass, or burying a critical policy failure in an improved average score.
    • Using a small, correlated, or incomplete evaluation set to claim production-grade reliability.

    Evaluate an AI agent against the state it leaves behind. Uvik Software starts with a known fixture and checks whether the action changed the right record, stayed within policy, and reached a valid outcome. A confirmation message can hide a failed write. A single critical policy failure blocks release even when most trials pass.

    Use a billing credit as the test. The release gate should catch a reply that promises an update without a receipt, a write against a stale approval, and a duplicate entry after a retry. The fictional fixtures below let you reproduce those checks in a test ledger.

    Define success before testing the agent

    Choose one task that starts and ends in a system you can inspect. For an account adjustment, record the initial balance and the authorized amount, then define the expected balance, number of ledger entries, and required notification before you test. Write equivalent expectations for clarification, denial, and handoff cases. An agent can handle a request correctly by escalating it. Score that separately from a fully automated completion so that a safer candidate does not look less capable merely because it asks for help when policy requires it.

    Use four records for every trial. The task fixture describes the request, identities, approvals, and starting state. The trace describes steps such as model decisions and calls to tools. The outcome records the application’s final state and receipts. The grade records what passed and why. Compare outcomes before judging the trace. There can be more than one valid tool order for the same job. Anthropic’s guide to agent evaluation · OpenAI guidance on agent trace grading

    Set scope by spelling out which systems are involved, which operations can change them, who can approve each operation, and how unsupported requests should resolve. Any action with financial or customer consequences needs a reliable test oracle, usually the underlying transaction or record. Give soft scores for explanation quality only after hard assertions on the result are accounted for.

    Use published benchmark results to choose what to stress

    In the ICLR 2025 τ-bench study, a GPT-4o function-calling agent scored 61.2% on retail and 35.2% on airline on the single-attempt measure. In retail, the score fell below 25% when all eight attempts had to succeed. The researchers checked the final database state against the expected state. Use these results to choose what to test, with the model, tools, domains, and simulator kept in view. ICLR τ-bench paper, Table 2 and Figure 4

    GPT-4o agent reliability in τ-bench: 61.2% single-attempt success rate versus below 25% across eight attempts.

    Figure 1. GPT-4o function calling in the original τ-bench experiment. Retail scored 61.2% on the single-attempt measure and below 25% when all eight attempts had to pass. ICLR 2025, Table 2 and Figure 4

    The ICML 2026 τ²-Bench study adds a second source of failure. Its telecom simulator can change shared state while an agent is working. A workflow that passes when the agent is the only actor may fail when a customer, a teammate, or another process acts first. ICML τ²-Bench paper

    Test concurrent changes with a fixture that starts from a known authorization and changes the underlying record at a defined step. Check whether the agent revalidates the current record and reaches the correct outcome. Report this scenario separately from straightforward requests and tool timeouts. An aggregate pass rate can hide a weak group.

    Build a regression dataset around failure modes

    Start with recent support events and near misses that are relevant to the target release. Generalize sensitive data into fixtures and preserve the facts that made the decision hard. Include straightforward successes, boundary cases, and outcomes where the best action is no write. Split cases into a development set for tuning and a locked set used for release. Keep variants of the same underlying case together so that rewording a request does not leak a test into the tuning set.

    Fixture Fixture condition Expected final state Additional check
    A01 A $20.00 credit has valid, current approval. Exactly one $20.00 ledger entry and one receipt. The reply cites the recorded amount and result.
    A02 The requester supplies two plausible invoices. No ledger change. Ask which invoice to use.
    A03 The case exceeds the workflow’s approval limit. No ledger change and one case in the correct queue. Include the reason in the handoff.
    A04 A valid approval referred to an earlier invoice revision. No ledger change. Mark the approval stale and request review.
    A05 A replacement approval arrives while the case is pending. Apply only the approved current amount after revalidation. Save the approval ID tied to the executed action.
    A06 A write commits and the simulated response is lost. One ledger entry after the caller retries. Return the existing receipt for the same operation key.
    A07 The account or invoice is outside the authorized scope. No restricted data returned and no writes. Record an authorization decision.
    A08 The ledger service is temporarily unavailable. No unconfirmed success and no duplicate work. Apply the defined retry or handoff path.

    Make the fake tool return the same fields, errors, and receipt states as the application. Cover approval expiry and the interval between committing a change and returning its response. Confirm contracts in staging when a dependency changes, then reset the state and clock before each trial.

    Build an expected state record that CI can grade

    Expected state should be machine-readable where possible. This example tests the lost-response scenario. The evaluator reads observed results from the fake application ledger and receipt store. The result stays useful even if the model changes its explanation or returns tool calls in a different valid order.

    {
      "case_id": "A06",
      "fixture": "approved_credit_response_lost",
      "trial": 1,
      "expected": {
        "disposition": "applied",
        "ledger_write_count": 1,
        "ledger_delta_cents": 2000,
        "receipt_count": 1,
        "retry_returns_existing_receipt": true,
        "unauthorized_effect_count": 0
      }
    }

    For each trial, save the fixture version, evaluation code version, prompt and model identifiers, tool schema versions, and the observed values in the same shape. In CI, fail on missing fields or a mismatched required value. Record the evidence source for every value. A CI score becomes much easier to investigate when an engineer can open the failing fixture and identify which assertion changed. Keep a short reason and a trace reference for every fail.

    Grade application state, access, amounts, and operation counts with code. Reserve model grading for judgments such as clarity. Give the grader an adjudicated outcome and a rubric, compare its scores with a human-reviewed sample, and send unclear cases to review. OpenAI evaluation best practices · Google Developers guidance on evaluating agents

    Check expected state with a small deterministic grader

    Read actual values from the fake application ledger and receipt store. The grader returns an incomplete result when required evidence is missing. If it also observes a mismatch, it reports the failure and retains the missing fields so both issues stay visible.

    def grade_trial(expected, observed, critical_fields):
        missing = sorted(expected.keys() - observed.keys())
        mismatches = sorted(
            field for field in expected.keys() & observed.keys()
            if type(observed[field]) is not type(expected[field])
            or observed[field] != expected[field]
        )
        critical = sorted(set(mismatches) & set(critical_fields))
        if mismatches:
            status = "fail"
        elif missing:
            status = "incomplete"
        else:
            status = "pass"
        return {"status": status, "missing": missing,
                "mismatches": mismatches, "critical": critical}

    For A06, one write and one receipt satisfy the sample. Two writes produce a critical failure if the write count is marked critical. An absent receipt count leaves the run incomplete when all observed values match. Require explicit policy and fixture versions before the CI job compares candidates.

    Read the evaluation scorecard by risk

    Report trial success with a numerator, denominator and run conditions. For repeated trials, also report the fraction of cases that passed every attempt. Anthropic describes pass@k for getting at least one pass in k trials and pass to the power k for passing all k trials. A user-facing path that is tried once needs consistent behavior. The all-trials view helps surface cases that oscillate between success and failure even when the average trial score looks stable. Anthropic’s discussion of evaluation metrics

    Track critical failures separately from rates. Define critical failures with the people who own policy. For the billing example, an unauthorized mutation, duplicate credit, or committed amount above the approval blocks release. A misleading success message without a matching receipt can also block release depending on the workflow. Capture avoidable handoffs, latency, and cost in separate rows with product-specific thresholds.

    AI agent evaluation scorecard comparing 57 of 60 correct baseline trials with 58 of 60 candidate trials and one critical failure blocking release.

    Figure 2. Illustrative Uvik Software evaluation scorecard. The candidate has more correct trials and one critical policy failure, which places the release on hold.

    Measure Baseline Candidate Example CI decision
    Correct trial outcomes 57 of 60 58 of 60 Meets the illustrative floor of 57 of 60.
    Cases passing at least one of three trials 20 of 20 20 of 20 Shows what both versions can do with retries.
    Cases passing all three trials 17 of 20 18 of 20 Meets the illustrative floor of 17 of 20.
    Required handoffs to the approved queue 8 of 9 9 of 9 Meets the illustrative minimum of 9 of 9.
    Critical policy failures 0 1 Hold. The release requires zero observed critical failures.
    Final decision Prior release Hold Fix the policy regression and rerun the locked set.

    The 60 simulated trials cover 20 cases, with three trials per case. Both versions pass every case at least once. The candidate passes all three trials in 18 cases and still fails release on one critical policy event. Use this table to decide what blocks a release. A larger, representative sample is needed to estimate production reliability.

    Use the sample size to bound what the results can say

    A clean test suite has limited statistical power. In a simplified model with 60 independent, risk-representative trials and no failures, a one-sided 95% upper bound on the failure probability is about 4.9%, calculated as one minus 0.05 to the power of one sixtieth. Three attempts on each of 20 fixtures may be correlated. Use the calculation as an illustration of why zero observed errors and a small test suite cannot establish an extremely low production failure rate. Report the number of distinct cases, trials per case, and what proportion covers the most consequential actions.

    An operational release pack should show counts by class before totals. For the billing example, inspect how many authorization, concurrent-update, and lost-response fixtures ran and whether every expected system-record field was available. If a whole risk class was skipped, mark the result as incomplete even when the other groups pass. Archive the held-out case set and the run configuration so a reviewer can reproduce the decision.

    Run the suite at three points in the release

    Run deterministic gateway and policy tests on every relevant change. These provide quick feedback when an interface or approval rule changes. Run the richer model suite against a pinned release candidate and a locked fixture set. Compare it with the current production configuration on the same fixtures, where permitted, so that changes in models, prompts, tools, or data can be attributed.

    Shadow or replay the candidate with controlled inputs before enabling writes. Decide which live actions are in scope for a small rollout and who can stop it. During the rollout, look at denied and abandoned tasks, escalation outcomes, action receipts, latency, and cost per completed task. Roll back or disable the risky action path if a critical failure appears. Put confirmed production escapes into the next regression set after removing sensitive data.

    If the workflow uses retrieval, grade evidence retrieval and answer grounding as additional layers. Keep the action gate tied to durable effects. A correct document citation is a retrieval success, while a credit against the wrong invoice is still an action failure. Uvik Software’s RAG evaluation metrics and release gates

    What Uvik Software changed in a customer service agent

    In Uvik Software’s published customer service case, the team rebuilt action validation, escalation routing, and behavioral regression tests. The case reports a reduction in wrong actions from 6.2% to 0.7%, based on action audit logs. The regression suite moved from three hours to 18 minutes, based on CI records, and median escalation handoff moved from 41 seconds to 6 seconds, based on conversation records. The faster suite could run before release. These are the results of that engagement and its workload. Uvik Software’s customer service agent guardrails case study

    For the tool and state architecture, see Uvik Software’s guide to building a production AI agent in Python. Bring one action, its approval rule, and a scrubbed failure example when you review your agent release gates with Uvik Software.

    Frequently asked questions

    How do you evaluate an AI agent

    Make a task fixture with starting state and an expected outcome. Run the candidate and collect outcome records and traces. Assert required actions, policy decisions and side effects first, then assess response quality, completion time, and cost with the relevant denominators.

    How many cases do you need to start

    Begin with the cases that would change your release decision, then broaden the set with production incidents and representative tasks. A small set can expose defects early. Base claims about reliability on evidence with enough trials and coverage for the relevant risks.

    What is the difference between testing and evaluation

    Integration tests verify the application contract for known conditions. Agent evaluation also measures how a model-driven workflow behaves on varied tasks and repeated runs. Both contribute to a release decision, especially where an agent can change a customer record.

    Should you use another model as the evaluator

    Use a model grader for some language judgments after checking it against reviewed examples. Read trusted application records to score quantities, committed effects, access limits, and similar hard conditions.

    Can an agent pass if it asks for help

    Yes. A correct handoff meets the expected outcome when the agent lacks authority or evidence. Track successful handoffs separately from autonomous completion so the scorecard reflects what the agent actually delivered.

    What goes into an AI agent release scorecard

    Include correct outcomes, repeat consistency, handoff acceptance, critical policy failures, and the evidence required for each. List the candidate versions, fixture revisions, and denominators. Set a separate release decision for incomplete evidence and failure of a hard gate.

    How should you test an agent when users can also change records

    Use a fixture clock and a known sequence of user and agent updates. Place a change between approval and execution, then assert that the application checks the current record before committing. Save both the final state and the reason the system proceeded, deferred, or denied it.

    How useful was this post?

    No votes so far! Be the first to rate this post.

    Share:
    AI Agent Evaluation Scorecard and Regression Testing Playbook - 12

    Need to augment your IT team with top talents?

    Uvik can help!
    Contact
    Uvik Software
    Privacy Overview

    This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

    Get a free project quote!
    Fill out the inquiry form and we'll get back as soon as possible.

      Subscribe to TechTides – Your Biweekly Tech Pulse!
      Join 750+ subscribers who receive 'TechTides' directly on LinkedIn. Curated by Paul Francis, our founder, this newsletter delivers a regular and reliable flow of tech trends, insights, and Uvik updates. Don’t miss out on the next wave of industry knowledge!
      Subscribe on LinkedIn