Menu

AI Agent Observability for Production Tool Calls and Incidents

AI Agent Observability for Production Tool Calls and Incidents - 9
Paul Francis

Table of content

    Summary

    Key takeaways

    • Measure an AI agent workflow by its verified terminal outcome, not by the HTTP response or an individual span alone.
    • Give each accepted task a workflow ID and link model, tool, policy, and handoff spans to an application outcome record.
    • Preserve durable action and queue receipts even if the diagnostic trace is sampled out.
    • A tool timeout can be recovered by a bounded retry while the overall workflow still succeeds.
    • Count one terminal disposition per workflow, and track still-open work separately.
    • Use explicit denominators for autonomous resolution, correct handoffs, tool retries, latency, and cost per successful task.
    • Alert on user impact with traffic floors, release context, and a dedicated route for confirmed policy violations.
    • Check durable operation status before replaying a write with uncertain completion.
    • Redact prompts, outputs, and sensitive identifiers by default; retain selected diagnostic traces under access controls.

    When this applies

    Use this approach for production AI agents that make model and tool calls, perform retrieval, hand work to queues or operators, or require on-call response to failures, retries, latency, and changing task costs.

    When this does not apply

    A tracing dashboard by itself is not proof that a task completed, that a handoff reached its destination, or that a write was safe to replay. The workflow also requires application-owned outcome and receipt records, agreed counting rules, and privacy controls.

    Checklist

    1. Choose one accepted production workflow and define the outcome expected by its user.
    2. Assign an opaque workflow ID at the entry point.
    3. Propagate trace context through the API, tool adapters, and workers.
    4. Link deferred work back to the originating workflow.
    5. Instrument agent runs, model calls, retrieval, tools, policy checks, approvals, and handoffs.
    6. Pin the supported OpenTelemetry semantic convention version.
    7. Record workflow type, release, retry reason, and error class in appropriate fields.
    8. Write a durable terminal outcome and action or queue receipt.
    9. Count one terminal disposition per workflow and track open tasks separately.
    10. Define the numerator and denominator for resolution and correct handoff rates.
    11. Include retry and failed-run costs in cohort cost calculations.
    12. Measure end-to-end and queue acceptance latency by workflow.
    13. Set alert traffic floors and compare short and long observation windows.
    14. Require a receipt or operation-status check before replaying uncertain writes.
    15. Keep payload capture off by default and save scrubbed incidents as regression fixtures.

    Common pitfalls

    • Treating an HTTP 200 response as proof of a completed task.
    • Marking a whole agent run as failed when an approved retry and handoff succeeded.
    • Counting multiple retry spans as multiple terminal workflows.
    • Omitting still-open tasks from the reporting context.
    • Combining autonomous resolutions and correct handoffs without defining the outcome denominator.
    • Ignoring model and tool costs incurred by failed or retried runs.
    • Alerting on isolated tool errors without checking queue receipts or user outcomes.
    • Retrying a timed-out write without checking whether it already committed.
    • Keeping customer content or sensitive identifiers in standard traces.

    AI agent observability should tell the on-call engineer whether the user got the intended result and where a run went wrong. Uvik Software recommends starting with one production workflow. Link model calls, tool calls, and handoffs to a durable receipt or terminal application state. Use those outcomes to alert on failures, stalled work, and changes in cost per successful task.

    A support agent can return a 200 response before it reaches its queue. It can also recover from a tool timeout and complete the handoff correctly. The fictional support trace below shows which evidence separates those cases.

    Follow an agent run to its terminal outcome

    Give each accepted task an opaque workflow ID. Propagate trace context through the API, tool adapters, and workers. Link deferred work to the originating run when a parent span alone cannot represent it. Store the terminal outcome and any action receipt in the application so operators can verify completion even when a trace was sampled out.

    Track the agent run, model inference, retrieval, tool calls, policy checks, approvals, and queue handoff as separate spans or events when they affect the workflow. Parent-child relationships describe call order. Business outcome records establish what finished. A tool call can fail while the overall run succeeds if the agent uses an approved fallback. OpenTelemetry context propagation · Microsoft Foundry agent tracing

    Use the OpenTelemetry generative AI conventions your instrumentation supports, including separate agent invocation and tool execution where available. Pin the supported convention version and keep your application’s workflow outcome, release version, and retry reason in documented application fields. Use bounded values for metrics, such as workflow type and tool name. Keep per-run identifiers available for trace lookup and controlled evidence joins. OpenTelemetry agent spans · OpenTelemetry tool execution spans

    Define a terminal outcome record before adding dashboards

    Link the workflow to an outcome event written by the application that accepted the handoff. In this example the queue accepts the case at 3.71 seconds and the agent returns at 3.84 seconds. The application event records the handoff when it is accepted. Request latency still ends at the response.

    {
      "workflow_id": "wf_demo_0418",
      "trace_id": "af04e55f870b4e02baf9806396cb5aca",
      "workflow_type": "support_account_change",
      "release": "support-2026-11-20.1",
      "outcome": "handed_off",
      "queue_receipt": "queue_demo_360",
      "event_time_utc": "2026-11-20T15:46:03.710Z",
      "elapsed_ms": 3710
    }

    Record one terminal disposition per workflow in outcome aggregates. A retried tool span may appear several times. The workflow still contributes one result. Join by the application workflow ID and use the trace ID for inspection. Keep both IDs out of metric labels and store them only where access and retention rules permit. OpenTelemetry context propagation

    Inspect a redacted support workflow trace

    The run below starts with a customer asking about an account change. The agent can look up the status but a person must approve the change. The first tool call times out. A bounded retry succeeds and the agent routes the case to an operator. The queue receipt makes this a successful handoff even though the agent did not resolve the request itself.

    A 3.84-second support run includes authorization, model routing, a first status attempt that times out, a successful read retry, a follow-up model call, a queue handoff, and a customer reply.

    Uvik Software’s illustrative support trace. A timed-out read is followed by a bounded retry and a confirmed queue handoff.

    Relative time Span or event Result worth keeping
    0.00 to 3.84 s Support routing run Release ID, workflow type, terminal outcome handed off
    0.00 to 0.18 s Authorize and retrieve context Access permitted, one policy record used
    0.18 to 0.76 s Model routing call Chooses the read-only status tool
    0.76 to 1.91 s Status tool attempt one Deadline exceeded on a read
    1.96 to 2.43 s Status tool attempt two Same read succeeds after bounded retry
    2.43 to 2.91 s Model follow-up Requests human review for the account change
    2.91 to 3.71 s Queue handoff Accepted by the support queue with a receipt
    3.71 to 3.84 s Respond to the customer Confirmed handoff reported

    This example uses elapsed time since the run started. The trace shows both failed and successful calls. For latency budgeting, keep the durations tied to the run and account for overlaps when calls run concurrently. A retry count should describe actual attempts. Include the tool’s error class, timeout, and retry policy, and keep the payload itself outside standard traces.

    Measure outcomes and cost with explicit denominators

    Establish terminal statuses such as resolved, handed off, failed, and cancelled. Count them by accepted workflow cohort and close the cohort after an agreed observation window. Track still-open runs separately so a stalled handoff cannot disappear from the denominator. Record cost for retries and failed runs as well as successful runs. Use the same definition of metered cost when comparing versions.

    Dashboard measure Suggested definition What the operator checks
    Successful autonomous resolution Verified autonomous completions divided by accepted tasks with an outcome, with open tasks tracked alongside Ledger, ticket, or other action receipt confirms the claimed result
    Correct handoff rate Correct queue receipts divided by tasks expected to hand off, when that expectation can be determined Queue receipt, handoff age, and policy sample
    Tool timeout and retry rate Timeouts divided by calls and retried runs divided by accepted runs, by tool Dependency traces, timeout configuration, error class, and deployment version
    End to end and handoff latency p50 and p95 by workflow, plus time from handoff request to acceptance Longest path, open queue age, and dependency wait
    Cost per successful task Metered model and tool cost across the cohort divided by approved successful outcomes Model mix, number of calls, retrieval, retries, and outcome denominator
    Policy and audit events Count of confirmed critical violations and missing required receipts Durable application events and named incident owner

    For a shared support automation dashboard, publish both cost per autonomous resolution and cost per correctly handled task. In a constructed cohort of 100 tasks with terminal outcomes, assume 62 resolve autonomously, 23 reach the right human queue, 15 fail, and metered model and tools cost $8.68. Cost per autonomous resolution is $0.14. Cost per correctly handled task is about $0.10 when the definition includes verified handoffs. Human labor is outside this particular cost measure. State which costs and outcomes you count before the first comparison.

    Build alerts around the effect on users. Set a minimum traffic floor for percentage alerts and check error or cost changes in a short and a longer window. A confirmed policy violation deserves its own alert path. The readiness thresholds in development and the incident thresholds in production may use different cohorts. Use engineering and operations review to choose both. Google SRE guidance on monitoring distributed systems

    Turn dashboard measures into alerts with a traffic floor

    Use different thresholds for external impact and root-cause hints. Suppose normal status tool timeouts are 6 out of 400 calls in a 30-minute window, or 1.5%, and a new release produces 20 of 400, or 5.0%. A team with an illustrative 3% timeout threshold and a minimum of 100 calls has enough volume to investigate this window. Check the user outcome at the same time. If the accepted queue receipts also fell from 397 of 400 attempted handoffs to 366 of 400, the handoff acceptance rate would move from 99.25% to 91.50%. These are synthetic figures with separate denominators. The second measure shows why alerting on tool errors alone leaves a gap.

    The failed read consumed 1.15 seconds, about 30% of this 3.84-second run. Both model calls together took 1.06 seconds. Compare healthy and affected cohorts before changing timeouts. A single slow trace helps locate the suspect; the distribution tells you how much work was affected.

    Use a short incident runbook when retries spike

    Suppose queue receipts fall and the status tool retry rate triples after a rollout. Start by identifying which users lack a recorded result. Check whether the retry path is still handing work to the queue.

    1. Identify the affected workflow and release. Check the share of accepted tasks reaching a terminal result and the age of cases with no queue receipt.
    2. Open a trace from the affected cohort and a recent healthy one. Confirm whether retries occur on the same failure class, which dependency fails first, and whether the agent reaches its handoff logic.
    3. Apply the approved mitigation for that workflow. For a read tool, contain retries or direct eligible cases to human support. For any write with uncertain completion, read its receipt or operation status before deciding whether to retry.
    4. Verify the mitigation from durable outcomes. Check that new handoffs are accepted, outstanding eligible cases are routed once, and incorrect or duplicate operations have an owner.
    5. Save a scrubbed incident fixture. Add a regression assertion on bounded retries and the correct terminal outcome before restoring full rollout.

    Label alerts with the workflow, release, denominator, threshold, and dashboard link. Assign one owner for the recovery decision even when the issue spans the model provider, API integrations, and a support team. An incident timeline can be assembled from the trace and durable events without requiring someone to inspect a full chat transcript.

    Decide what to do when telemetry disagrees

    Evidence pattern First action
    A read times out, a bounded retry succeeds, and an accepted queue receipt exists Keep the valid handoff. Investigate the dependency if the timeout rate crosses its agreed threshold.
    The response reports a handoff and the queue has no receipt Mark the workflow unresolved or open, check the operation record, and route it once through the approved recovery path.
    A write times out and the caller cannot tell whether it committed Check the durable operation status before any replay. Send an unresolved case to an operator if the implementation has no safe way to recover.
    The trace is absent and a verified receipt exists Count the application outcome. Inspect sampling, collection, or propagation if the missing trace rate is unexpected.

    This decision table uses the application’s recorded effects as evidence. It applies even if two integrations report different span outcomes for the same attempt.

    Retain useful evidence and minimize content exposure

    Disable payload capture by default for prompts, completions, tool arguments, and tool results. Give the instrumentation an allowlist of safe fields and apply redaction before export. Prevent secrets and sensitive identifiers from entering trace baggage. Store any approved diagnostic samples in a restricted system with a retention policy and audit trail. Treat user, invoice, and case data as sensitive even when the user does not see it in the application. OpenTelemetry guidance on handling sensitive data · OpenTelemetry baggage guidance

    Keep independent counts for denominators and durable evidence for actions and approvals. Sampled traces support investigation. They do not replace the system that records a receipt or the metrics used to measure all accepted tasks. Tail sampling can retain selected slow or erroneous traces after a collector has more context. Pair it with capacity planning and rules for the errors and workflows you care about. OpenTelemetry sampling overview

    Capture evidence that helps find the first failure

    The AgentRx study in Findings of EMNLP 2026 published 170 manually annotated failed trajectories across 11 task settings. On its τ-bench comparison, AgentRx identified the exact failure step in 42.7% of traces on average across three runs, compared with 18.8% for an adapted Who&When baseline. The benchmark compares diagnostic methods under study conditions. In production, store enough validated evidence for an engineer to tell whether a tool error was recovered or became the first unresolved failure. AgentRx study, Table 3

    Exact failure-step accuracy was 18.8 versus 42.7 percent on τ-bench, 22.7 versus 36.4 on Magentic, 64.3 versus 82.5 on Flash, and 32.6 versus 61.5 on RelWork, for the adapted baseline and AgentRx respectively.

    Figure 2. Mean exact-step accuracy on four AgentRx study test sets. The comparison uses the study’s adapted Who&When baseline. Findings of EMNLP 2026, Table 3

    LangChain’s State of Agent Engineering report offers a separate adoption measure. Among more than 1,300 professional respondents, about 89% reported using some form of observability and 62% reported detailed tracing of agent steps and tool calls. For your workflow, verify that traces join to the application record and that the join works on failed runs as well as successful ones. LangChain State of Agent Engineering

    How Uvik Software instrumented a production AI workflow

    In a published prototype-to-production engagement, Uvik Software added per-query cost and latency monitoring, including p50, p95 and p99, along with routing across models and budget alerts. The team used this evidence to validate the workflow’s operating costs against the product’s pricing. The case reports that a validated workflow shipped in 14 weeks and that production p95 was under 4.5 seconds on the standard workflow. The trace and cost values in this article are illustrative. Uvik Software’s generative AI prototype case study

    In a separate observability-platform engagement, Uvik Software’s published case reports that trace-based evaluation brought the time to detect a regression from nine days to 40 minutes. That result describes the detection workflow in that engagement. Pair it with your own baseline for time to detect and time to confirm recovery. Uvik Software’s trace-based evaluation case study

    For tooling choices, see Uvik Software’s comparison of LLM observability tools for Python agents. If a running workflow is hard to diagnose, bring a scrubbed trace and one failed or delayed outcome when you review your agent telemetry with Uvik Software.

    Frequently asked questions

    What is AI agent observability?

    It is the ability to understand what the agent, its model and tools did, why the run ended as it did, and whether the user’s task reached the expected outcome. It combines traces, aggregate metrics, and records of observed effects.

    What should you instrument first?

    Start with one user workflow and its entry point, model call, tool calls, and terminal outcome. Add the queue or action receipt, release version, and timestamps. Track the same workflow on a dashboard before expanding to more tools and agents.

    How is agent tracing different from agent evaluation?

    A trace records a particular run. Evaluation places an expected result alongside the observed run and scores it. Production traces help identify useful regression cases after sensitive details are removed.

    How do you calculate agent cost per task?

    Divide metered costs for model and tool calls in an agreed cohort by the number of accepted outcomes in that cohort. Include costs from retries and failed runs. Define how you count handoffs and outstanding tasks.

    Can you debug an agent without storing customer conversations?

    Often you can isolate a problem from the error class, workflow, span durations, operation status, and release version. Use restricted, approved sample data when root-cause analysis truly requires more content.

    Should a team keep every production trace?

    Keep unsampled counts for the metrics used in release and incident decisions. Choose trace retention based on traffic, privacy, and operational risk. Preserve required action and approval records in the system responsible for those records.

    What should be in an agent incident review?

    Include the affected workflow, release, exposed task cohort, error and outcome rates, and the action that restored service. Attach a scrubbed trace and identify the missing assertion or alert that would have shortened detection.

    How useful was this post?

    No votes so far! Be the first to rate this post.

    Share:
    AI Agent Observability for Production Tool Calls and Incidents - 12

    Need to augment your IT team with top talents?

    Uvik can help!
    Contact
    Uvik Software
    Privacy Overview

    This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

    Get a free project quote!
    Fill out the inquiry form and we'll get back as soon as possible.

      Subscribe to TechTides – Your Biweekly Tech Pulse!
      Join 750+ subscribers who receive 'TechTides' directly on LinkedIn. Curated by Paul Francis, our founder, this newsletter delivers a regular and reliable flow of tech trends, insights, and Uvik updates. Don’t miss out on the next wave of industry knowledge!
      Subscribe on LinkedIn