Menu

LLM Observability Tools for Production Python Agents

LLM Observability Tools for Production Python Agents - 9
Paul Francis

Table of content

    Summary

    Key takeaways

    • An HTTP 200 response can conceal duplicate committed writes or actions completed without a required approval event.
    • Choose LLM observability tools by replaying incidents your team must diagnose, not just comparing feature lists.
    • The five candidates are Langfuse, LangSmith, Arize Phoenix, Braintrust, and Datadog Agent Observability.
    • Traces describe requests or tasks, spans describe individual operations, and evaluations assess results against rules or reference examples.
    • Record operation keys, durable write receipts, approval checkpoints, and final dispositions for state-changing workflows.
    • Export OpenTelemetry traces with available model and token usage fields, verify application-owned fields after export, and allowlist sensitive data.
    • In Uvik Software’s scripted 100-run Python exercise, all responses were HTTP 200. Retrieval and model-cost rules flagged 10 runs; adding receipts and approval events increased the count to 20.
    • At 100,000 monthly runs and eight spans per run, the example workload produces about 800,000 spans before sampling.
    • Retain incident traces under an approved policy, but do not estimate incident rates from a failure-biased sample alone.
    • Give a named alert owner authority to pause the affected action and follow an incident through to a corrective change.

    When this applies

    This approach applies when production Python agents retrieve evidence, call models, perform tool actions, or hand work across services. It is useful for teams comparing tracing and evaluation tools that need to reconstruct business outcomes, investigate approval or retry failures, and estimate retention costs using the same test incidents.

    When this does not apply

    Trace inspection alone does not establish how many writes committed if durable receipts are missing. HTTP status and retry counts cannot replace business-outcome checks, and a failure-biased trace sample cannot establish the overall incident rate. A tool comparison without a named alert owner or a test of incident resolution does not complete the operating handoff described here.

    Checklist

    1. Identify the production agent incidents engineers need to investigate.
    2. Shortlist tools against the same tracing, evaluation, deployment, and cost needs.
    3. Represent each request or task as a trace and its operations as spans.
    4. Capture retrieval, model-call, and tool-attempt information where available.
    5. Record an operation key that connects related tool attempts.
    6. Record durable receipts that distinguish committed writes from retry attempts.
    7. Capture the approval checkpoint, its timestamp, and final disposition.
    8. Include available model and token usage information in the trace.
    9. Verify that application-owned outcome fields survive export to each candidate tool.
    10. Apply a field allowlist before export and use synthetic payloads in the pilot.
    11. Recreate control, empty-retrieval, duplicate-write, approval-bypass, and cost-overrun scenarios while retaining HTTP 200.
    12. Join receipts by operation key within each run and apply the same detection rules.
    13. Send equivalent traces to candidates in staging, time investigations, and record missing fields or links.
    14. Calculate spans before sampling; compare billable units, storage, evaluation, operator time, and retention on the same basis.
    15. Assign an alert owner with authority to pause the action and test the handoff through a resolving code or workflow change.

    Common pitfalls

    • Treating HTTP 200 as proof that an agent completed the intended business action correctly.
    • Counting tool retries without checking how many downstream writes committed.
    • Omitting the shared operation key needed to join related attempts and receipts.
    • Recording that approval was required without checking for an earlier approval event.
    • Assuming custom trace fields arrived in the destination without verifying the export.
    • Sending unnecessary sensitive fields instead of using an export allowlist and synthetic test payloads.
    • Comparing candidate tools with different scenarios or failing to rotate their review order.
    • Estimating platform cost without matching telemetry units and retention windows.
    • Calculating overall incident rates from traces sampled preferentially for failures.
    • Creating alerts without an owner who can pause the affected action and drive resolution.

    Five tools compared, with a repeatable trace exercise for the failures a successful HTTP response can hide.

    A support agent can return HTTP 200 after posting the same reply twice. The first write committed, the response timed out, and the retry made a second write. A request counter sees one success. The engineer investigating the incident needs to see both attempts and the operation they shared.

    Uvik Software recommends choosing LLM observability tools by replaying the incidents your team needs to diagnose. The first pilot should show whether an engineer can reconstruct a tool action across services, find the version that triggered it, and estimate the cost of keeping enough traces to catch it again.

    Five tools to shortlist

    Arize AI, the maker of Phoenix, was a Uvik Software client. Product capabilities were checked against vendor documentation on November 15, 2026. Arize AI engagement

    Tool Best fit What to verify in a pilot
    Langfuse Self-hosted or managed tracing and evaluation, with OpenTelemetry ingestion over HTTP. Check custom tool fields and which telemetry units your plan bills. Langfuse tracing · OTLP support
    LangSmith LangChain and LangGraph teams using run traces and evaluation datasets. OpenTelemetry ingestion is available. Follow a trace across workers and check retention. Self-hosting is an Enterprise add-on. LangSmith tracing · Deployment
    Arize Phoenix Self-hosted Python tracing and evaluation using OpenInference and OTLP. Verify field mapping. Budget for storage and backups. Phoenix uses the Elastic License 2.0. Phoenix tracing · License
    Braintrust Teams using production traces to build evaluation datasets and compare experiments. Test trace-to-dataset review. Price processed data and scoring. Braintrust tracing · Pricing
    Datadog Agent Observability Teams using Datadog application performance monitoring (APM) to follow AI calls into surrounding services. Follow a Python agent trace into a downstream worker and related APM spans. Correlation guide

    Record the business outcome

    A trace represents one request or task. Its spans capture individual operations such as retrieval, a model call, or a tool attempt. An evaluation checks the result against a rule or reference example. In a workflow that acts on customer records, include the operation key, durable receipt, approval checkpoint, and final disposition. A retry count tells you how often the client called a tool. The receipts show how many writes committed.

    Teams can export traces using the OpenTelemetry Protocol (OTLP). Record model and token usage where the integration exposes them. Add application-owned fields for the outcome you need to investigate. Check those fields in the destination after export. Apply a field allowlist before traces leave the application and use synthetic payloads for the pilot. OpenTelemetry GenAI attributes · Sensitive data guidance

    What HTTP status hid in a trace exercise

    In Uvik Software’s scripted Python exercise, all 100 simulated support-agent runs returned HTTP 200. Rules using retrieval and model-cost fields flagged 10 runs. Adding tool receipts and approval events raised that count to 20. The exercise included 80 controls and five runs per failure scenario, with one fault assigned to each affected run.

    Scenario Runs Signal used by the rule Flagged
    Control 80 Three retrieved documents, one recorded write, and $0.012 model cost. 0
    Empty retrieval 5 The agent returned an answer that required evidence after the retrieval span recorded zero documents. 5
    Duplicate write 5 The downstream service recorded two committed writes with the same operation key. 5
    Approval bypass 5 A write committed with approval required and no prior approval event. 5
    Cost overrun 5 The simulated model cost reached $0.035 against a $0.020 per-run budget. 5

    Recreate the exercise from the control, changing only the fields shown for each scenario. Keep HTTP status at 200. Join receipts by operation key within each run and record the approval timestamp. In staging, send equivalent traces to each candidate and time how long reviewers take to find the cause. For unresolved incidents, record the field or link they needed. Rotate the tool order to reduce learning effects.

    Price the traces you expect to keep

    At 100,000 agent runs per month and a mean of eight spans per run, a team produces about 800,000 spans before sampling. Start your estimate there. A vendor may bill on traces, stored observations, processed data, or evaluation usage. Add the cost of storage, evaluator calls, and operator time where applicable. Record the retention window in the estimate so each quote covers the same investigation period. Langfuse billable units · LangSmith pricing · Braintrust pricing

    Count incidents and total runs before sampling so both numbers cover the same traffic. Retain traces that show policy violations, failed actions, or expensive exceptions under your approved retention policy. Use those traces to investigate causes. A sample that favors failures will overstate their rate if you calculate it from the sample alone.

    Assign an owner to the alert

    Uvik Software reports that a rebuild of Arize AI’s trace ingestion and evaluation pipeline shortened detection of a quality regression from nine days to 40 minutes in that engagement. The team moved from nightly, sampled evaluation to continuous evaluation of arriving traces and checked judge scores against human labels. The operating decision was how quickly a suspect trace reached a reviewable result. Uvik Software’s trace evaluation case study

    Before choosing a tool, assign an owner to the alert and give that person the authority to pause the affected action. Test one incident from the first alert through to the code or workflow change that resolves it. That handoff gives the pilot a clear measure of success.

    Discuss an LLM evaluation and observability audit with Uvik Software

    How useful was this post?

    No votes so far! Be the first to rate this post.

    Share:
    LLM Observability Tools for Production Python Agents - 10

    Need to augment your IT team with top talents?

    Uvik can help!
    Contact
    Uvik Software
    Privacy Overview

    This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

    Get a free project quote!
    Fill out the inquiry form and we'll get back as soon as possible.

      Subscribe to TechTides – Your Biweekly Tech Pulse!
      Join 750+ subscribers who receive 'TechTides' directly on LinkedIn. Curated by Paul Francis, our founder, this newsletter delivers a regular and reliable flow of tech trends, insights, and Uvik updates. Don’t miss out on the next wave of industry knowledge!
      Subscribe on LinkedIn