Summary
Key takeaways
- A proof of concept tests a bounded task; a pilot tests approved users and operational limits; production requires monitored and recoverable operation.
- Define one supported workflow, eligible users, allowed actions, verified outcomes, escalation, and named owners before hardening a prototype.
- Grade task outcomes and durable system effects on held-out cases, including unsupported requests and tenant access boundaries.
- Test with redacted replay and disabled side effects before moving to approved internal and customer cohorts.
- Calculate cost per verified successful outcome from all eligible attempts, including calls, retries, retrieval, tools, and human review.
- In the illustrative 1,000-request comparison, Candidate A costs $78.96 for 840 verified outcomes, while Candidate B costs $108.00 for 800.
- Apply quality, policy, latency, cost, and recovery thresholds separately; passing most gates does not justify expansion.
- Candidate A misses the illustrative 85% quality threshold at 84%, so the decision is revise despite acceptable cost and latency.
- Expand through replay, internal users, a limited customer cohort, and wider traffic only after the relevant gates pass.
- Version model, prompt, retrieval, permissions, and tool configurations, and rehearse rollback and reconciliation with an on-call operator.
When this applies
Use this process when moving a bounded AI prototype or pilot into a real service with authorized users, measurable business outcomes, external dependencies, and a team responsible for operating it. It fits a service desk workflow that answers from approved knowledge or hands a case to a staffed queue.
When this does not apply
A successful demo alone does not establish production readiness, and the worked thresholds and unit costs are examples rather than universal launch standards. Narrow or stop a pilot if its supported workflow cannot satisfy the business, policy, or recovery constraints.
Checklist
- Choose one bounded workflow with a business-verifiable outcome.
- Record eligible users, request classes, and authorized data sources.
- Define allowed outcomes and restricted actions.
- Name product, support, engineering, and emergency recovery owners.
- Specify a staffed fallback and a durable accepted handoff.
- Build a held-out evaluation set with common, high-impact, and unsupported cases.
- Test tenant boundaries, missing information, and policy restrictions.
- Version model, prompt, retrieval, permissions, and tool schemas.
- Replay redacted inputs with side effects disabled.
- Measure verified outcomes across all eligible attempts.
- Calculate variable cost including retries, tools, retrieval, and human review.
- Set independent task-quality, policy, latency, and cost release gates.
- Rehearse failure containment, fallback, rollback, and in-flight reconciliation.
- Limit the first release to approved internal users and an eligible customer cohort.
- Review evidence and on-call capacity before each traffic increase.
Common pitfalls
- Using demo success as evidence that operators can support real customers.
- Defining success by fluent responses instead of verified outcomes or accepted handoffs.
- Tuning and releasing against the same examples.
- Ignoring tenant-specific access violations hidden by a strong aggregate score.
- Comparing token prices while omitting retries, tools, and paid human review.
- Allowing a pass on latency and cost to override a failed task-quality gate.
- Expanding traffic before a failure and recovery path has been rehearsed.
- Rolling back a deployment without reconciling committed downstream actions.
- Depending on a few reviewers without recording support coverage by hour.
- Changing a model or retrieval configuration without rerunning the release evaluation.
Take an AI proof of concept into production by choosing one workflow, defining an outcome the business can verify, and limiting the first release to an approved cohort. Uvik Software uses a launch decision based on quality, permissions, latency, cost per successful task, and a recovery path. Freeze the scope when a required measure misses its threshold. Expand when the workflow and its operators have proved they can handle real traffic.
A successful demo answers a feasibility question. A production workflow needs a named customer, an available data source, an owner for every consequential action, and a way to keep serving that customer if a dependency fails. Decide what must be true before you invest in hardening the prototype.
Decide what each stage must prove
Pick a workflow with a measurable result. For an internal service desk, the first version might read an account’s approved knowledge, propose a resolution with sources, and route unresolved cases to the right queue. The handler that receives the handoff can confirm when it is accepted. Define what counts as a complete task for each path.
| Stage | Question to answer | Evidence required to exit |
|---|---|---|
| Proof of concept | Can the model and available data solve a bounded task | A representative set of successful, failed, and unsupported inputs |
| Pilot | Can the workflow serve approved users inside known limits | Evaluation on held-out cases, cost per verified outcome, and a rehearsed manual path |
| Production | Can the team operate the workflow through real changes | Monitored SLOs, permissions, on-call coverage, rollback, and controlled expansion |
RAND interviewed 65 experienced data scientists and engineers to study why AI projects fail. The researchers identified recurring issues including misunderstanding the problem, missing data, and inadequate infrastructure. Use those observations to decide which uncertainty to resolve first. The interviews describe failure patterns, and they do not give a launch probability for your pilot. RAND study on the root causes of AI project failure
Replace the demo script with a release contract
Document the authorized user, input, data sources, allowed actions, expected evidence, and escalation policy. Give product and operations the same definition of success. If a handoff is part of the design, record when the receiving queue accepted it. If a refund or access change requires approval, check the application’s record before marking the task complete.
| Release contract field | Example for a service desk pilot |
|---|---|
| Eligible users and requests | Two approved customer teams and their supported ticket categories |
| Data and access | Approved knowledge base and tenant-scoped account records |
| Allowed outcomes | Verified answer or accepted handoff, recorded with evidence |
| Restricted actions | No autonomous billing or entitlement changes |
| Fallback | Route to a staffed queue when evidence is missing or policy blocks the action |
| Owners | Product owns correctness and scope. Support owns escalation. Engineering owns reliability and rollback. |
Choose a production owner before designing the dashboard. Identify who can pause the feature, turn off a tool, change model routing, and approve an emergency runbook. Give the product a safe option to use during a provider outage. For a workflow with consequential actions, test that the archived receipt, request ID, and policy decision remain available after rollback.
The NIST Generative AI Profile describes risks such as confabulation, data privacy, and value chain and component integration. Map those risks to observable controls for this workflow. A confidential data leak needs explicit access and output checks. A provider dependency needs a failure drill and a decision on which data and request classes can be routed to any fallback. NIST Generative AI Profile
Evaluate outcomes and policy before adding traffic
Separate the release set from examples used to tune the prototype. Sample common requests, high-impact cases, unsupported intents, tenant boundaries, missing data, and changed information. Use human-reviewed expected outcomes for important decisions. Grade durable effects such as assigned tickets and correct record changes alongside response evidence, rather than relying on fluency.
Use the same versioned fixtures for changes to the model, prompt, routing, retrieval index, and tools. Compare each candidate against a baseline, review real failures, and record which policy or customer segment each failure affects. Repeat unstable cases and keep the pass conditions explicit. A high average can coexist with repeated failure on the customer that needs the workflow most.
Limit exposure in live testing. Use redacted replay with side effects disabled, then test a controlled cohort with approved access. Check prompt and policy versions on both resumed and new work. A cohort can have a good overall outcome rate while one tenant’s access rule is wrong. Any confirmed unapproved action should block scale-up until its cause and containment are understood.
Keep evaluation failures actionable. For an answer grounded in account documents, review whether the required record was retrievable, retrieved, correctly scoped, and used in the final response. For a handoff, confirm the accepted queue and durable ticket ID. An owner can then fix a missing source, update a policy check, or change routing. Preserve that failure label in the regression set and verify it under the next release candidate.
Model cost per verified outcome
Price the complete workflow. Include initial calls, retries, retrieval, tool usage, and the human review required to finish work. Use all eligible attempts as the population and verified successful outcomes as the denominator. Agreement on what counted as a success matters as much as the token bill.
| Worked comparison for 1,000 eligible requests | Candidate A | Candidate B |
|---|---|---|
| Initial calls | $18.00 at $0.018 per call | $11.00 at $0.011 per call |
| Retries | $2.16 for 120 retries | $2.20 for 200 retries |
| Retrieval and tools | $4.80 | $4.80 |
| Human reviews at $0.60 each | $54.00 for 90 reviews | $90.00 for 150 reviews |
| Total variable workflow cost | $78.96 | $108.00 |
| Verified successful outcomes | 840 | 800 |
| Cost per verified outcome | About $0.094 | $0.135 |
Figure 1. Uvik Software’s worked cost comparison for 1,000 eligible requests per candidate. Candidate B’s cheaper initial calls are outweighed by more human review.
Candidate B’s initial calls cost $7 less across the cohort. Its extra review work costs $36 more. Divide the workflow total by confirmed successful outcomes before routing more traffic to a cheaper model. The sample isolates variable workflow cost. Add hosting commitments, incident support, or platform fees when the launch decision depends on the full margin.
Stanford HAI’s 2025 AI Index reported more than a 280-fold reduction from November 2022 to October 2024 in the inference cost of reaching a GPT-3.5-level reference capability. That historical measure is scoped to the report’s benchmark and period. For a live workflow, rerun the evaluation and cost comparison whenever the model, routing, or pricing changes. Stanford HAI 2025 AI Index
Translate spend into a decision the product owner can make. In the modeled comparison, a price of $0.20 per confirmed successful task leaves just over $0.10 of variable contribution for each Candidate A success before fixed costs. Candidate B leaves $0.065 per success at the same price. Recompute with your actual pricing units and a separate line for work handled entirely by support. Set a budget alert per customer so a change in request mix does not surprise the team at month-end.
Set explicit launch and rollback criteria
Set thresholds from the product’s customer promise and operating model. Treat a guardrail, a completion rate, and a speed target as separate conditions. The following example assumes a pilot with 1,000 eligible attempts and an independently instrumented record of verified outcomes. Report automated response latency separately from the time to accepted human handoff.
| Decision measure | Illustrative gate | Candidate A |
|---|---|---|
| Task quality | At least 85% verified outcomes | 840 of 1,000, or 84%. Revise. |
| Critical policy violations | Zero observed and unresolved in approved coverage | Zero observed. Keep monitoring. |
| Automated response time | p95 at or below 5 seconds for the specified cohort | 4.6 seconds in the sample. Pass. |
| Variable cost | At or below $0.10 per confirmed successful task | About $0.094. Pass. |
| Recovery | Stop new automation, route safely, and reconcile in-flight work in a rehearsal | Rehearsal accepted by on-call. Pass. |
The decision is revise. The team missed the quality gate despite acceptable latency and cost. Review the missed cases and resolve the cause before expanding. An unresolved critical violation should trigger containment and review. If the team cannot define a supported workflow within the customer and policy constraints, stop the launch and narrow the product.
Set a rollback plan across the whole runtime. A versioned deployment should record the model identifier, system prompt, retrieval settings, permissions, and tool schema. Test disabling actions independently from reading or responding. Reconcile tasks already committed to source systems, and honor the customer-facing status when redirecting new work. Google Cloud’s MLOps guidance stresses continuous validation and deployment. Use it to keep evaluation, rollout, and monitoring part of every production change. Google Cloud MLOps guidance
Rehearse a failure with a timer and a named operator. For the service desk, turn off an approved test dependency during a staging run. Confirm the workflow stops retrying before its deadline, sends only allowed data to fallback, and leaves one durable handoff record. Let support receive the result through its normal queue. Record time to containment, the number of accepted tasks still unresolved, and whether recovery created duplicates. Review the run with product and operations before signing the gate.
Move from internal access to a limited customer cohort
Start with replay, then an internal cohort, then one customer or a narrowly scoped share of eligible traffic. Review the same outcome and cost definitions at each expansion. Wait long enough to cover an important workflow cycle and have enough observations for the risk being tested. A low-volume workflow may need an extended cohort period before percentages settle.
Figure 2. Uvik Software’s release sequence. Keep replay side effects disabled, review the first eligible cohort, and resolve any failed gate before expanding.
Before each increase, review evidence with a named decision maker and an on-call operator. Confirm the fallback queue has capacity and the estimates account for support. If a release flag stops new tasks, also check worker queues and pending operations. Keep ownership of downstream reconciliation with the team that can inspect the durable record.
At every expansion, attach the decision to a specific model configuration and retrieval state. Record the release owner and the person who can restore it. Check post-release feedback for wrong-document access, missed escalations, a change in review load, and users working around the feature. Rollouts that depend on a few experienced reviewers should record coverage by hour and plan for what happens outside that window.
What Uvik Software changed to ship a generative AI prototype
Uvik Software’s published case reports a validated production workflow 14 weeks from kickoff and a first paid customer cohort at week 16. The team introduced an evaluation harness, prompt versioning, fallbacks, and per-query cost and latency telemetry. The case reports production p95 under 4.5 seconds on the standard workflow. Those figures describe the scope and workload in that engagement. Uvik Software’s generative AI production case study
Start with Uvik Software’s production AI agent development guide for architecture and tool design. For an existing prototype or stalled pilot, bring one eligible workflow, a scrubbed failure, and the proposed release gate to Uvik Software’s AI application rescue team.