Summary
Key takeaways
- Measure completed correct responses, p95 latency, errors, and offered traffic together.
- Use production-shaped route mixes, payloads, tenants, dependencies, and cache conditions instead of relying on an empty endpoint.
- Open arrival-rate tests expose growing queues, but only when the load generator sustains scheduled starts.
- In Uvik Software’s single-worker lab, awaited I/O had a 42.57 ms p95 at 40 requests per second, versus 2,969.22 ms with blocking I/O.
- A blocking call inside an async handler can stall unrelated work on the event loop.
- Identify where requests wait: worker or thread, database connection pool, query execution, external dependency, or serialization.
- Size worker counts and database pools together, accounting for replicas, overflow, and rolling deployments.
- Define explicit p95, failure-rate, dropped-iteration, and response-correctness gates before comparing releases.
- Change one measured bottleneck at a time, rerun matched traffic, and verify recovery after dependency degradation.
When this applies
Use this process when an existing FastAPI service has rising tail latency, uncertain concurrency limits, database or dependency waits, or a planned change to worker, pool, or deployment settings. It applies to controlled staging runs and approved low-risk production validation with clear stop conditions.
When this does not apply
A synthetic single-route or timer-only result cannot establish the production capacity of an entire service. Do not use heavy or write-heavy tests directly against production without an approved scope, permitted writes, an abort owner, and a rollback path.
Checklist
- Select a critical route and a route competing for the same dependency.
- Record arrival rates, route mix, burst length, tenants, result sizes, and cache state.
- Use representative authentication, payloads, database shape, and dependency delays.
- Record application revision, replicas, workers, instance size, pool limits, and autoscaling settings.
- Define permitted writes, isolation, maximum load, abort conditions, and recovery ownership.
- Collect baseline completed correct responses, retries, failures, dropped starts, and per-route latency percentiles.
- Use a sustained open arrival rate and verify load-generator scheduling lag and dropped iterations.
- Inspect event-loop lag and blocking calls inside async handlers.
- Measure thread-pool acquisition and activity for synchronous dependencies.
- Measure database connection acquisition separately from SQL execution.
- Inspect query plans, rows processed, lock waits, and result sizes.
- Account for serialization time when large payloads stretch p95.
- Test workers and database pool limits together, including rolling-deployment headroom.
- Set explicit p95, failure, correctness, and dropped-iteration release thresholds.
- Change one bottleneck, rerun the same workload, check dependency degradation and recovery, and preserve raw trial outputs.
Common pitfalls
- Benchmarking only an empty route rather than the real authenticated request path.
- Using a closed-loop generator that reduces offered traffic as the service slows.
- Ignoring missed starts or dropped iterations in the load generator.
- Calling a blocking synchronous utility directly inside an async handler.
- Counting eventual successes while ignoring work that missed the test window.
- Looking only at query execution while connection acquisition creates the delay.
- Adding workers without accounting for the database connections each process can open.
- Using one average dependency delay instead of a representative distribution.
- Changing worker count, connection pools, and caching in the same test.
- Approving a release on HTTP status alone without response correctness and recovery checks.
Tune FastAPI against the request rate, data shape, and dependency latency your service sees in production. Uvik Software starts with a baseline for completed work, p95 latency, and errors. Follow slow requests through the worker, connection pool, database, external calls, and response serialization. Change one suspected bottleneck and rerun the same traffic.
An empty endpoint tells you the fastest path through part of the stack. A typical production handler also authenticates a user, checks a tenant, validates a payload, and calls other systems. FastAPI’s benchmark page is a starting point for understanding framework overhead. Measure your actual endpoint to decide whether to change its implementation or deployment. FastAPI benchmark documentation
Build a workload from the traffic the service must handle
Choose one important route and one that competes for the same dependency. For an order API, include a detail read, a page of recent orders, and an approved write. Use a representative ratio, different tenants, common result sizes, and the data skew that makes large tenants slow. Start with a scrubbed or generated dataset in a production-like environment.
| Workload feature | Record before the first run |
|---|---|
| Traffic shape | Accepted requests per second, route mix, burst length, tenant distribution, and warm or cold caches |
| Request path | Auth and policy, payload size, database query shape, and expected external dependency latency |
| Release boundary | Worker count, application revision, instance size, autoscaling settings, and pool limits |
| Success measure | Completed correct responses, retries, errors, dropped starts, and per-route latency percentiles |
| Guardrails | Test isolation, permitted writes, max load, and the abort and recovery owners |
Measure offered traffic and completed work together. A closed-loop test starts the next iteration when one finishes. As the service slows, the generator can send fewer requests and hide growing queues. An open arrival-rate test schedules starts independently of response time. Check that your load generator kept up, as missed starts also erase traffic the service would have needed to handle. Grafana k6 constant arrival rate · Grafana k6 dropped iterations
Reproduce the difference between blocking and awaited I/O
Uvik Software ran a controlled local test with one Uvicorn worker, no proxy, and two FastAPI routes. Each returned the same response after a fixed 35 ms simulated dependency. One route awaited a timer. The other called a blocking timer inside an async handler. A separate process generated open-loop traffic over localhost. We ran each route three times at 12 and 40 requests per second for six seconds per trial, alternating route order.
import asyncio
import time
from fastapi import FastAPI
app = FastAPI()
@app.get("/awaited")
async def awaited():
await asyncio.sleep(0.035)
return {"ok": True}
@app.get("/blocked")
async def blocked():
time.sleep(0.035)
return {"ok": True}
The benchmark reports the median of three trial p95s. Each trial measured from the scheduled start to the received response, including any queue delay on that path. The p95 scheduling lag inside the generator stayed at or below 2.1 ms in every trial. At 40 scheduled requests per second, the awaited route completed 239 of 240 requests during the six-second window at the median across trials. The blocked route completed 167 and drained the remaining requests after the window.
| Test condition | Median p95 at 12 requests/s | Median p95 at 40 requests/s |
|---|---|---|
| Awaited 35 ms timer | 40.51 ms | 42.57 ms |
| Blocking 35 ms timer inside async handler | 42.68 ms | 2,969.22 ms |
Figure 1. Measured Uvik Software single-worker test. At 40 requests per second the median p95 was 42.57 ms with awaited I/O and 2,969.22 ms with blocking I/O.
All requests eventually returned the expected response. At 40 requests per second, p95 covered nearly three seconds on the blocked route even though the response check passed every time. A single loop spending 35 ms on each sequential block can complete at most roughly 28.6 such operations per second before HTTP and application overhead. The six-second run reveals queue growth. The absolute times apply to this controlled lab. Use the accompanying FastAPI load-test bundle to rerun it, then compare a real dependency under your own traffic.
FastAPI runs normal def path functions and dependencies in an external thread pool. A synchronous utility you call directly from async def still runs in its caller’s context. Starlette documents a default shared capacity of 40 AnyIO thread tokens. Moving slow synchronous work to that pool protects the event loop only while the pool can absorb it. Measure thread wait and dependency limits as well. FastAPI concurrency documentation · Starlette thread pool documentation
Use the lab as a way to verify the shape of your test harness. Its blocked route can report a successful response after the traffic window ends. Count work completed inside the window and again after drain. Then take the same measurement for a handler that calls a real dependency in a staging environment with realistic limits. Retain the raw per-trial outputs and the runtime versions so the next run can distinguish a code change from a load-generator or infrastructure change.
Find the first queue that grows
Instrument the ingress, event loop, thread pool, database pool, query executor, and external calls. For dependencies, capture acquisition and execution separately. If the path spends an extra 250 ms waiting for a connection, a database execution span alone can look healthy. Attribute serialization or response rendering when payload size changes the tail.
| Symptom under load | Signal to check | First change to test |
|---|---|---|
| Loop delay rises while requests wait across unrelated routes | Event-loop lag and synchronous functions in async handlers | Use an async client or isolate blocked work |
| Many routes wait on a synchronous dependency | Thread pool activity and acquisition wait | Bound the slow dependency and replace or move the expensive operation |
| Database connection wait rises before query duration does | Pool occupancy, active sessions, open transactions, and waiter counts | Shorter transactions, fewer requests holding connections, then a pool change |
| Queries slow down and CPU or disk rises on the database | A repeatable query plan, rows processed, lock wait, and result bytes | Reduce query work or concurrency before opening more connections |
| Response p95 grows with rows returned | Serialization time, payload size, and response-model work | Paginate, fetch fewer fields, or return a smaller contract |
Figure 2. Uvik Software’s triage for the first growing queue. Verify the corresponding metric before tuning the worker, pool, or query.
Repeat the test under controlled dependency delays as well as under a stable baseline. Slow a staging-only dependency or use a stub with a fixed distribution. Observe timeouts and retry counts. Bound retries by a request deadline so a degraded dependency does not increase offered work indefinitely. Protect the API when it cannot safely admit more traffic. Uvicorn can limit concurrent tasks or connections and return HTTP 503 over the configured limit. Decide whether those responses meet the client contract and how they will be counted. Uvicorn resource-limit settings
For a production-shaped profile, start with a sampled dependency distribution for each route rather than replaying one average delay. In a modeled 100-call fixture, 90 calls complete in 45 ms, eight in 180 ms, and two in 700 ms. Its nearest-rank dependency p95 is 180 ms. In the application, those slow calls can hold connections or workers and change later requests too. Give the fixture a repeatable ordering rule, record timeouts, and check the p95 of the complete request under both steady and burst traffic.
Size workers and database pools together
More workers can increase parallelism, memory use, and database demand. Suppose four replicas run two workers each, and every worker has one database pool with size 8 and overflow 4. The configuration permits up to 96 application connections from that process group. Compare that exposure with the database limit after reserving capacity for migrations, jobs, and operations. A rolling deployment can briefly run both generations of workers. SQLAlchemy documents that pool size plus max overflow is the maximum simultaneous connection count for the configured pool. SQLAlchemy connection pooling
At a steady rate of 120 requests per second, a measured mean of 180 ms corresponds to about 22 requests in flight by Little’s law. That rough calculation uses the stable average duration. It includes requests that never use a database connection and says little about burst headroom. Use it to dimension the load generator, then check observed concurrency and dependency queues before changing worker or pool limits.
Run three comparisons before resizing a production deployment. In the first, keep the same hardware and load and vary workers. In the second, replay the evening peak on the most promising result and watch database sessions and process memory. In the third, repeat one sustained window with a dependency degradation and check recovery after it clears. Keep route mix, session lifetimes, and test-account data fixed for each matched run. If autoscaling is enabled, record how long new workers take to accept traffic and how many pools open while old workers drain.
Set a release gate the load test can enforce
Define the load constant, the critical route, and the thresholds before running the comparison. The example uses a read-only QA route and sends 120 iterations per second for five minutes. Substitute your own workload, identity, and dataset. Add explicit check and error gates so HTTP success alone does not approve an invalid response.
import http from "k6/http";
import { check } from "k6";
export const options = {
scenarios: {
order_detail: {
executor: "constant-arrival-rate",
rate: 120,
timeUnit: "1s",
duration: "5m",
preAllocatedVUs: 64,
maxVUs: 160,
gracefulStop: "30s",
},
},
thresholds: {
http_req_duration: ["p(95)<500"],
http_req_failed: ["rate<0.005"],
dropped_iterations: ["count==0"],
checks: ["rate>0.995"],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/v1/orders/test-order`);
check(res, {
"order response is correct": (r) => {
if (r.status !== 200) return false;
try {
return r.json("id") === "test-order";
} catch {
return false;
}
},
});
}
The example’s p95 measures request duration from the HTTP metrics in k6. Report queue and lost starts separately. Compare candidate and baseline on the same environment, route mix, and test data. Extend the run through connection churn and any scheduled workload that competes with requests. A synthetic single-ID test is a quick check. A release decision needs representative keys, payloads, and authentication.
Deploy first to a small cohort with a rollback path. Let the team verify application and dependency metrics as well as the customer action. If the service still misses its goal, return to the first queue or resource that deteriorated. Avoid combining a new worker count, a pool change, and a caching rewrite in the same trial.
Record the change as one hypothesis the team can revisit. If independent I/O is stalling the loop, the expected signatures are lower loop lag and shorter latency on otherwise unrelated routes at the same offered rate. Verify that error rates and the work done by downstream dependencies also stayed within budget. If the pressure simply moved to a connection pool, use the new wait evidence to choose the next experiment.
How Uvik Software measured a FastAPI production workflow
In Uvik Software’s published generative AI deployment case, the FastAPI application used monitoring for latency, cost and model routing. The case reports end-to-end production p95 below 4.5 seconds on the standard workflow and below 8 seconds on complex variants. That observation covers the entire workflow, including downstream work. The loopback test above measures a narrower question about one worker and a simulated wait. Uvik Software’s prototype-to-production case study
If you are still choosing a web framework, see Uvik Software’s FastAPI versus Flask comparison. To investigate a live FastAPI service, bring the critical route, an approved traffic profile, and one scrubbed slow trace to Uvik Software’s FastAPI engineers.