Menu

FastAPI Performance Tuning With Production Load Tests

FastAPI Performance Tuning With Production Load Tests - 9
Paul Francis

Table of content

    Summary

    Key takeaways

    • Measure completed correct responses, p95 latency, errors, and offered traffic together.
    • Use production-shaped route mixes, payloads, tenants, dependencies, and cache conditions instead of relying on an empty endpoint.
    • Open arrival-rate tests expose growing queues, but only when the load generator sustains scheduled starts.
    • In Uvik Software’s single-worker lab, awaited I/O had a 42.57 ms p95 at 40 requests per second, versus 2,969.22 ms with blocking I/O.
    • A blocking call inside an async handler can stall unrelated work on the event loop.
    • Identify where requests wait: worker or thread, database connection pool, query execution, external dependency, or serialization.
    • Size worker counts and database pools together, accounting for replicas, overflow, and rolling deployments.
    • Define explicit p95, failure-rate, dropped-iteration, and response-correctness gates before comparing releases.
    • Change one measured bottleneck at a time, rerun matched traffic, and verify recovery after dependency degradation.

    When this applies

    Use this process when an existing FastAPI service has rising tail latency, uncertain concurrency limits, database or dependency waits, or a planned change to worker, pool, or deployment settings. It applies to controlled staging runs and approved low-risk production validation with clear stop conditions.

    When this does not apply

    A synthetic single-route or timer-only result cannot establish the production capacity of an entire service. Do not use heavy or write-heavy tests directly against production without an approved scope, permitted writes, an abort owner, and a rollback path.

    Checklist

    1. Select a critical route and a route competing for the same dependency.
    2. Record arrival rates, route mix, burst length, tenants, result sizes, and cache state.
    3. Use representative authentication, payloads, database shape, and dependency delays.
    4. Record application revision, replicas, workers, instance size, pool limits, and autoscaling settings.
    5. Define permitted writes, isolation, maximum load, abort conditions, and recovery ownership.
    6. Collect baseline completed correct responses, retries, failures, dropped starts, and per-route latency percentiles.
    7. Use a sustained open arrival rate and verify load-generator scheduling lag and dropped iterations.
    8. Inspect event-loop lag and blocking calls inside async handlers.
    9. Measure thread-pool acquisition and activity for synchronous dependencies.
    10. Measure database connection acquisition separately from SQL execution.
    11. Inspect query plans, rows processed, lock waits, and result sizes.
    12. Account for serialization time when large payloads stretch p95.
    13. Test workers and database pool limits together, including rolling-deployment headroom.
    14. Set explicit p95, failure, correctness, and dropped-iteration release thresholds.
    15. Change one bottleneck, rerun the same workload, check dependency degradation and recovery, and preserve raw trial outputs.

    Common pitfalls

    • Benchmarking only an empty route rather than the real authenticated request path.
    • Using a closed-loop generator that reduces offered traffic as the service slows.
    • Ignoring missed starts or dropped iterations in the load generator.
    • Calling a blocking synchronous utility directly inside an async handler.
    • Counting eventual successes while ignoring work that missed the test window.
    • Looking only at query execution while connection acquisition creates the delay.
    • Adding workers without accounting for the database connections each process can open.
    • Using one average dependency delay instead of a representative distribution.
    • Changing worker count, connection pools, and caching in the same test.
    • Approving a release on HTTP status alone without response correctness and recovery checks.

    Tune FastAPI against the request rate, data shape, and dependency latency your service sees in production. Uvik Software starts with a baseline for completed work, p95 latency, and errors. Follow slow requests through the worker, connection pool, database, external calls, and response serialization. Change one suspected bottleneck and rerun the same traffic.

    An empty endpoint tells you the fastest path through part of the stack. A typical production handler also authenticates a user, checks a tenant, validates a payload, and calls other systems. FastAPI’s benchmark page is a starting point for understanding framework overhead. Measure your actual endpoint to decide whether to change its implementation or deployment. FastAPI benchmark documentation

    Build a workload from the traffic the service must handle

    Choose one important route and one that competes for the same dependency. For an order API, include a detail read, a page of recent orders, and an approved write. Use a representative ratio, different tenants, common result sizes, and the data skew that makes large tenants slow. Start with a scrubbed or generated dataset in a production-like environment.

    Workload feature Record before the first run
    Traffic shape Accepted requests per second, route mix, burst length, tenant distribution, and warm or cold caches
    Request path Auth and policy, payload size, database query shape, and expected external dependency latency
    Release boundary Worker count, application revision, instance size, autoscaling settings, and pool limits
    Success measure Completed correct responses, retries, errors, dropped starts, and per-route latency percentiles
    Guardrails Test isolation, permitted writes, max load, and the abort and recovery owners

    Measure offered traffic and completed work together. A closed-loop test starts the next iteration when one finishes. As the service slows, the generator can send fewer requests and hide growing queues. An open arrival-rate test schedules starts independently of response time. Check that your load generator kept up, as missed starts also erase traffic the service would have needed to handle. Grafana k6 constant arrival rate · Grafana k6 dropped iterations

    Reproduce the difference between blocking and awaited I/O

    Uvik Software ran a controlled local test with one Uvicorn worker, no proxy, and two FastAPI routes. Each returned the same response after a fixed 35 ms simulated dependency. One route awaited a timer. The other called a blocking timer inside an async handler. A separate process generated open-loop traffic over localhost. We ran each route three times at 12 and 40 requests per second for six seconds per trial, alternating route order.

    import asyncio
    import time
    from fastapi import FastAPI
    
    app = FastAPI()
    
    @app.get("/awaited")
    async def awaited():
        await asyncio.sleep(0.035)
        return {"ok": True}
    
    @app.get("/blocked")
    async def blocked():
        time.sleep(0.035)
        return {"ok": True}

    The benchmark reports the median of three trial p95s. Each trial measured from the scheduled start to the received response, including any queue delay on that path. The p95 scheduling lag inside the generator stayed at or below 2.1 ms in every trial. At 40 scheduled requests per second, the awaited route completed 239 of 240 requests during the six-second window at the median across trials. The blocked route completed 167 and drained the remaining requests after the window.

    Test condition Median p95 at 12 requests/s Median p95 at 40 requests/s
    Awaited 35 ms timer 40.51 ms 42.57 ms
    Blocking 35 ms timer inside async handler 42.68 ms 2,969.22 ms

    On a logarithmic latency axis, awaited and blocked routes have p95 values around 41 and 43 milliseconds at 12 requests per second. At 40 requests per second they measured 42.57 and 2,969.22 milliseconds respectively.

    Figure 1. Measured Uvik Software single-worker test. At 40 requests per second the median p95 was 42.57 ms with awaited I/O and 2,969.22 ms with blocking I/O.

    All requests eventually returned the expected response. At 40 requests per second, p95 covered nearly three seconds on the blocked route even though the response check passed every time. A single loop spending 35 ms on each sequential block can complete at most roughly 28.6 such operations per second before HTTP and application overhead. The six-second run reveals queue growth. The absolute times apply to this controlled lab. Use the accompanying FastAPI load-test bundle to rerun it, then compare a real dependency under your own traffic.

    FastAPI runs normal def path functions and dependencies in an external thread pool. A synchronous utility you call directly from async def still runs in its caller’s context. Starlette documents a default shared capacity of 40 AnyIO thread tokens. Moving slow synchronous work to that pool protects the event loop only while the pool can absorb it. Measure thread wait and dependency limits as well. FastAPI concurrency documentation · Starlette thread pool documentation

    Use the lab as a way to verify the shape of your test harness. Its blocked route can report a successful response after the traffic window ends. Count work completed inside the window and again after drain. Then take the same measurement for a handler that calls a real dependency in a staging environment with realistic limits. Retain the raw per-trial outputs and the runtime versions so the next run can distinguish a code change from a load-generator or infrastructure change.

    Find the first queue that grows

    Instrument the ingress, event loop, thread pool, database pool, query executor, and external calls. For dependencies, capture acquisition and execution separately. If the path spends an extra 250 ms waiting for a connection, a database execution span alone can look healthy. Attribute serialization or response rendering when payload size changes the tail.

    Symptom under load Signal to check First change to test
    Loop delay rises while requests wait across unrelated routes Event-loop lag and synchronous functions in async handlers Use an async client or isolate blocked work
    Many routes wait on a synchronous dependency Thread pool activity and acquisition wait Bound the slow dependency and replace or move the expensive operation
    Database connection wait rises before query duration does Pool occupancy, active sessions, open transactions, and waiter counts Shorter transactions, fewer requests holding connections, then a pool change
    Queries slow down and CPU or disk rises on the database A repeatable query plan, rows processed, lock wait, and result bytes Reduce query work or concurrency before opening more connections
    Response p95 grows with rows returned Serialization time, payload size, and response-model work Paginate, fetch fewer fields, or return a smaller contract

    Three diagnostic paths match loop lag to unawaited blocking, connection acquisition wait to pool pressure, and execution time to query or database performance.

    Figure 2. Uvik Software’s triage for the first growing queue. Verify the corresponding metric before tuning the worker, pool, or query.

    Repeat the test under controlled dependency delays as well as under a stable baseline. Slow a staging-only dependency or use a stub with a fixed distribution. Observe timeouts and retry counts. Bound retries by a request deadline so a degraded dependency does not increase offered work indefinitely. Protect the API when it cannot safely admit more traffic. Uvicorn can limit concurrent tasks or connections and return HTTP 503 over the configured limit. Decide whether those responses meet the client contract and how they will be counted. Uvicorn resource-limit settings

    For a production-shaped profile, start with a sampled dependency distribution for each route rather than replaying one average delay. In a modeled 100-call fixture, 90 calls complete in 45 ms, eight in 180 ms, and two in 700 ms. Its nearest-rank dependency p95 is 180 ms. In the application, those slow calls can hold connections or workers and change later requests too. Give the fixture a repeatable ordering rule, record timeouts, and check the p95 of the complete request under both steady and burst traffic.

    Size workers and database pools together

    More workers can increase parallelism, memory use, and database demand. Suppose four replicas run two workers each, and every worker has one database pool with size 8 and overflow 4. The configuration permits up to 96 application connections from that process group. Compare that exposure with the database limit after reserving capacity for migrations, jobs, and operations. A rolling deployment can briefly run both generations of workers. SQLAlchemy documents that pool size plus max overflow is the maximum simultaneous connection count for the configured pool. SQLAlchemy connection pooling

    At a steady rate of 120 requests per second, a measured mean of 180 ms corresponds to about 22 requests in flight by Little’s law. That rough calculation uses the stable average duration. It includes requests that never use a database connection and says little about burst headroom. Use it to dimension the load generator, then check observed concurrency and dependency queues before changing worker or pool limits.

    Run three comparisons before resizing a production deployment. In the first, keep the same hardware and load and vary workers. In the second, replay the evening peak on the most promising result and watch database sessions and process memory. In the third, repeat one sustained window with a dependency degradation and check recovery after it clears. Keep route mix, session lifetimes, and test-account data fixed for each matched run. If autoscaling is enabled, record how long new workers take to accept traffic and how many pools open while old workers drain.

    Set a release gate the load test can enforce

    Define the load constant, the critical route, and the thresholds before running the comparison. The example uses a read-only QA route and sends 120 iterations per second for five minutes. Substitute your own workload, identity, and dataset. Add explicit check and error gates so HTTP success alone does not approve an invalid response.

    import http from "k6/http";
    import { check } from "k6";
    
    export const options = {
      scenarios: {
        order_detail: {
          executor: "constant-arrival-rate",
          rate: 120,
          timeUnit: "1s",
          duration: "5m",
          preAllocatedVUs: 64,
          maxVUs: 160,
          gracefulStop: "30s",
        },
      },
      thresholds: {
        http_req_duration: ["p(95)<500"],
        http_req_failed: ["rate<0.005"],
        dropped_iterations: ["count==0"],
        checks: ["rate>0.995"],
      },
    };
    
    export default function () {
      const res = http.get(`${__ENV.BASE_URL}/v1/orders/test-order`);
      check(res, {
        "order response is correct": (r) => {
          if (r.status !== 200) return false;
          try {
            return r.json("id") === "test-order";
          } catch {
            return false;
          }
        },
      });
    }

    The example’s p95 measures request duration from the HTTP metrics in k6. Report queue and lost starts separately. Compare candidate and baseline on the same environment, route mix, and test data. Extend the run through connection churn and any scheduled workload that competes with requests. A synthetic single-ID test is a quick check. A release decision needs representative keys, payloads, and authentication.

    Deploy first to a small cohort with a rollback path. Let the team verify application and dependency metrics as well as the customer action. If the service still misses its goal, return to the first queue or resource that deteriorated. Avoid combining a new worker count, a pool change, and a caching rewrite in the same trial.

    Record the change as one hypothesis the team can revisit. If independent I/O is stalling the loop, the expected signatures are lower loop lag and shorter latency on otherwise unrelated routes at the same offered rate. Verify that error rates and the work done by downstream dependencies also stayed within budget. If the pressure simply moved to a connection pool, use the new wait evidence to choose the next experiment.

    How Uvik Software measured a FastAPI production workflow

    In Uvik Software’s published generative AI deployment case, the FastAPI application used monitoring for latency, cost and model routing. The case reports end-to-end production p95 below 4.5 seconds on the standard workflow and below 8 seconds on complex variants. That observation covers the entire workflow, including downstream work. The loopback test above measures a narrower question about one worker and a simulated wait. Uvik Software’s prototype-to-production case study

    If you are still choosing a web framework, see Uvik Software’s FastAPI versus Flask comparison. To investigate a live FastAPI service, bring the critical route, an approved traffic profile, and one scrubbed slow trace to Uvik Software’s FastAPI engineers.

    Frequently asked questions

    How should I load test FastAPI

    Replay a representative arrival rate and route mix with realistic dependencies. Track correct responses, errors, dropped starts, and latency by route. Keep one variable different between the baseline and the candidate.

    Does async def make a FastAPI endpoint faster

    It allows a suitable I/O operation to yield control while it waits. A blocking call inside the handler stalls the event loop. Check the libraries actually used in the handler and verify the effect under load.

    How many Uvicorn workers should I use

    Start with an allocation that reflects CPU, memory, connection limits, and deployment replicas. Test splits under the same arrival rate. Add workers only when the end-to-end results improve without excessive demand on dependencies.

    Should I increase the database connection pool

    Investigate connection wait, transaction duration, concurrent sessions, and database saturation first. A larger pool can improve handling of brief contention or move the bottleneck into the database.

    What should a good FastAPI p95 be

    Pick it from the product’s service expectation and the route’s job. Compare parameters such as dependency latency, data volume, and the response needed before applying another endpoint’s target.

    What is a dropped iteration in k6

    It is a scheduled iteration that did not start. In an arrival-rate test, investigate whether the generator lacked available virtual users, or whether a slowdown consumed the headroom needed to sustain those arrivals.

    Can I test against production

    Use a production-like test environment for heavy or write-heavy runs. Validate with approved low-risk production traffic only when the rollout owner has set the scope, a stop condition, and a rollback path.

    How useful was this post?

    No votes so far! Be the first to rate this post.

    Share:
    FastAPI Performance Tuning With Production Load Tests - 12

    Need to augment your IT team with top talents?

    Uvik can help!
    Contact
    Uvik Software
    Privacy Overview

    This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

    Get a free project quote!
    Fill out the inquiry form and we'll get back as soon as possible.

      Subscribe to TechTides – Your Biweekly Tech Pulse!
      Join 750+ subscribers who receive 'TechTides' directly on LinkedIn. Curated by Paul Francis, our founder, this newsletter delivers a regular and reliable flow of tech trends, insights, and Uvik updates. Don’t miss out on the next wave of industry knowledge!
      Subscribe on LinkedIn