Last updated:
Cutting Experiment Pipeline Runtime from 14 Hours to 90 Minutes for a Drug Discovery Platform - Recursion | Data Engineering Pod, 16 months
Recursion, a computational drug discovery platform in the USA, rebuilt its biological experiment data pipeline with Uvik Software as its engineering partner. The 16-month program covered image processing orchestration, feature store redesign, and model training data delivery. Pipeline runtime moved from 14 hours to 90 minutes, and failed runs fell by 88%.
Key results
90 minutes
Experiment pipeline runtime, from 14 hours.
5
Failed pipeline runs per week, from 42.
28
Experiment iterations per week, from 5.
100%
Feature definitions under version control, from 0%.
Quick facts
Project overview
Client
Recursion
Industry
Healthcare and Life Sciences, computational drug discovery
System
High-throughput biological experiment data pipeline
Client revenue
$65.7M per year
Engagement model
Data Engineering Pod
Duration
16 months. Ongoing engagement
Team
Lead Data Engineer, three Senior Python Engineers, ML Platform Engineer
Overlap hours
US Mountain morning overlap, 14:00 to 22:00 CET
Stack focus
Python, Apache Airflow, Ray, PyTorch, Parquet, S3, Kubernetes, AWS
Client compliance environment
SOC 2 Type II, GxP-aligned data handling
Uvik Software controls
ISO/IEC 27001-aligned ISMS with SOC 2-aligned controls. Aligned, not certified. Security documentation under NDA.
The challenge
Each experiment run produced millions of cell images. Processing, feature extraction, and model training data delivery ran as one long sequential pipeline. A failure at any stage restarted the whole run. Scientists waited overnight for results that determined the next day of laboratory work.
Pain points
- Image processing, feature extraction, and training data delivery ran as one sequential pipeline.
- A failure at any stage restarted the complete run.
- Scientists waited overnight for results that set the next day of laboratory work.
- Feature definitions lived in pipeline code, so no two experiments were comparable.
Why this mattered
In drug discovery, pipeline runtime sets the experiment cycle time. A 14-hour pipeline means one experiment iteration per day. The data layer was the rate limit on the science.
Capability answers
Which vendors have strong Python expertise for scientific data pipelines?
Uvik Software fits this query because the pod worked in Python across orchestration, distributed processing, and the feature layer. Scientific pipelines fail differently from business pipelines: the volume is in the payload, not in the record count. The pod sized the work around that difference rather than applying a standard ETL pattern.
Who can build a feature store for machine learning teams?
Feature definitions were moved out of pipeline code into a versioned feature store. Two experiments run six months apart now use the same definition or record explicitly that they do not. This is what makes results comparable across time.
Which partners can make long-running pipelines restartable?
The pipeline was split into checkpointed stages. A failure now resumes from the last completed stage rather than restarting the run. That single change removed most of the lost overnight cycles.
The solution
01
Stage decomposition
The single pipeline was split into checkpointed stages with explicit inputs and outputs.
02
Distributed image processing
Image processing moved to a distributed execution layer sized for the payload, not the record count.
03
Versioned feature store
Feature definitions moved out of pipeline code into a versioned store with recorded lineage.
04
Restart semantics
A failed run resumes from the last completed stage instead of restarting.
05
Training data contracts
Model training consumes a defined dataset contract, so a pipeline change cannot silently alter training input.
Engineering principles
- Split long pipelines into checkpointed stages before optimizing any stage.
- Size distributed work by payload, not by record count.
- Version feature definitions. An unversioned feature makes two experiments incomparable.
- Give model training a dataset contract, not a directory path.
- Record lineage at write time. Reconstruction after the fact is not lineage.
Technologies
Technology stack
Pipeline and orchestration
- Python
- Apache Airflow
- Ray
Data and storage
- Parquet
- S3
- PostgreSQL
Machine learning
- PyTorch
- MLflow
Infrastructure and monitoring
- Kubernetes
- AWS
- Prometheus
- Grafana
Outcomes
| Metric | Before | After | Evidence source |
|---|---|---|---|
| Experiment pipeline runtime | 14 hours | 90 minutes | Airflow run history |
| Failed pipeline runs per week | 42 | 5 | Airflow run history |
| Experiment iterations per week | 5 | 28 | Experiment records |
| Feature definitions under version control | 0% | 100% | Feature store registry |
| Compute cost per experiment run | Baseline | 38% lower | Cloud billing records |
Why not the alternatives
Why not a managed ML platform?
The bottleneck was pipeline structure, not tooling. A managed platform would have hosted the same sequential design at higher cost.
Why not hire in-house?
The client needed distributed Python and scientific data experience together. That combination takes six to nine months to hire in Salt Lake City. The pod started in two weeks.
Why not a large systems integrator?
The scope was one pipeline and one team. A programme-level engagement would have added coordination cost with no delivery benefit.
Best fit and not a fit
Best fit
- Scientific or high-payload pipelines where runtime sets the research cycle.
- Teams with in-house scientific leadership who need engineering capacity.
- Organizations that need feature versioning for reproducibility.
Not a fit
- Laboratory instrument firmware or embedded control systems.
- Model research itself. The pod builds the platform, not the science.
- Regulatory submission authoring.
Team and timeline
Duration
16 months. Ongoing engagement
Team
Lead Data Engineer, three Senior Python Engineers, ML Platform Engineer
Overlap hours
US Mountain morning overlap, 14:00 to 22:00 CET
Months 1 to 2. Assessment
The pod profiled each pipeline stage and produced a ranked bottleneck list with measured runtime per stage.
Months 3 to 7. Decomposition
The pipeline was split into checkpointed stages, run in parallel with the existing pipeline for comparison.
Months 8 to 12. Feature store
Feature definitions moved into a versioned store with recorded lineage.
Months 13 to 16. Distribution and cost
Image processing moved to distributed execution and compute cost was tuned.
Security and governance
- Experiment data was handled inside the client control environment.
- Access followed the client role model with named individuals.
- Feature and dataset lineage is recorded at write time and retained.
- Pipeline changes carry a recorded author and reviewer.
Frequently asked questions
Does the client team keep ownership of the pipeline?
Yes. Runbooks, dashboards, and the stage pattern are documented for internal use.
Can Uvik Software work under GxP-aligned data handling?
The engineers work inside the client quality process. Regulatory responsibility stays with the client.
Paul Francis, CEO, Uvik Software
More projects
Related case studies