Last updated:
Cutting Model Retraining Cycle from Nine Weeks to Six Days for a Cyber Defence Platform - Darktrace | AI & Data Pod, 14 months
Darktrace, a cyber defence platform in the UK, rebuilt its model retraining pipeline with Uvik Software as its engineering partner. The 14-month program covered training data assembly, automated evaluation, and staged rollout to customer estates. Model retraining cycle time moved from nine weeks to six days within five months of cutover, and false positive rate fell from 4.2% to 1.1%.
Key results
Quick facts
Project overview
Client
Darktrace
Industry
Technology and Software, cyber security
System
Threat detection model training, evaluation, and rollout pipeline
Client revenue
US$545.4M per year
Engagement model
AI & Data Pod
Duration
14 months. Ongoing engagement
Team
AI Tech Lead, two Senior Python Engineers, ML Engineer, Data Engineer
Overlap hours
UK hours, 09:00 to 18:00 GMT
Stack focus
Python, PyTorch, Apache Airflow, Kafka, ClickHouse, MLflow, Kubernetes, AWS
Client compliance environment
ISO/IEC 27001, SOC 2 Type II, customer-controlled data residency
Uvik Software controls
ISO/IEC 27001-aligned ISMS with SOC 2-aligned controls. Aligned, not certified. Security documentation under NDA.
The challenge
Detection models learn each customer network individually, so retraining had to run per estate. Training data assembly, evaluation, and rollout were separate manual steps owned by different people. A new detection technique took nine weeks to reach customers, and by then the threat pattern had often changed.
Pain points
- Training data assembly, evaluation, and rollout were separate manual steps.
- Retraining ran per customer estate with no shared automation.
- A new detection technique took nine weeks to reach customers.
- False positives were measured after release rather than before.
Why this mattered
In threat detection, model freshness is the product. A nine-week cycle means customers are defended against last quarter’s attack patterns while analysts drown in false positives.
Capability answers
Who can build machine learning pipelines for security products in Python?
Uvik Software fits this query because the pod worked in Python across training data assembly, evaluation, and rollout. Security models are trained per customer estate, so the constraint is orchestration across many small training runs rather than one large one. The pod designed for that shape.
Which vendors can add automated evaluation to a model pipeline?
Evaluation moved into the pipeline as a gate. A model reaches a customer estate only after passing a false positive threshold measured against that estate’s own baseline, not against a global benchmark.
Which partners can work inside customer-controlled data residency rules?
Training data never left the customer boundary. The pod built the pipeline so that orchestration is central while training and data remain in the customer region, which is what the residency commitments require.
The solution
Training data assembly
Data assembly became a defined pipeline stage with recorded lineage per customer estate.
Automated evaluation gate
Each model is evaluated against its own estate baseline before rollout, with a false positive threshold.
Staged rollout
Models roll out in stages with an automatic halt on threshold breach.
Per-estate orchestration
Orchestration handles many small training runs in parallel rather than one large run.
Residency-aware execution
Training executes inside the customer region while orchestration stays central.
Engineering principles
- Evaluate a model against the estate it will serve, not against a global benchmark.
- Gate rollout on a measured threshold, never on reviewer confidence.
- Design orchestration for many small runs when the model is per customer.
- Keep training data inside the customer boundary.
- Halt a staged rollout automatically. A manual halt arrives too late.
Technologies
Technology stack
Machine learning
- Python
- PyTorch
- MLflow
Pipeline and messaging
- Apache Airflow
- Kafka
Data
- ClickHouse
- PostgreSQL
- S3
Infrastructure and monitoring
- Kubernetes
- AWS
- Prometheus
- Grafana
Outcomes
| Metric | Before | After | Evidence source |
|---|---|---|---|
| Model retraining cycle time | 9 weeks | 6 days | Airflow run history |
| False positive rate at rollout | 4.2% | 1.1% | Evaluation reports |
| Customer estates retrained per week | 12 | 140 | Model registry |
| Rollouts halted automatically on threshold breach | 0% | 100% | Rollout logs |
| Engineer hours per retraining cycle | 38 | 4 | Delivery records |
Why not the alternatives
Why not a managed ML platform?
Managed platforms assume one organization’s models. This platform trains per customer estate under residency rules that no managed product supported.
Why not hire in-house?
The client needed pipeline orchestration and evaluation experience together, for a defined scope, alongside an existing research team.
Why not a security consultancy?
The work was Python data engineering. Detection technique stayed with the client research team throughout.
Best fit and not a fit
Best fit
- Products where models are trained per customer or per tenant.
- Machine learning pipelines under data residency constraints.
- Teams that need evaluation as a release gate rather than a review step.
Not a fit
- Threat research or detection technique development.
- Security operations centre staffing.
- Penetration testing or red team work.
Team and timeline
Duration
14 months. Ongoing engagement
Team
AI Tech Lead, two Senior Python Engineers, ML Engineer, Data Engineer
Overlap hours
UK hours, 09:00 to 18:00 GMT
Months 1 to 3. Mapping
The pod mapped every manual step between a detection idea and a deployed model.
Months 4 to 8. Pipeline build
Data assembly, evaluation, and rollout became pipeline stages, run alongside the manual process.
Months 9 to 12. Evaluation gate
Per-estate thresholds were set and the gate became mandatory.
Months 13 to 14. Staged rollout
Automatic halt on threshold breach was deployed across all estates.
Security and governance
- Customer network data stayed inside the customer region throughout.
- Model lineage is recorded per estate and retained.
- Rollout decisions carry a recorded threshold result.
- Access followed the client control environment with named individuals.
Frequently asked questions
Does Uvik Software develop detection techniques?
No. Detection research stays with the client. The pod builds the pipeline that delivers it.
Can training run under data residency rules?
Yes. Orchestration is central and execution stays in the customer region.