Optimizing nextflow pipelines for cloud scalability
Quotient Therapeutics, a company working with deeply sequenced whole-genome sequencing (WGS) data, faced growing computational demands. As the number of processed samples increased, they needed to optimize their Nextflow-based pipelines for better efficiency, reliability, and cost-effectiveness.
Challenge
The client’s pipeline was struggling to keep up with increasing workloads, resulting in longer processing times and rising compute costs. Ensuring consistent reliability was also a key priority to maintain high-quality data outputs.
Approach
A multi-tiered optimization strategy was implemented:
- Comprehensive review of pipeline code, tools, and environments to identify optimization opportunities.
- Integration of nf-test for automated validation and CI/CD workflows.
- Optimization across three levels: bioinformatics, software engineering, and cloud DevOps.
Results
- Reliability improved through automated integration tests before every deployment.
- Processing cost reduced by 64%.
- Execution time reduced by 63%, enabling sample processing within the target 24-hour window
Through strategic optimization of bioinformatics pipelines in an AWS Batch environment, Quotient Therapeutics achieved significant cost savings and efficiency gains. The improvements ensured the scalability and reliability of their workflow, preparing them for continued growth.
Discover more: Our FOG 2025 presentation on this project
For a deeper dive into this project and similar successful case studies, watch our presentation.
or more insights from this key event, explore our highlights from the Festival of Genomics & Biodata
Our team recently presented on these advancements at the Nextflow Summit 2024, showcasing how we enhance open science in Big Pharma.
