OpenAI field report: coding agents hit 60x speedups but can’t verify the science
OpenAI and academic partners published a field report covering eight case studies in which research teams used coding agents, primarily Codex and Claude Code, to modernize scientific software. The projects spanned basic build maintenance, language migrations, and full rewrites in Rust. The headline numbers are real. So are the failure modes.
What the agents actually did
The range of work is wide. On the simple end, GPT-5.5 modernized the build and installation process for cyvcf2, a Python library for reading genetic data. On the complex end, Claude Code and Codex took turns acting as developer and reviewer to port roughly 10,000 lines of MHCflurry from TensorFlow to PyTorch.
The most ambitious project, rustar-aligner, rebuilt STAR from scratch in Rust. STAR is a genome sequencing alignment tool with more than 20,000 lines of C and C++. It is no longer actively maintained but still runs inside many research pipelines. Tested against 10,000 short sequencing reads from yeast cells, the Rust rewrite agreed with the original STAR output in 99.815 percent of single-end cases and 99.883 percent of paired-end cases.
The benchmark numbers
RustQC delivered the largest speed gain by consolidating 15 separate quality-control tools into one program. Runtime on a large dataset dropped from 15 hours and 34 minutes to 14 minutes and 54 seconds, a speedup of more than 60 times.
HelixForge replaced a CPU-based tool for generating synthetic genomic data with a GPU-native version. On a test using one donor and a 10-million-base-pair genome section, it completed the full pipeline 59.6 times faster than BamSurgeon. The core compute step alone ran 98.6 times faster.
hifiasm, a genome assembler, cut runtime on real human genome data by nearly 15 percent after GPT-5.5 found targeted optimizations against a pre-built test harness. HI.SIM, a genetic simulation library, reduced runtime by about 31 percent across two optimization passes without changing its output.
The failure modes researchers actually hit
Every participant who commented on accuracy flagged the same problem: agents present wrong code with the same confidence as correct code.
“With coding agents, it’s quite easy to go fast; for now, to go far in science, there’s still a need for expert guidance, understanding, taste, and care.” — Brent Pedersen, cyvcf2 developer
“Eloquent, convincing, and confidently wrong in ways that are easy to miss.” — Philip Ewels, RustQC lead
Ewels never allowed the models to evaluate their own output. He built an independent test harness instead.
The bayesm Rust rewrite ran between 2 and 20 times faster than the original R version, but early builds contained two bugs that were invisible from normal output. In one method, the agent inverted a key control parameter, using the reciprocal of the intended value. Researchers only caught it by running a calibration test against thousands of synthetic datasets with known results. A separate method called HART produced plausible output overall but still contained needlessly expensive calculations and an incorrectly scaled correction factor.
An earlier attempt to port MHCflurry to PyTorch had failed in early 2025. Developer Sergey Feldman attributed that failure to the model generation available at the time, not to the tooling itself.
The division of labor that worked
Across all eight projects, the pattern was consistent. Humans defined the goals, set the success criteria, and built the validation methods. Agents handled implementation. The report’s authors describe the main bottleneck as having shifted away from writing code and toward validation, scientific review, and clear ownership of maintenance.
The maintenance problem cheap rewrites create
The report includes rough cost estimates. If coding agents could resolve between one-quarter and one-half of all installation problems affecting research software, the saved research time across 100 packages would be worth between $600,000 and nearly $5 million. For NumPy alone, the authors estimate agents could save about 650 hours of maintenance work per year.
But low-cost rewrites also fragment user communities and spread already thin maintainer time further. Teams handled ownership differently. Some merged changes back into the original projects. The rustar-aligner moved to the scverse research consortium since STAR had no active maintainer. The FastQC author declined to replace the original tool with its Rust rewrite. Instead, the team ported the performance improvements back into the original Java codebase, achieving the same threefold speedup.
The report is part of OpenAI’s broader science push. In April, the company introduced GPT-Rosalind, a reasoning model for life sciences, and released a free life sciences plugin for Codex that connects to more than 50 public databases and biology tools.
The operator read
For developers using AI coding tools on production systems, the bayesm case is the one to study. The bug was invisible until someone ran a purpose-built calibration suite against thousands of known-output datasets. Plausible results do not mean correct results. Independent validation infrastructure is not optional if the output feeds anything that matters.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
