There's no reason for software to be slow anymore — Dan Luu on the new economics of optimization

For years, slow software was explained by the economics of expertise: making a program fast required engineers with a rare skill set, so most projects shipped without it. Engineer Dan Luu argues in a new post that this trade-off has broken down — coding agents have cut the cost of optimization by orders of magnitude.

The argument

The piece opens with a viral claim that critics of the slow, bloated code written by LLMs will "eat crow" once everything is rewritten in super-optimized assembly. Luu doesn't go that far: we are not heading toward hand-written assembly, but optimizations that used to require a person or a team with rare skills can now be done by anyone who can type a few sentences. He estimates the human-time cost has dropped "frequently 1000x / 10000x / 1000000x", with roughly a 1000x fall in dollar cost compared with the Bing engineer who once wrote the compilers and JITs inside its search index.

Concrete numbers: regex and ripgrep

The post backs the claim up with experiments on FRE, a regex engine Luu built by letting an agent loop for a month against the rebar benchmark suite. The overfitting was so strong that the agent had to be told a holdout benchmark existed before its optimizations generalized. One experiment had ripgrep compile patterns to native code in a background thread and switch over when compilation finished: 2x–4x faster on a few long, simple queries, and about a 7% speedup on representative holdout queries where the native path applies — at the cost of slower short queries, since a thread is lost to compilation.

Software fitted to a single workload

Marc Brooker compared the trend to FFTW and demoscene tricks — code shaped for one particular problem rather than a class of problems — and predicted more "dynamic custom software" built against a specific workload. Michael Malis, who works on pgrust, makes a related point: the meme that "code was never the hard part" only holds in some domains. JIT compilers are rare because implementing one used to be too expensive to be worthwhile, and databases were historically the hardest software to build. Luu notes he worked on BitFunnel, Bing's search index that won a best paper award at SIGIR; the Bing version contains multiple JIT compilers — a scale of optimization that is now genuinely "I could do that in a weekend".

What still doesn't change

Luu is careful about the limits. Optimizations that change results need a way to measure whether the change is worth it, and current state-of-the-art models are poor at experimental design, so the evaluation framework still has to come from a human. Overfitting remains a risk: tuning for one workload breaks when the workload changes. And an agent told simply to "optimize" will do things that are wrong and have to be caught. Still, his conclusion is that decent performance is no longer a specialized skill — it is a reasonable expectation for anyone using coding agents seriously.