Local Qwen is not a cheaper Opus, OpenFaaS founder reports after two years

Alex Ellis, the founder behind OpenFaaS, SlicerVM, Actuated and Inlets, has published a long account of running local language models inside a small software business. His verdict contradicts a popular claim on social media: a local Qwen model is not a cheaper Claude Opus, it is a different tool that has to be aimed at the right work.

What the hardware cost

The first attempt was a single RTX 3090 in 2023, which proved so hard to use that the experiment was abandoned. Qwen 3.5 was the first generation that produced results worth keeping. Today the work runs on an RTX 6000 Pro Blackwell with 96 GB of VRAM, bought for about $12,000; the same card now sells for roughly $15,400. Two smart plugs measure power at the wall: the RTX 6000 Pro draws around 600 W during inference and stays quiet, while the two 3090s pull closer to 750 W together and are extremely loud.

The benchmark gap is real

Qwen 3.6 27B scores 77.2 on SWE-Bench Verified, against 88.6% for Claude Opus 4.8 — and Ellis argues that benchmarks are a moving target models can be tuned for, since SWE-Bench is built from Python issues while his own code is Go. Frontier models are estimated at 0.5 to 2 trillion parameters, a different class of capacity from anything a single consumer GPU can hold at full fidelity. Quantising a 27B model down to fit one card is also where the worst behaviour appears.

Where the money came back

Two workflows justified the purchase. A tool called diag captures a complete snapshot of a customer's OpenFaaS installation, which is then analysed by a local model inside an ephemeral VM, so no client data reaches a cloud provider. Separately, feeding a telemetry database into the local model revealed a customer who had been under-reporting licences and under-paying by roughly four to five times for more than a year — recovery that alone covered the card.

Loops, hallucinations and operations

The failure mode Ellis keeps hitting is looping: asked to suggest new faas-cli commands, the model repeated the same five suggestions for half an hour while burning 600 W, and in automated code review it invented concurrency problems and race conditions, which ended that experiment. Serving the model is an operations problem in itself — identity, quotas, routing, power monitoring, and two independent llama.cpp instances to preserve full context length. Speculative decoding pushed throughput from a stable 67 to 130–200 tokens per second. His advice is to keep local models on bounded work such as support analysis and end-to-end testing, to follow the tuning notes on the model card, and never to leave them unsupervised on long-horizon tasks. Cost remains a genuine argument too: cloud coding plans sit near $200 a month, and Uber recently capped employee AI spending at $1,500 per developer per tool per month.