So you want to use OpenRouter: lessons from 18 million messages

The model is not the provider

Mo Moustafa, who runs Olly, an AI assistant that lives in iMessage, has published a field guide to OpenRouter. Olly has transacted over 18 million messages, roughly a third of them on open models routed through the service — enough volume, he writes, to hit every edge case at least once. His vocabulary is blunt: the model is the weights; the provider is whoever OpenRouter routes you to, hosting those weights on its own GPUs, at its chosen precision, with its own parsers and therefore its own list of bugs. Asking for deepseek/deepseek-v4-flash gets you one of roughly 20 companies. "The same model on paper, but very different models in real life."

Benchmarks, vision and the effort knob

OpenRouter publishes per-provider benchmarks for identical weights. On the board for DeepSeek V4 Flash 0731 dated September 7, first-party DeepSeek scored 90% on GPQA Diamond and 81% on TAU-Bench Airline, a tool-calling task, while DigitalOcean scored 75% and 58%. Most hosts cluster five to seven points below first-party on tool calling, and four fall off a cliff on knowledge; in July, Fireworks scored 46% on TAU, a 30-point gap. Vision is just as uneven: sending three tiny images to every host of two open vision models, Moustafa found DeepInfra's Qwen endpoint read a K as an R, called red blue and described the word "umbrella" as "funny", while four other hosts got everything right; Venice and Together did not see the MiniMax images at all, yet still returned 200 OK. And reasoning.effort is accepted everywhere but honoured unevenly: most providers respect it, while digitalocean, gmi-cloud, mancer and venice largely ignore it.

Quantization, parsing and hollow answers

Filtering by declared precision does not buy quality. After a month of running the fp8 filter on DeepSeek, fp4 hosts landed in the middle of the fp8 pack, and the three worst GPQA scores were one fp4 host, one fp8 host and one that declares nothing; on GLM, the best scorer on both boards declares nothing at all. Precision is a bad proxy for quality, and a hard filter shrinks the pool OpenRouter can fall back to. Parser misses surface as raw markup in the reply, so callers end up parsing on their own side. Failures also hide behind success codes: a reasoning model can return HTTP 200 with null content and finish_reason "stop" after 345 completion tokens, and some endpoints return no usage object at all. In July, StreamLake accounted for about 20% of his DeepSeek traffic and 92% of his empty completions.

History rules, testing and pinning

The contract is per provider, not per model: SiliconFlow returns a 400 with code 20015 when thinking-mode history comes back with empty reasoning, while Baidu, Alibaba and Cloudflare accept the same history. Test from production, not a laptop — Venice and Novita worked from Moustafa's Mac but returned 429s to nearly every probe from his infrastructure in the same minute with the same key. Pinning is not safety either: with cloudflare, baidu and alibaba pinned and fallbacks disabled, all three eventually failed, and the top model on OpenRouter went down along with Olly.