LLM providers often serve 32K context no matter what the model card says

A developer who runs a hosted chat and coding agent on open-weight models warned on dev.to on 26 September that the context window on a model card is rarely the window an endpoint actually serves.

The number on the card is not the number you get

The window on the card is a property of the weights; the window a client receives is a property of whoever serves them. Across the endpoints the author tested, most serve around 32K tokens regardless of what the card claims, a handful reach 256K, and he has not found one that actually serves the 1M figures from release announcements.

Operators have a reason: maximum context length is a KV-cache budget balanced against concurrency, and a million tokens per request for everyone would be ruinous. The trade-off is rarely documented and fails silently.

Nothing errors

There is no 413 and no warning. The request is served against whichever ceiling is lowest, and the front of the context is simply gone. In chat this is nearly invisible: the conversation just gets slightly dumber over time.

For agentic runs it is the binding constraint. A long build hits the ceiling, compaction fires, the goal statement is summarised into vagueness, and the model, no longer knowing precisely what it was doing, quietly restarts the plan. From outside it looks like a model bad at long tasks; the author blames the serving configuration.

How to find the real ceiling

Send a deliberately oversized prompt and read the error: providers usually leak the true maximum there, so pushing 500K tokens at an endpoint that claims 1M shows what comes back. Otherwise, find where the earliest content stops influencing output and count backwards.

Documentation and the model card proved unreliable.

Three consequences worth internalising

A benchmark number scores a model at one served context length, not a model: the same weights at 32K and at 200K are not the same agent. Comparing providers on cost per token without pinning the ceiling means calling the difference price.

In a gateway with fallbacks the ceiling can change mid-session: a run that fails over from a 200K provider to a 32K one does not error, it truncates. That is a correctness problem, not a performance one.

In RAG it silently invalidates retrieval tuning: chunk size, top-k and reranking are tuned against a budget taken from the card. A system tuned for 128K but served 32K over-retrieves, its reranked chunks get truncated, and the answer returns confident and wrong; teams then tune the embedding model, which was never the problem.

What he wants is a discoverable way to read the effective served context per request, in model metadata or a response header. He discloses that he builds Grunz, a chat and coding agent on open-weight models.