Can gzip be a language model? A text generator built on DEFLATE

A blog post published on nathan.rs asks a question that sounds like a joke: can gzip, the compression tool that ships with almost every operating system, work as a language model? The author built gzipt, a small experiment that generates text using nothing but the compressor — no neural network, no learned parameters.

Compression is prediction

The starting point is a paper called "Language Modeling is Compression" (arXiv 2309.10668), which states the compression-prediction equivalence: every prediction model is inherently a compressor, and every compression algorithm is a prediction model. The intuition comes from information theory — a symbol the model expects costs fewer bits to encode. Specifically, the cost is −log₂ p, where p is the probability the model assigns to that symbol. High probability means a short code.

gzip uses the DEFLATE algorithm over a 32 KiB sliding window. When the next bytes repeat something already in the window, DEFLATE encodes them as a cheap back-reference instead of literal bytes. That gives the experiment its score: compress the context together with a candidate continuation and measure the length of the result. The shorter the compressed output, the more "predicted" the candidate is. Priming gzip with a corpus puts that corpus into the window.

From scoring to generation

Scoring is not generation. Picking the single next byte that compresses best fails, because gzip returns an integer byte count without fractions. Adding one byte often does not change the length at all, so many candidates tie and the signal drowns in quantization noise. gzipt works around this with beam search: it keeps the beam_width most compressible partial continuations, extends each one by every byte that occurs in the corpus, scores the results by compressed length and prunes back. It repeats this for a horizon of bytes before committing the best span, or sampling among finalists when the temperature is above zero.

One design detail matters: only the last bytes of generated output stay in the scoring context. DEFLATE codes nearby matches more cheaply than distant ones, so a model that could see its entire history would find it cheapest to fall into verbatim loops, repeating the text it just produced.

What the output looks like

Primed on tiny Shakespeare and given the prompt "MENENIUS:", the tool returned lines attributed to Menenius, Marcius and Lartius — not coherent, but clearly shaped by the source text. The code is a single file of standard-library Python that uses zlib; the name GziPT, the author writes, was too good to pass up. The footnotes add that the paper's authors tried something similar and performed poorly, and that adding beam search improved generation quality significantly.

The takeaway is narrow but real: a general-purpose compressor can carry a usable implicit model of text. It is nowhere near a neural language model, and the author says so. But it is a clean demonstration that prediction and compression are the same problem seen from two sides.