Compression is prediction: how compressors and LLMs solve the same problem

ngrok developer educator Annie Sexton published a long essay on 11 August arguing that data compression and large language models are two expressions of the same underlying mathematics. The post, titled "Compression is prediction", walks through the workings of a compressor step by step and ends at the point where the two fields meet.

Redundancy, not just shorthand

Sexton starts with minification: stripping a JavaScript snippet of comments, whitespace and long variable names shrinks it from 156 to 62 characters, yet nobody counts that as data compression. Real compression exploits redundancy. A 28-character string such as "AAAAAAAAABBBBCCDAAADDDDDDDDD" can be rewritten as "A9B4C2D1A3D9" through run-length encoding alone, taking it from 224 bits to 96, or 57 percent smaller.

Modern tools combine three parts: transforms, models and entropy coders. The model assigns probabilities to symbols, and the entropy coder turns those probabilities into a bitstream. Arithmetic coding can represent the whole string "ABABAAC" as a single binary fraction, 0.3876953125, which needs 10 bits instead of the 56 of raw ASCII.

Entropy is a floor

The average number of bits per symbol is entropy, the Shannon limit that lossless compression cannot beat. Skewed data compresses better: a 12-character string dominated by one letter averaged 0.82 bits per symbol, where the earlier, more balanced example needed 1.38. Context sharpens the odds further. The letter U appears with a probability of 0.028 in English text, but after a Q that rises to 0.999, worth about 0.001 bits against 5.16. Applying an order-1 model to the phrase "TO BE OR NOT TO BE" cut the compressed output from roughly 47 bits to 21.

LLMs as predictors

A 2023 Google DeepMind paper argued that language modelling and compression are two views of the same thing, and Sexton follows the logic: an LLM returns a probability distribution for the next token, and the number of bits needed to store that token is minus the logarithm of the probability the model gave it. Cross-entropy, the quantity language models minimise during training, is calculated with the same formula. On a quote from Dickens, an order-1 model needed 434 bits while GPT-2 needed only 176, or 10 percent of the original.

Practical limits remain. Shipping a multi-gigabyte model to compress HTTP responses would cost more than the bytes saved, which is why gzip and Brotli, built on fixed, lossless coders, still carry the web. The open question is not how close a coder gets to the theoretical limit but how far down a better predictor can push that limit.