How LLMs actually work: a walkthrough of the transformer stack

On June 1, the blog 0xkato published "How LLMs Actually Work", a walkthrough of what happens inside a large language model. Its premise is that modern LLMs are mostly built by stacking transformer blocks over and over, so understanding that machinery gets you most of the way there

From text to numbers

Models do not read text; they read integer IDs. Tokenization turns a prompt into a sequence of integers, each pointing to an entry in a fixed vocabulary of tens of thousands to a few hundred thousand pieces, usually subwords rather than whole words. Each integer then gets meaning from the embedding matrix, a lookup table with one row per vocabulary entry; in many 7B-class models a token becomes 4,096 numbers, and similar tokens end up with similar vectors. The embedding says nothing about position, so models add positional information — sine and cosine patterns in the original 2017 transformer paper, rotary schemes such as RoPE in most modern ones.

Attention and the feed-forward network

Attention is how tokens share information: each token produces a query, a key and a value, queries are matched against keys with a scaled dot product, softmax turns the scores into weights that sum to one, and the values are averaged accordingly. Because GPT-style models generate left to right, causal masking stops a position from attending to tokens that do not exist yet. Multi-head attention runs many of these operations at once, and the heads specialize without being told to — researchers have found heads that track grammar or resolve pronouns. A single layer might have 32 heads, and modern decoder-only models mostly use Grouped-Query Attention, where groups of heads share keys and values, to cut memory pressure. After attention, each token passes through a feed-forward network that processes it alone; most parameters in a dense transformer live there, and some frontier models replace it with a Mixture of Experts such as Mixtral 8x7B, where eight experts exist per layer but only two run for any given token.

The prediction loop, and what differs

Generation takes the final vector of the last token, turns it into one score per vocabulary entry — logits — and converts those into probabilities with softmax. Temperature, top-k and top-p control how deterministic the output is, and each chosen token is appended to the input so the loop runs again, usually reusing a KV cache instead of recomputing the whole prefix. The base model is trained on nothing but next-token prediction; instruction following, preference and safety come from post-training. What separates GPT, Claude, Gemini and LLaMA is not the broad design — the post argues they sit in the same transformer-family space — but the trained weights, the configuration and the post-training. Since 2023 the stack has converged on pre-norm placement, RMSNorm, RoPE, SwiGLU, Grouped-Query Attention and Mixture of Experts.