Transformer Explainer: an interactive look inside a GPT-2 language model

Transformer Explainer is an interactive website that shows how a transformer language model turns a prompt into the next word. A live GPT-2 (small) model with 124 million parameters runs in the browser, so every chart on the page reacts to the text a visitor types. The project comes from the Georgia Institute of Technology, where the team of eight includes Aeree Cho, Grace C. Kim and Polo Chau; it is published with a paper in the ACM Digital Library and a video tutorial.

From tokens to probabilities

Input text is split into tokens, words or word fragments, and each token becomes a 768-dimensional vector. GPT-2's vocabulary holds 50,257 tokens, so the embedding matrix alone carries roughly 39 million parameters. Positional information is added on top, because the architecture has no built-in sense of word order.

The vectors then pass through 12 identical transformer blocks, each combining multi-head self-attention, which lets tokens exchange information, with a multilayer perceptron that refines each token on its own. Attention scores are masked so that a position sees only tokens to its left, the trick that makes next-token prediction possible without peeking ahead.

Inside attention and the MLP

Queries, keys and values are built from the embeddings through learned weight matrices. The site compares them to a web search: the query is what you type, keys are the result titles, values are the pages' content. GPT-2 (small) splits them into 12 heads, one perhaps tracking syntax, another broader meaning. Their dot products are scaled, masked and passed through softmax, then combined with the value matrix.

The MLP widens each token from 768 to 3,072 dimensions using a linear layer and a GELU activation, then compresses it back to 768. A final linear layer projects the representation onto the 50,257-token vocabulary as logits, and softmax turns those into a probability distribution to sample from.

Sliders, sampling and implementation

Visitors can move a temperature slider, which divides the logits: below 1 the distribution sharpens and output becomes more predictable, above 1 it flattens and the text grows more varied. Top-k and top-p settings restrict sampling to the likeliest candidates.

The demo runs entirely in the browser: the PyTorch GPT implementation from Andrej Karpathy's nanoGPT project was converted to ONNX Runtime, while the interface uses Svelte and D3.js.