Stanford's CS336 Teaches Students to Build a Language Model From Scratch

Stanford has published the materials for CS336, a Spring 2026 course titled “Language Modeling from Scratch” that leads students through the entire process of building a language model rather than using one. The class is taught by Percy Liang and Tatsu Hashimoto, with lectures on Mondays and Wednesdays, 3:00–4:20pm in Skilling Auditorium, and recordings posted to a public YouTube playlist.

What the course covers

The course takes its cue from operating systems classes that have students build an entire OS from scratch. Here the pipeline runs from data collection and cleaning for pre-training, through transformer construction and model training, to evaluation before deployment. The instructors say the goal is a deep understanding of language models at a time when a single general-purpose system is expected to handle a wide range of downstream tasks.

Prerequisites and workload

CS336 is a five-unit, implementation-heavy class with deliberately minimal scaffolding: students are told to expect to write at least an order of magnitude more code than in other AI courses. The listed prerequisites are proficiency in Python and software engineering, experience with deep learning and systems optimisation (PyTorch, the memory hierarchy), college calculus and linear algebra, basic probability and statistics, and introductory machine learning. A significant part of the work involves making neural language models run quickly and efficiently on GPUs across multiple machines.

Five assignments, one model

Assignment 1 covers the basics: tokenizer, model architecture and optimizer, then training a minimal language model. Assignment 2 moves to systems — profiling and benchmarking, writing a Triton implementation of FlashAttention2, and building memory-efficient distributed training code. Assignment 3 examines scaling: what each Transformer component does, and fitting a scaling law through a training API. Assignment 4 turns raw Common Crawl dumps into usable pretraining data with filtering and deduplication. Assignment 5 applies supervised fine-tuning and reinforcement learning so that models reason through maths problems, with an optional second part on safety alignment methods such as DPO.

Compute, honour code and schedule

For self-study, the course lists cloud GPU options for a single B200 GPU, with public prices recorded on March 28, 2026: Modal at $6.25 per hour with $30 of free monthly compute, RunPod at $4.99, Nebius at $5.50 (or $3.05 preemptible), Lambda Labs at $6.69, and Together at $7.49 with a minimum of eight GPUs. The honour code allows study groups and permits LLM prompting for low-level programming or conceptual questions, but prohibits using it to solve assignments, and encourages disabling AI autocomplete. The 19-lecture schedule runs from March 30 to June 3, ending with guest lectures by Daniel Selsam and Dan Fu.