Muon: What Happens When an LLM Optimizer Treats a Weight Matrix as a Matrix

Muon treats a neural network's weight matrix as a matrix, not as a flat bag of scalar coordinates. AdamW keeps an elementwise running average and normalizes each coordinate independently; Muon takes the momentum update, discards the singular-value magnitudes and keeps the singular directions, approximating that with a few Newton–Schulz iterations. A dev.to explainer by Shrijith Venkatramana traces the idea.

A 4096×4096 transformer weight matrix holds over 16 million values: AdamW sees 16 million coordinates, Muon sees a linear transformation with meaningful directions.

Muon turns an update like U diag(20, 3, 0.2) V^T into approximately U diag(1, 1, 1) V^T: Ortho(G) ≈ UV^T. Directions survive; magnitudes are levelled.

The idea appeared publicly in October 2024, during the NanoGPT speedrunning competition. On October 15, 2024, a Muon-based run set a training-speed record, improving the previous result by about 35%; the original experiments also found that treating Q, K and V as separate matrices worked better than fusing them.

Newton–Schulz instead of an SVD

A full SVD at every optimizer step would be too slow, so Muon never computes one. It normalizes the matrix by its Frobenius norm and repeatedly applies a polynomial — X_next = aX + b(XX^T)X + c(XX^T)²X — with tuned coefficients and about five iterations. Because the polynomial acts on the singular values alone, they converge toward 1 while the singular vectors stay in place; the whole operation runs efficiently in bfloat16, and the extra cost is near 0.5% in the author's example.

From a speedrun to real training runs

Moonshot AI addressed scaling in the 2025 paper “Muon is Scalable for LLM Training”. An orthogonalized matrix's RMS behaviour depends on its dimensions, so Moonshot's rule matches Muon's update RMS to AdamW's; their experiments reported roughly 2× compute efficiency at compute-optimal training, matching AdamW counterparts with about 52% of their training FLOPs. They also trained Moonlight, a 3B/16B mixture-of-experts model, on 5.7 trillion tokens.

Memory differs too: Adam-like optimizers keep two moment tensors per parameter, while Muon's core state holds one momentum buffer — about 400 GB instead of 800 GB at 100B parameters. PyTorch now documents torch.optim.Muon with several update-scaling modes, and DeepSpeed added support in June 2026.

Replacing AdamW everywhere is not the intent: Muon targets 2D hidden weight matrices, while embeddings, biases, LayerNorm parameters and output heads stay on AdamW. Early examples with a fixed learning rate of 0.02 should not be copied blindly, since Jordan's, Moonshot's and Bernstein's scaling rules differ.

In a controlled comparison, architecture, data, tokens, batch size and hardware stay fixed while validation loss is measured against tokens, FLOPs and GPU-hours. An optimizer encodes assumptions about what a parameter means: Adam treats parameters as coordinates, Muon as linear operators.