44% on ARC-AGI-1 for 67 cents: small transformer trained in 1.5 hours

Researcher Mithil Vakde has published the third post in his ARC-AGI series. This time he trained a small transformer from scratch that scores 44% on the ARC-AGI-1 public evaluation set and 7% on ARC-AGI-2. According to the author, the total lifetime compute cost — training from initialization plus inference across all tasks — came to 67 cents.

The run takes about 1.5 hours on a single RTX 5090 GPU, and the result matches the reported scores of TRM and HRM, the models he now compares against: Vakde notes that he only benchmarks against approaches that use similar test-time training. The code is open source.

How the approach works

Each input-output pair of a puzzle is converted into a sequence of tokens, and a small transformer is trained autoregressively on those sequences from scratch at test time, across the benchmark's puzzles, with the test outputs hidden. To support cross-task learning, every puzzle gets its own learned additive embedding, and positional information is handled with 3D RoPE embeddings, since each sequence contains two 2D grids.

Training data is augmented with color and dihedral permutations. At inference the test inputs are augmented as well, the inverse transformation is applied to the outputs produced, and the two most frequent answers are submitted.

What changed since the previous version

The biggest score gains, per the post, came from a modernized architecture — SwiGLU instead of GELU, RMSNorm instead of LayerNorm — from more diverse and better shuffled data, and from scaling up from 4 to 8 layers. The cost reductions came from using far fewer augmentations, switching the optimizer from AdamW-only to NorMuon with an auxiliary AdamW, and moving to flash attention over variable-length packed sequences.

Why it matters

Vakde frames sample efficiency as the most important unsolved problem in AI, and ARC as a good place to test it: roughly 1,000 puzzles, each with its own rule, and few priors required. His stated aim is to find the limits of what today's deep learning methods can do and to keep iteration cheap enough that anyone can work on the problem.

An ablation analysis found that a substantial part of the performance survives even without augmentations or synthetic data, and across multiple runs the union of solved tasks reaches 55%. The author says he now believes 65% is reachable inside the plain transformer framework, and sees another 10x cost reduction as possible with hand-written GPU kernels.