A 125M-parameter model that autocompletes piano on-device

The author of a blog published under the name SimEdw has trained a 125-million-parameter transformer that continues a MIDI piano performance in real time. The model runs entirely on-device and produces roughly 108 notes per second on an iPhone 15 — more than live playing requires. The app built on it, RollTab, is free on the App Store for anyone with a MIDI keyboard and an iPhone or iPad; the project took about a year and fourteen experiments.

Choosing a note representation

A MIDI file stores events rather than recorded sound: key presses with pitch and velocity, key releases, sustain pedal changes. A token per event inflates the vocabulary — note-on and note-off alone produce up to 16,512 combinations, since 128 pitches multiply with 128 velocity values.

The final format packs a whole note into a single step: NOTE(pitch, delta_onset, duration, velocity). There is no separate time-shift event; silence lives in the next note's delta, and chords are several notes sharing a zero delta. Sustain is baked into note duration during preprocessing, so the model predicts only pitch, onset, duration and velocity. The transformer therefore advances the music one note at a time instead of spending four passes per note.

Data, training and evaluation

The dataset holds a few hundred thousand MIDI files — roughly 300 million note events — mostly older public-domain classical material. Cleaning mattered more than scale: the author selected piano-focused material, filtered by density, deduplicated with fingerprints that ignore transposition and tempo, and grouped alternate versions of a piece into the same split. Scaling the data up about fivefold made the models worse.

Training is cross-entropy over five heads — type, pitch, delta, duration, velocity — so errors stay trackable per attribute. Scheduled sampling, which feeds the model its own predicted pitch during training up to 50% of the time, hurt validation loss (2.9998 versus 2.9495) but improved continuations: in pairwise comparisons scored by Gemini 3.5 Flash, that model was preferred 64.3% of the time. The preference data then fed DPO post-training.

On-device and open limits

The architecture is a standard decoder-only transformer: RMSNorm, rotary positional embeddings, causal self-attention and SwiGLU blocks. Three sizes were trained — about 33M, 64M and 125M parameters. The medium model nearly matched the large one, and the author is now trying to close that gap to reduce footprint and latency in the iOS app, which runs through Core ML.

The author is candid about the limits: the model loops occasionally, short prompts remain hard, and unprompted generation is hit or miss. “Think GPT-2, but for piano,” he writes. Still, he says the goal is met — sitting down, playing a few notes and improvising with the model is finally enjoyable.