A 10 year old Xeon is all you need

Hardware that has no business running this

Christina Emilie Sørensen's third post on Gemma 4's MTP drafters asks how to run the result on a recycled server that should not manage it. The machine has 128 GB of RAM, but it is DDR3, five to six times slower than the best laptop memory today. Its CPU is a single Intel Xeon E5-2620 v4 from 2016 with 8 physical cores, 16 threads, AVX2 only — no AVX-512, no BF16 — 20 MiB of L3 cache and 2 MiB of L2, and no GPU. ollama and even stock llama.cpp do not expose enough knobs, so the answer is the ik_llama.cpp fork and a 25-flag command line. On CPU, generation is bounded by memory bandwidth: the processor finishes the matrix maths and then waits while weights are hauled from RAM into cache.

Speculative decoding and MoE routing on a CPU

Gemma 4 26B-A4B is a mixture-of-experts model with 128 experts, 8 active per token, giving roughly 3.8B active parameters out of about 25.2B total. The flags --spec-type mtp, --draft-max 3, --draft-p-min 0.0 and --spec-autotune pair the 26B verifier with the small MTP drafter, accepting up to three tokens per draft. Speculative decoding pays off more on CPU than GPU, Sørensen writes, because CPU cycles are cheap next to streaming the verifier's weights through cache, while the drafter's working set fits in L3. --cpu-moe tunes expert routing for the CPU's cache hierarchy so the model does not thrash it jumping between 128 experts; --merge-up-gate-experts fuses two per-expert projections into one matmul (logs: fused_up_gate = 1). The thread count, -t 8, matches physical cores rather than the 16 SMT threads, since extra threads only add scheduling cost on a memory-bound workload.

Pinning, repacking and flags that fail quietly

--mlock asks the kernel to keep the ~27 GB of weights in physical RAM, and by default the request fails: "failed to mlock 27628376064-byte buffer ... Try increasing RLIMIT_MEMLOCK". The flag is fine, the memlock limit is not, and black-box tools never ask for the optimisation at all, leaving the model to be swapped to disk. --run-time-repack rearranges the weight matrices in memory before inference to match the CPU's cache layout (265 tensors repacked, per the logs), and --no-kv-offload short-circuits the search for a GPU to hold the KV cache. The graph-split flag does not survive MTP: the engine reports "Split mode 'graph' is not supported for Gemma4 external MTP => changing split mode to 'layer'" and falls back to sequential layer splitting.

Flash Attention on a CPU, and the memory bill

ikawrakow, the author of ik_llama.cpp, wrote custom CPU kernels for Flash Attention, which fuses the attention softmax with its matmuls so the N×N attention matrix is never written to RAM — an optimisation originally built for GPUs. --mla-use 3 enables Multi-Head Latent Attention, which compresses the KV cache into a smaller latent representation. Both took: flash_attn = 1, fused_moe = 1, fused_up_gate = 1. The final accounting is 82,355 MiB — about 25 GB of weights and 56 GB of KV cache at the full 262K context, so the cache is larger than the model itself. Sørensen's conclusion: the barrier for local open-weight AI is not silicon, but understanding the inference engine deeply enough to tune it.