Back

Homelabber Runs 2.8T Kimi Model at 3.11 t/s With Two RTX 3090s and Optane
SiTech AI Team3 min read

Homelabber Runs 2.8T Kimi Model at 3.11 t/s With Two RTX 3090s and Optane

A homelabber's second-generation Optane build added two RTX 3090s to run Moonshot's 2.8-trillion-parameter Kimi K3 model at 3.11 tokens per second, with a RAM drive boosting generation by about 41%.

A homelabber detailed improvements to a second-generation Optane AI build in a follow-up Reddit post, adding two RTX 3090s to run the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second. Moving almost half of the model into a RAM drive raised generation speed by about 41% and prompt evaluation by about 26%. The build still needs extra fans because the Optane memory gets very hot.

Performance with and without a RAM drive

With both GPUs but no RAM drive, the build ran at 2.20 tokens per second for generation and 13.3 tokens per second for prompt evaluation. With 45 of 94 shards, or 414 GB of the 865 GB model, in a tmpfs RAM drive and the rest in Optane, generation reached 3.11 tokens per second and prompt evaluation reached 16.8 tokens per second. The builder's measurements used llama.cpp, with DAX on for the Optane portion.

Such an arrangement is not new to enthusiasts who set aside system memory for extra-fast performance in a pseudo-non-volatile manner, where it appears like a disk, although not everyone has that much RAM free these days.

How the GPUs and memory are used

The two RTX 3090s have 24 GB of VRAM each, or 48 GB total, about 5.5% of the 865 GB model file, so they are not holding the model. The model is a mixture-of-experts type with 896 experts; 16 are chosen per token, plus two shared, with 104 billion active parameters out of 2.8 trillion. The routed-expert tensors stay on the CPU side, in Optane and the RAM drive, while the RTX 3090s handle attention, the dense feed-forward network layer, shared experts, other active-dense components, the KV cache, stored conversation context, and GPU compute buffers.

DRAM is slower than VRAM, but it is still capable of giving a decent speedup, and the speed with the GPUs was within the range the builder expected, after guessing somewhere between 2 and 4 tokens per second in the original build thread.

Model size and quantization

All 2.8 trillion parameters would take about 5.6 TB at 16-bit precision. Moonshot's own release uses a 4-bit MXFP4 format and comes to about 1.56 TB, while the Q2_K file used here shrinks the model to 865 GB, an average of under 3 bits per parameter. The report notes that the 865 GB figure is likely in gibibytes, but uses the GB values as written by the homelabber.

Cooling and future plans

The GPU installation waited in part for case modifications, which required cutting some metal and relocating the power supply to make room for the GPUs and their cooling. Cooling remains a major issue, and the two 12W blower fans for the Optane DIMMs are kept in place even with dual 350W-class GPUs. The builder sees more optimizations ahead, including "customized quantization," which may be similar to techniques used by lawyer Nick Mermiges, and notes that speed alone is not enough because output quality must also be considered.

Sources: Tom's Hardware

SiTech

SiTech — AI-powered web development

We build fast, modern websites and bring AI into real business workflows. Have a project or a question? We'd love to help.