Janas-LLM outperforms llama.cpp on 13 models running larger-than-RAM LLMs on a 32 GB laptop

Performance Results

Janas-LLM achieved 37 tokens per second writing with Qwen3-Next-80B, a 48 GB model, on a laptop that has only 32 GB of RAM. Without speculative drafts the engine still produced 24 tokens per second, while llama.cpp managed 2.8 tokens per second on the same hardware. Prompt reading reached 62 tokens per second versus 16 for llama.cpp. Across 13 models quantized to Q4_K_M Janas-LLM was faster in both reading and writing.

Hardware and Software Setup

The benchmarks ran on Debian 13 (trixie) with kernel 6.12.107, Mesa 26.1.6 Vulkan driver, GCC 14.2, an Intel Core Ultra 9 185H processor (6 performance, 8 efficiency, 2 low-power cores), 32 GB DDR5 memory and a 1 TB NVMe SSD capable of about 5.7 GB/s sequential reads. The same machine under Windows 11 showed comparable write speeds and up to 1.8x faster prompt reading for dense models when the integrated Arc GPU was used through Intel's Windows driver.

Model Support and Usage

Janas-LLM streams the experts required for each token from the NVMe drive, keeping frequently used experts in a memory cache. This allows models larger than physical RAM to run, including mixture-of-experts models such as Qwen3-30B-A3B, Qwen3-6-35B-A3B and Qwen3-Next-80B-A3B, as well as dense models from the Gemma 4 family. The project provides a chat binary, an HTTP server compatible with the OpenAI API, an MCP server for tool use, and a code-completion endpoint for editors. All components are written in pure C with no external dependencies beyond the C library and optional Vulkan for GPU acceleration.

Sources: GitHub