DeepSeek open-sources DeepInfer inference framework for V4.1-Flash

DeepInfer is now open source

DeepSeek said it is open-sourcing DeepInfer, the inference framework that powers its large-scale production services. The project repository says the release supports DeepSeek-V4.1-Flash and runs on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. DeepSeek plans to add support for future systems, according to Bloomberg.

DeepSeek said on X that it was sharing its production stack to help the community lower compute costs, reduce energy use and make AI more affordable. The company is headquartered in Hangzhou.

What the framework includes

DeepInfer includes optimized kernels and parallelism methods for attention and mixture-of-experts layers. It also provides a decoding method that uses draft tokens and confidence estimates to schedule verification, plus asynchronous scheduling that overlaps CPU batch scheduling with device execution.

The release also covers batch-invariant inference, deterministic sampling and structured outputs. Batch-invariant inference and deterministic sampling work on Nvidia hardware only. The kernel list includes custom TileLang kernels and covers Huawei Ascend 950 chips.

Deployment limits and requirements

The release provides offline inference and leaves out parts of a complete inference service. DeepSeek said users must supply a KV cache system, prefill and decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode.

DeepInfer requires Linux, GCC 13.0 or later and Python 3.12, plus CUDA 13 on Nvidia hardware or CANN 9.2 on Ascend. It is released under the Apache License 2.0. DeepSeek said the framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.

Why the release matters

Bloomberg said the release reveals methods DeepSeek relies on to hold down inference costs for advanced AI, techniques that AI companies tend to guard. Firms that host or run DeepSeek models in the cloud can apply the methods on their own, and Bloomberg said the price of serving a given model can vary widely with the serving software.

No outside party has yet benchmarked DeepInfer's efficiency gains, according to Bloomberg. The framework applies to DeepSeek's own models, but outside developers might read the code and borrow its techniques for their own systems. Last month, DeepSeek announced a partnership with Huawei to build programming tools for Ascend chips and open-sourced six software modules for the platform, including a version of TileLang adapted to the Ascend 950.

Sources: Quartz · GitHub · Bloomberg · Techmeme