DeepSeek publishes DSec, a sandbox platform for agentic AI training at scale

DeepSeek published a technical report on arXiv on 19 September about DeepSeek Elastic Compute (DSec), the production system behind its agentic training. During reinforcement learning (RL), a model issues commands, reads files and calls tools inside isolated environments, the workload that strains the infrastructure most.

According to the report, one production unit spans nearly 160 CPU nodes with 30,000 cores and about 250 TB of DRAM. On a typical day it serves roughly 3 million sandboxes; peak concurrency reaches 380,000 and the creation rate exceeds 5,000 per second.

One SDK, four kinds of sandbox

DSec exposes four backends through one Python client, libdsec: FnCall for short stateless tasks, containers for software-engineering work and tool use, Firecracker microVMs for stronger isolation, and full virtual machines where a complete OS is needed, such as Android. The authors say no single sandbox abstraction covers every agentic task.

Environments are composed from independently versioned layers: a base image, a workspace carrying the task's repository and frequently updated toolkits. They are merged with overlayfs instead of being fused into one monolithic image. In a single production week the platform served more than 11,000 base images and over 102,000 workspaces.

Density, memory and 3FS

Agent sandboxes are stateful and mostly idle: about 90% of containers and microVMs use at most 5% of the CPU they request. DSec therefore packs nodes tightly, up to 3,200 containers or 800 microVMs each, while virtio-pmem and DAMON cut peak host memory by 40.2% and time-integrated use by 21.2%. SCHED_IDLE with core scheduling lowers SMT-induced slowdown for latency-sensitive tasks from 45.2% to 17.3%.

Image data is loaded on demand from 3FS, DeepSeek's distributed filesystem, with EROFS metadata kept local. In a burst of 8,192 containers, eager full-image pulling stretched completion time 1.71 times and wrote about 57% more data: sandboxes touch only 4% to 13% of their images.

Co-designed with the RL framework

DSec was co-designed with DeepSeek's RL framework. GPU training jobs are routinely preempted, so rollout execution moved onto DSec, outside the training GPU pool; a pause request lets the platform reclaim memory while preserving sandbox state. The report also documents unwanted agent behaviour and the mitigations it prompted: AppArmor access controls and eBPF network allowlists. The storage components, a Rust OverlayBD and a ublk library, were open sourced.