GLM built its own inference infrastructure with an AI agent

Z.ai published a research post on 17 September describing how an Infra Agent powered by GLM-5.3 built the inference service that GLM-5.3-Flash runs on today. The post is titled “Toward Recursive Self-Improvement,” though the company is explicit that the goal has not been reached — only its early forms are visible.

100,000 accelerators and two weeks

All production inference for GLM-5.3-Flash runs on a cluster of more than 100,000 Chinese-made AI accelerators. According to the company, no one had previously deployed a cluster of this scale. The team faced limited chip memory and bandwidth, a new model architecture, a 1M-token context window and multimodal requests, on top of an immature ecosystem and incomplete kernel support.

A large part of the work was carried out not by infrastructure engineers alone but by the GLM-5.3-powered Infra Agent. Aggressive memory optimizations — ReplaySSM, W8A8 quantization, mixed-precision cache quantization across INT8, FP8 and BF16, Layer Split and an Encode-Prefill-Decode architecture — improved end-to-end serving performance roughly threefold, while per-token cost and hardware utilization approached levels comparable to mainstream NVIDIA GPUs. Going from initial model adaptation to production readiness took less than two weeks.

“Dense feedback”

The post’s central argument is that an agent’s engineering effectiveness depends less on its ability to write code than on how detailed the feedback it receives is. An end-to-end metric tells an agent that results got worse, but never why. The team therefore gave the agent directly usable correctness tests, microbenchmarks, execution traces and runtime events. This “dense feedback” has three properties: it is local, cheap and fast to obtain, and it supports objective verification through reference implementations and controlled experiments.

Two cases in detail

On the Context Parallelism path of the KDA kernel, the agent found a numerical accuracy bug: tl.dot defaulted to TF32 computation even with FP32 inputs, so errors accumulated over long contexts. The fix was to set input_precision="tf32x3"; it was merged upstream into Flash Linear Attention (PR #1180).

On a KV Transfer bottleneck, the acceptance criterion allowed a performance gap of no more than 5%, yet the agent measured more than 20%. The cause was that DeepEP v1.2.1’s intranode_dispatch and intranode_combine did not release the Python GIL, blocking the Mooncake Transfer thread. After releasing the GIL, the gap fell below 1%.

Limits and result

Before launch, GLM-5.3-Flash was tested anonymously on OpenCode and OpenRouter as Ox-Alpha, and within a week it became the most-used model on both platforms, processing more than 62 trillion tokens in six days. The post stresses that choosing objectives, setting boundaries and assessing risk remain human responsibilities.