Qwen3.8-Flash-Next: Alibaba's Qwen team previews the architecture behind Qwen4

Alibaba's Qwen team published Qwen3.8-Flash-Next on August 26, an open-weight experimental model that the company says previews the architecture that will underpin Qwen4. The weights are released on Hugging Face under the qwen-community-1.0 license and work with Transformers, vLLM, SGLang and TokenSpeed.

Hybrid attention and QSA

The core change is architectural: the Gated DeltaNet plus Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Instead of selecting individual tokens, QSA picks the relevant context at micro-block granularity, which cuts long-context latency significantly — a gain the team describes as critical now that agentic workloads dominate real-world use.

Gated Residual and n-gram embeddings

Gated Residual modulates the information flowing through widened residual streams with an element-wise, data-dependent read gate and a per-branch scalar write gate, keeping training stable while adding expressiveness across layers. N-gram embedding opens a separate axis for parameter scaling: it needs less computation than a mixture-of-experts and is easier to offload, which helps on memory-constrained accelerators.

Scale and benchmark results

The model carries 125 billion parameters, of which only 6 billion are active at a time, plus 51 billion n-gram embedding parameters and 4 billion MTP parameters. It has 48 layers and 512 experts, with 10 routed and one shared expert per pass. The context window is 262,144 tokens natively and extensible to one million. In the published results, Qwen3.8-Flash-Next scores 62.5 on SWE-bench Pro, 58.7 on DeepSWE 1.1, 81.0 on SWE-bench Multilingual, 48.1 on NL2Repo-Bench and 73.9 on CoWorkBench.

What it means for developers

On training, Qwen says it applies the Muon and AdamW optimizers to specific weight categories and removes the traditional batch-size warmup, starting directly at the target batch size to cut optimizer steps. A production counterpart, Qwen3.8-Flash, runs in the cloud with a one-million-token context by default and built-in tools, while the open weights remain available for self-hosting. The technical report is published in the Qwen GitHub repository.