Benchmarks Expose AI Reasoning Gaps as New Open Weight Models Trail China

Import AI 476 covers benchmarks probing what current AI models cannot do, new open weight models from Western startups, and government efforts to pool supercomputing for AI training.

Humanity's Sixth Sense exposes visual reasoning gaps

AI startups Elorian and Scale AI have released Humanity's Sixth Sense (HSS), a benchmark measuring how models reason through information a visual scene implies but does not explicitly show. Its 522 questions cover 288 images and 234 short video clips totalling 17.6 hours, across temporal and causal dynamics, physical and spatial logic, social understanding, and abstract and contextual inference.

People score an aggregate pass@1 of 93.1, while the best AI system, GPT 6 Astra, reaches 53.6. Chinese open weight models lag behind, with GLM-5.3-flash at 26.8 and Kimi K3 at 25.5. The most common failures were missing a detail that decides the answer, at 21 percent, and misidentifying an object, person, or role, at 20 percent. The results undercut the assumption that visual reasoning requires dedicated 2D and 3D world models.

AI research taste is improving quickly

P-Zero Research's TasteVal benchmark measures whether models can pick interesting problems, design experiments, and interpret results across eight tasks spanning pretraining data curation, preference modeling, and robustness to adversarial prompts. Results are checked against 24 experts from organizations including OpenAI, Google DeepMind, NVIDIA, and Microsoft Research.

The authors say experimental research taste has doubled every 3.0 months since December 2025, and the best model, Opus 5.5, exceeds the human baseline with a compute multiplier of 2.30x at roughly one thirtieth of the expert baseliners' average cost per attempt. Jack Clark argues this strengthens the case that full recursive self-improvement may arrive in 2027 rather than 2028.

New open weight models trail Chinese rivals

Reflection has revealed Beam, a 501 billion parameter mixture of experts model with 23 billion active parameters, promoted for token efficiency. It was pretrained on 6,144 GB300 NVL72 GPUs for four weeks, then trained with reinforcement learning on about 10,500 of the same GPUs, using 23.8 trillion tokens. Beam scores 44.4 on DeepSWE 1.1 versus 61 for GLM-5.3, but matches GLM-5.2 while using three to four times less inference compute. Weights and technical report are planned for later this month.

Mistral has announced Mistral Large 4, an open weight model tuned for cyber tasks with fewer refusals than proprietary models from Anthropic and OpenAI, trained on 3,800 NVIDIA Grace Blackwell GPUs in European datacenters. It scores 62 on DeepSWE 1.1 versus 68 for Kimi K3, and Mistral says internal tests showed users preferred ML4 for CAD and STEM while performing on par with GLM-5.3 in finance and coding.

Government labs pool supercomputers for AI

Two US Department of Energy projects show how federated training can spread AI work across national labs. Researchers trained LLaMA-based models across Sandia, Lawrence Livermore, and Los Alamos systems including El Capitan, and fine-tuned LLaMA-2 7B on a chemistry dataset across about 1,700 Intel, NVIDIA, and AMD GPUs at Argonne, Oak Ridge, and Berkeley Lab. The authors note the scale is roughly 100x off the frontier, but see the federated approach as a start.

Sources: Import AI