AI Scientists Launch Trillium Labs to Do High-Stakes Research in the Open

Two industry scientists, Nathan Lambert and Tom Zick, have founded a nonprofit AI lab that plans to work in the open. Trillium Labs launched on 2 October and will publish details of its experiments so outside researchers can study and replicate them, including areas that many frontier labs treat as too sensitive to share.

The case for openness

Lambert argues that the secrecy of frontier labs limits scrutiny of their methods and ideas; letting outside experts see how models are built and tuned could be crucial to mitigating risks. “Over the past few millennia, humanity has had the scientific method in our toolbox as a way to mitigate harms and build better futures,” Lambert tells WIRED. “The current closed trajectory of frontier AI development is taking us a step backwards.”

What the lab will study

Trillium Labs will initially focus on post-training, where large models are fine-tuned after they are built. Another key area is recursive self-improvement (RSI), a process in which AI contributes to research on new models; the prospect of indefinite progress and a loss of human control has alarmed many researchers. Recently, a researcher who left Anthropic warned that RSI could pose an existential threat to humankind.

The nonprofit will also examine how reinforcement learning shapes the character and behavior of AI models. It has made agents far more capable, but also more inclined to do unexpected things; models can become overly sycophantic. “To understand something like how reinforcement learning scales in post-training, you need significant compute and a lot of careful experimentation,” Zick says.

Funding and the wider debate

Trillium Labs has raised an undisclosed sum from Schmidt Sciences, Halcyon Futures, and others. The founders aim to raise $40 million to $100 million and plan to spend $30 million on training over the next 18 months.

The debate over access is sharpening as models grow more powerful. Models from OpenAI and Anthropic are only available through an app or an API; other developers, especially in China, release models anyone can download and run. Xiaomi recently published live details of a major training run, and researchers at Stanford are pretraining the model Marin in the open. Proponents of limited access want such capability kept in trusted hands; Lambert and Zick believe a shared understanding of the risks leaves everyone better off.

Lambert previously worked at Ai2 and Hugging Face and founded American Truly Open Models, which encourages US companies to release more open models. Zick worked at Harvard University and helped Charles Schwab devise policies around responsible AI. “We’re in an era of AI discourse dominated by a few world views,” Lambert says. “We believe that the scientific method and careful measurement of recent events is the best way to understand new behaviors of AI models.”