Making Games Legible to LLM Agents

Developers using LLMs for real game development often hit a wall. While LLMs excel at web-based stacks, applying them to traditional game development yields poor results. Games contain elements that are opaque to LLMs, meaning default models struggle to read and modify them effectively. To fix this, developers need to adopt harnesses that make games legible to AI agents.

Why LLMs Struggle With Games

LLMs are bad at reading games in their default state. Unlike web apps with vast amounts of open source code, games lack standardized tooling and have many elements that are not easily representable by code. When developers rely on LLMs to puppet these opaque elements, the models fall over. The core issue is a context problem. Since games are not just code, existing harnesses that work with code are insufficient. Developers need to crack open the black box of the game and make more of it addressable directly by an LLM.

Determinism and Reproducible States

To make a game legible to an agent, determinism is key. A game should be able to easily and exactly reproduce a specific state. This requires a single source of seeded randomness from the very start, ensuring every call to non-deterministic functions uses this source. If code calls a random float function in 30 places, the 30th call must be identical every time. Reproducible states allow agents to reach the same point from scratch repeatedly to test changes. Built in screenshot APIs are also recommended. Toggling UI and gameplay layers on and off reduces visual clutter, allowing the LLM to focus on the specific layer being addressed.

Making Games Legible to AI

Visual evaluation through screenshots can introduce context rot. Even multimodal models struggle to understand changes from a whole screenshot, acting as context pollution. Instead of naively showing images, developers should use image diffing. Tools like odiff can compare two images and output a quantitative difference value, like 85 percent similar. This provides a stable baseline for an LLM to evaluate progress across each turn as it tries to reduce the difference between the current state and a target image.

Architecture for Agents: Thing Driven Development

Architectural choices also matter. Maintaining deep object oriented programming hierarchies creates confusion for AI agents trying to determine where to make changes. Instead, developers should adopt Thing Driven Development. This assumes everything is a single object type, or Thing, with stat values and a type enum. AI likes this because there is a single source of truth. There is no ambiguity about whether to extend a class or create a new one. Saving and loading also becomes trivial. A game becomes just a giant list of Things, serializing through one path. Changes to these objects can be captured using the Command Pattern, storing modifications as serializable command objects.

Sources: kylekukshtel.com