Google DeepMind Releases Genie 4 World Model With Audio and Real-Time Steering

Google DeepMind released its newest world model, Genie 4, in research preview on Tuesday. The model generates the most photorealistic and interactive worlds yet and lets users type prompts mid-simulation to change the scene. It also generates real-time audio that matches the simulated world, including music, ambient noises, and interactive dialogue.

Model improvements

Genie 4 allows free-form edits such as changing the weather or camera angle in real time or adding new people or objects. It can maintain a consistent scene for up to two minutes, double its predecessor's limit, and offers better physics and more humanlike characters. Gemini is more closely integrated with Genie 4, acting as the director that plans steps and checks frames while Genie creates the visuals.

Robotics experiment

In a research experiment tied to Gemini Robotics, a Gemini 4 Argon model broke down tasks into steps that Genie 4 simulated in real time. The experiment shows a possible way to test robot scenarios in simulation before they happen in the real world. Tejas Kulkarni, research director at Google, said future hybrid world models may combine Gemini writing code and text with Genie ingesting that in real time and simulating reality at different levels of abstraction.

Industry context and uses

The release comes a couple of months after Runway released its second world model, GWM Worlds 2, a playable model that responds dynamically to user inputs and includes audio generation. The world model industry is also consolidating: Fei-Fei Li's World Labs is being acquired by AMD, Amazon is in talks to buy Decart, and Rivian acquired Third Dimension to bring simulation capabilities in-house for self-driving testing. Genie 4 is Google's latest move to stay ahead in the crowded field. Pitched commercial use cases include gaming, interactive learning, and robotics. One speculative interactive learning example would be entering a painting of a historical scene and chatting to the painting's subjects about the event portrayed. Trusted testers are pointing to a possible new use: prototyping films. Diego Rivas, a Google DeepMind engineer, said the real-time aspect becomes most valuable where keeping a state of flow in the creativity space is useful, such as ideation and storyboarding. Google also offers Veo, an AI video-generation model that makes high-definition clips with built-in synchronized audio from text, image, or video prompts, a tool growing in popularity among filmmakers. Rivas added that users could eventually rely on Gemini Omni, a multimodal large language model that can generate video from images, audio, and text, to turn prototypes into creative assets, though those capabilities are still a ways off.

Limitations and availability

Rivas stressed that it is still early days for Genie 4 and that it is not yet clear what the end-to-end product experience will be. The team acknowledged that details and objects can drift the longer a simulation runs, and they have not quite figured out how to ensure the audio will be accurate and any in-world text will be legible. Genie 4 is available now as a research preview for trusted testers. In the coming weeks, the model will roll out to Google AI Ultra subscribers in Project Genie, Google's AI tool for generating interactive worlds from text or image prompts, and early access partners of the Gemini API and Google AI Studio.

Sources: TechCrunch · Ars Technica · The Verge