Microsoft’s new AI model “Muse” can generate gameplay and might preserve classic games

Summary

Microsoft Research and Xbox Game Studios have developed an AI model that can understand and recreate gameplay while potentially bringing older titles to modern platforms.

Their system, called Muse, uses what they call a “World and Human Action Model (WHAM)” to process and generate game content. According to the paper, the research team trained Muse using 500,000 anonymized gaming sessions from the multiplayer combat game Bleeding Edge.

The researchers report that their system demonstrates strong “persistence” capabilities – meaning it can maintain consistent elements throughout generated sequences, a weakness of existing systems. When using five edited reference images instead of just one, WHAM achieved persistence rates of 85% or higher for all element types tested, although performance varied depending on starting locations and element types. Muse also generates physically accurate game sequences of up to two minutes in length – for example, characters navigate stairs correctly and respect wall boundaries.

Video: Microsoft

Testing revealed that larger models with more computing power produce better results. The largest version, with 1.6 billion parameters at 300 x 180 pixels encodes each image using 540 tokens, and offers significantly improved scene reconstruction compared to a smaller model running at 128 x 128 pixels with 256 tokens per image. The researchers haven’t yet addressed gaming-critical latency issues.

Video: Microsoft

Making classic games accessible to new audiences

One key goal is adapting older Xbox games for modern devices, helping preserve gaming history while reaching new players. Microsoft is developing a real-time version of Muse using other in-house games, with plans to release initial interactive AI experiences through Copilot Labs.

To support further development, Microsoft has made the Muse model available on Azure AI Foundry and Hugging Face, including model weights, sample data, and an interactive interface. The researchers emphasize that the technology remains in early stages.

Similar projects are emerging throughout the gaming sector. Stability AI’s SPAR3D generates high-quality 3D objects in near real-time, while GameGen-O creates open-world game simulations. Google DeepMind has introduced Genie 2, a “Foundation World Model” that generates consistent 3D environments lasting up to a minute from single images, along with their GameNGen game generator.

The gaming industry stands to be affected significantly from these AI advances, as games combine multiple elements – code, graphics, text, and 3D assets – all of which can be influenced by generative AI.