Researchers simulated the iconic CS:GO map "Dust2" entirely through neural networks on a single RTX3090 GPU. While these clips are both impressive and problematic, they illustrate the admirable progress that generative AI has made in simulating fully 3D gaming environments.
Eloi Alonso, one of the leaders of the project, showed off a running clip of the Diamond world modeling diffusion simulation on X/Twitter. At first glance, although the output is only 10FPS, if you remain patient, the gameplay is quite complete and coherent. Players can swing the gun, reload, see the muzzle flash, and even experience the recoil.
However, things start to get weird when you realize that this model isn't actually running CS:GO's engine. The researchers fed it a large number of deathmatch scenes from Dust 2 for training until the neural network could basically "ghost" its own approximation of classic maps and gameplay. The GitHub page states that they used over 5 million frames or 87 hours of gameplay. Then they played all these games using the RTX3090.
At this time you will find some small problems. Since GPU simulation does not grasp concepts such as gravity or collision detection, the physical effects of the game are impossible to talk about. Players can jump endlessly, basically fly, have weapons that deform strangely in certain lights, move quickly that breaks the environment into an abstract blurry mess, and can even walk through solid walls like some kind of ghost.
Of course, if you want a true, non-nightmarish Dust 2 fuel experience, you can download Counter-Strike 2 on Steam right now and enjoy the game at frame rates that don't look like a slideshow. Of course, this doesn't require an RTX3090, in fact, the game is optimized to run properly with just 1GB of video memory.
While Alonso's AI experiment is just an experiment, it represents an important milestone in the power of on-device AI processing. The model is trained entirely on a single GPU and then the same GPU drives generative real-time simulations.
Demonstrations like this are rare, but this isn't the first time generative AI has tried to recreate the gaming experience. For example, a team at Google recently launched GameNGen, which uses a custom stable diffusion model to generate a Doom level in real time.
Renowned developer Peter Molyneux predicts that artificial intelligence will eventually create "the majority" of games, from characters and animations to dialogue and game assets.