AI-powered conversations could make gaming more immersive
Language models can enable more realistic dialogue and interaction with non-player characters (NPCs) in video games.
LLMs can generate in-game dialogue that takes into account the context of the game and the player's interactions in real-time.
At CES 2024, Nvidia and Convai, a startup specializing in real-time generative conversations in virtual worlds, showed an updated version of a demo in which a player can have a real-time conversation with two non-player characters in a cyberpunk bar.
The new demo goes beyond character-to-character interaction by allowing game characters to collect items and navigate their environment based on conversations with players.
LLM conversations are not yet a replacement for hand-written dialogue in the specific context of a larger story. But they could make the mostly superficial and irrelevant NPC dialogue in large game worlds much more personal and varied, thus increasing immersion.
Smaller teams in particular could benefit from this technology, but Strauss Zelnick, CEO of gaming juggernaut Take-Two, hinted that it could be an interesting use case for games like GTA.
There are still challenges to implementing LLMs in games, such as latency or the risk of the AI making false statements or generating content that is inappropriate for the game. Developers can mitigate this to some extent by providing thoughtful prompts and guidelines for the AI.
Nvidia's Avatar Cloud Engine (ACE) for generative AI characters
The demo runs on Nvidia's Avatar Cloud Engine (ACE), a generative AI platform for more realistic dialog and interaction in video games. It includes several components, including NeMo for large language models, Riva for speech recognition and text-to-speech conversion, and Audio2Face for facial animation.
Convai's tools, combined with Nvidia's ACE, can help improve the latency and quality of non-playable AI characters in video games.
Startup Replica has developed an "AI Voice Plugin" for the Unity and Unreal game development platforms that supports various LLMs. The plugin can lip-sync sentences generated by LLMs with the AI voices of NPCs and display them with the corresponding body language. You can see the plugin in action in this Matrix demo.
AI News Without the Hype – Curated by Humans
As a THE DECODER subscriber, you get ad-free reading, our weekly AI newsletter, the exclusive "AI Radar" Frontier Report 6× per year, access to comments, and our complete archive.
Subscribe nowAI news without the hype
Curated by humans.
- Over 20 percent launch discount.
- Read without distractions – no Google ads.
- Access to comments and community discussions.
- Weekly AI newsletter.
- 6 times a year: “AI Radar” – deep dives on key AI topics.
- Up to 25 % off on KI Pro online events.
- Access to our full ten-year archive.
- Get the latest AI news from The Decoder.