Intelligent NPCs
ElevenAgents Unity SDK: talking NPCs in a Unity scene
ElevenLabs has released a Unity SDK that connects a game scene to a voice agent, so players can speak to a character and hear it answer from the character's own position.
On 5 November ElevenLabs released the ElevenAgents Unity SDK, a package that lets a player walk up to a character in a Unity game and talk to it out loud. The first tagged release is v0.1.0, published under an MIT licence.
ElevenLabs says the SDK listens through the player’s microphone, plays the agent’s voice through an ordinary Unity AudioSource, and runs the whole session through one Conversation API.
How it works
You create the agent in the ElevenLabs dashboard, where its prompt and voice live. In the game, a call to Conversation.StartSessionAsync opens the session. The SDK chooses the transport for the build target. The sample scene uses a trigger collider, so the conversation starts when the player enters it and ends when they leave.
A few features matter for game design:
- Voice from the character. Pass your own
AudioSourceand the voice is positioned in 3D space. AGetOutputVolume()call returns the agent’s level from 0 to 1, which can drive a mouth animation. - One agent, many characters. Dynamic variables fill placeholders such as
{{mood}}in the prompt, so two characters can share one agent. The sample uses a happy yellow cube and an angry red one. - Game events.
SendContextualUpdatetells the agent what has happened, for example that the player picked up a key, without adding a turn the player did not speak. - Actions. Client tools let the agent trigger game code. The example registers an
open_doorhandler, and the agent hears the result.
Platforms and requirements
The documentation lists Unity 6.3 LTS (6000.3.0f1) or later. The Editor and desktop builds on macOS, Windows and Linux are supported, and so is WebGL. iOS, Android and XR are in beta and are not yet covered by the SDK’s test suite.
The voice comes from ElevenLabs’ cloud service, so each conversation needs a connection and an ElevenLabs account. The blog post does not give pricing or latency figures, and we have not measured either.
Limits
The README carries a caution: this is an early-stage SDK and APIs may change without notice. On WebGL some AudioSource settings, such as mixer routing and doppler, do not carry over, and Play Mode does not work while WebGL is the active build target.
For shipping, ElevenLabs says to switch authentication back on. The game should fetch a short-lived signed URL from your own server, and the API key must never go into a build.
Our take
Most talking NPC demos stop at speech. What makes this SDK interesting is the two-way link to the game: the character can be told what the world just did and can change the world in return. That is the difference between a voice on a chatbot and a character who belongs to the game, which we cover in what makes an NPC intelligent.
Two problems remain for developers. Live voice adds a network dependency, and a character who hears game events still needs durable memory to stay consistent across a whole playthrough, as we discuss in memory in AI games. A related approach runs everything on the player’s device, as in Atelico’s on-device engine.