Interactive AI Characters with Voice Synthesis

The brief

One of the most excitiing uses of LLM technology is to create and orchestrate artificial characters for entertainment or artistic purposes. Nobody gets excited about interacting with a sales droid, but a phone-call from the future with a member of the eco-rebel alliance is something entirely different!

Likewise, if you're at the museum or art gallery and you want to deep dive on a topic, a warm, informative, high-bandwidth voice chat is far more useful than a text chat and more information efficient than static placards or infographics.

The brief here was a web app based interactive artwork where you make walkie-talkie calls to people from the future to find out what things are like in their time. The work was an immersive, participatory role-playing adventure about collective action, social movements and hope in the face of climate crisis and political despair.

Artists are used to fine-grained control over their work. They wanted an admin system where they could not only create detailed character and subject prompts for each call, but also direct the flow of conversation in a semi-scripted way, and customize the texture, tone, and mood of the agent voices.

So that's what we built. A unique system where you can call people from the future with their own unique sci-fi backstory, attitude, quirks, and opinions.

How it works

To get this working reliably and with low latency we leveraged a couple of different inference API endpoints. The user would hold down the "speak button", walkie-talkie style to trigger WebAudio recording, and this audio was then mp3 encoded when they let go of the button.

Next the mp3 was base64 encoded and appended to the running conversation history data structure, along with the global rules and character back-story/persona prompt. This payload was then sent to a multi-modal LLM endpoint (`gpt-4o-audio-preview`) running on OpenRouter.

The LLM would then send an in-character response in plain text. The system prompt directed the LLM to inject dramatic pauses as appropriate using special placeholder tokens. At this point if the LLM decided the conversation had reached a natural endpoint it could use the `hang_up_call` tool, passing a `goodbye_message` back and the user would see the call end.

The text response generated by the LLM was then synthesized into audio using the `minimax/voice-cloning` model on Replicate, and the voice reply was then played back to the audience member using WebAudio in the web app.

The voice-cloning model gave the artists a huge amount of flexibility to create their own voices. They did this by recording themselves and then tweaking the pitch, speed, and emotion parameters to tweak the mood of the speaker. We built a custom admin backend where they could input test response texts, tweak the parameters of the voice synthesis, and run full semi-scripted conversations.

Every point in this pipeline was optimized for efficiency and fast response times. To simulate the time-travel aspect we covered any latency with a spooky futuristic radio static sound.

In addition to audio we addressed accessibility by using the transcribed LLM text to show subtitles synchronized to audio playback progress using the WebAudio `currentTime` parameter. This meant the artwork was able to be experienced by a wide variety of audience members.

Results

Audience members loved it. People were able to suspend disbelief and talk to the characters as if they really were people calling from a distant, exotic future. The voice interactions were reliable, low-latency, and worked at scale with up to 60 participants in the work at the same time. By carefully selecting providers and endpoints, and doing a lot of work on-device, we were able to keep per-call costs low and audience engagement high.

This was a super fun job to work on and the architecture is directly applicable to any solution requiring real-time voice chat with arbitrarily specified characters such as artworks, entertainment, training, museums, galleries, guided tours, etc.

If you're interested in similar agentic voice and character solutions, please feel free to get in touch.