Building a Haunted House with NVIDIA Nemotron Open Models
Building a Haunted House with NVIDIA Nemotron Open Models MonsterHouse: A grumpy, interactive Halloween projection MonsterHouse is an interactive Halloween projection designed t...
By Hardware Team
Building a Haunted House with NVIDIA Nemotron Open Models
MonsterHouse: A grumpy, interactive Halloween projection
MonsterHouse is an interactive Halloween projection designed to listen to trick-or-treaters, answer in the voice of a grumpy old pirate ship, and become more welcoming when visitors remember their manners. The system uses the NVIDIA Nemotron Voice Agent with NVIDIA Nemotron ASR, Nemotron 3.5 Bolt E4B 4GB, Magpie TTS Zeroshot, and Audio2Face-3D. It runs on a single workstation.
The project was designed to give children visiting the house a distinctive interaction. Instead of making the house frightening, the concept was a building that controlled the treats while complaining about its visitors. The result was a garage door with a mouth, a pirate-ship backstory, a fiery grin, and a preference for the word “please.”
NVIDIA Nemotron Voice Agent provides the conversational foundation. Nemotron Speech handles listening, Bolt E4B generates replies, NVIDIA Magpie TTS Zeroshot supplies the live voice, and NVIDIA Audio2Face-3D animates the wooden mouth.
How MonsterHouse works
- Listening: NVIDIA Nemotron Speech ASR converts visitor speech into text. The confirmed model file is
nemotron-speech-streaming-en-0.6b.q8_0.gguf. - Thinking: NVIDIA Nemotron 3.5 Bolt E4B 4GB, an open small language model for conversational agents, generates each reply. A LoRA adapter provides the character’s personality.
- Speaking: NVIDIA Magpie TTS Zeroshot generates speech from a short reference sample.
- Animating: NVIDIA Audio2Face-3D converts clean speech into mouth animation in Unreal Engine 5.
- Seeing: NVIDIA Nemotron 3 Nano Omni describes costumes, while YOLO and ByteTrack count and track visitors.
- Running the system: NVIDIA Nemotron Voice Agent coordinates the components on one workstation with an NVIDIA RTX PRO 6000 Blackwell Max-Q and an NVIDIA RTX PRO 4000 Blackwell.
A grin full of splinters
The mouth was the first major design element. The garage door boards were imagined as jagged wooden lips, with flames illuminating the driveway behind them whenever the mouth opened. The character needed to look ancient, expressive, and mildly offended by the presence of visitors.
Creating a custom animation rig was one of the more labor-intensive parts of the project. After the rig was completed, the facial expression shapes could be exported for Audio2Face-3D to animate. The character’s skeletal mesh was built in Autodesk Maya and exported to Unreal Engine, where virtual fire and other effects were added behind the door. With no eyes, eyebrows, or blinking, the character required relatively few facial components.
The NVIDIA Nemotron Voice Agent simplified the remaining work by providing listening, conversation, and speech capabilities. Once the house had a mouth, the voice pipeline gave it something to say.
Teaching bad manners
The character’s backstory explains its cantankerous personality. The house was built from the timbers of a sunken pirate ship and remains haunted by its stormy past. It is effectively a ship reincarnated as real estate and unhappy about the change.
The interaction follows a conversation path from greeting to farewell while allowing the house to distribute treats. At first, it dismisses visitors with lines such as “I’m busy grumbling, go bother the neighbors!” Persistence gradually changes its response. Saying “Please” can produce an immediate improvement, while “thank you” earns a warmer farewell. Once a visitor wins the house over, that state is retained for the rest of the encounter.
The Voice Agent supports the use of different language models. Because the system was intended to operate off-grid and remain self-contained, memory and compute requirements were important constraints.
A simple prompt asking a model to “be grumpy” was not sufficient. The character needed to follow a conversation flow, respond to what visitors actually said, track the state of each encounter, and leave enough computing capacity for the rest of the pipeline.
After testing more than a dozen models, the project used Bolt E4B for live dialogue. It is an open small language model designed for local conversational agents and interactive characters. The 4-bit quantized release, identified by “4GB” in its name, is the default response model. The BF16 checkpoint is available for comparison.
The system uses Bolt E4B within the Nemotron Voice Agent pipeline to turn visitor speech into the next reply. Prompts, application logic, and a LoRA adapter trained on thousands of reviewed example conversations shape the general model into the specific pirate character.
The dataset contains more than 5,000 multi-turn conversations. LoRA adapters were trained for the models considered viable, with training runs taking approximately one to two hours. A local agent handled the process and benchmarked the results. The dataset was initially created and evaluated by that agent and a large language model.
The replies were reviewed for continuity, timing, tone, repetition, quality, and speed. The goal was for the house to eventually give in without becoming a cheerful concierge.
A voice worth waiting for
A haunted house loses its effect if it takes too long to respond. The pipeline streams speakable text before an entire answer is complete, allowing the house to begin speaking while the rest of the sentence is generated. Prepared greetings and candy-related jokes cover familiar interactions. A text bank reduces language-model work, while cached recordings avoid speech synthesis for common lines.
Median end-of-speech-to-audio with Bolt Q4 and TTS: 1.70 seconds
Median end-of-speech-to-audio with cached responses: 0.72 seconds
Note: speech-end detection currently waits for 300 ms of silence.
NVIDIA Magpie TTS Zeroshot generates the character’s voice using a short reference sample rather than a separately trained voice. MonsterHouse has seven moods, ranging from grumpy and menacing to amused and delighted. Emotion tags from the prompt template, selected by the language model, determine how each line is delivered.
Each emotion uses a different three- to ten-second voice sample. A “Sample your Voice” block in the interface allows a sample line to be recorded in different pirate voices, such as happy or angry. During an interaction, Bolt E4B selects the dialogue and Magpie converts it into speech with the appropriate emotion.
The next stage is Audio2Face-3D and the 3D character scene created in Epic’s Unreal Engine. Audio2Face-3D’s default model weights work best with a clean, neutral male or female voice that is neither too deep nor too high. MonsterHouse uses a very deep voice with echo and chamber effects. Clean speech drives the facial animation, while a separate audio stream adds adjustable vocal effects for the speakers.
The workload is divided between two GPUs. The language and voice pipeline runs on an NVIDIA RTX PRO 6000 Blackwell Max-Q, while Unreal Engine graphics and Audio2Face-3D run on an NVIDIA RTX PRO 4000 Blackwell. The main constraint is compute rather than memory. Separating the workloads helps maintain a consistent animation frame rate while the agent pipeline processes requests.
Voice, animation, and effects must remain synchronized. Audio delivery can be adjusted, including a 100-millisecond compensation for Audio2Face-3D buffering.
The monster has a control desk
The project eventually required an interface with controls for auditioning cached lines, adjusting the opening mood, following the conversation, changing voice treatment, and tuning mouth animation.
The interface includes tabs for conversations, model selection, recordings, prompts, performance monitoring, and camera feeds. A soundboard and custom dialogue field provide additional control over what the character says. Model-loading controls and an Unreal Engine connection indicator show whether the rendering system is connected.
Visual effects respond to delivery through editable rules. Live cues begin with speech, while recorded performances can follow a saved timeline. Mood changes can therefore be made without rebuilding the scene.
Several operating modes are available:
- Live mode: Runs the trick-or-treat interaction.
- Demo mode: Has the house explain itself.
- Microphone mode: Sends the operator’s speech directly through the house, bypassing recognition, dialogue generation, and speech synthesis. Clean speech drives the mouth while the audience hears the treated version.
Visual context for the next visitor
NVIDIA Nemotron 3 Nano Omni is used as an optional visual-language component. It interprets camera images and describes visible costumes so the character can respond to specific visual details.
YOLO and ByteTrack handle people detection and tracking separately. This allows the system to count and track a group without asking Omni to analyze every video frame.
For a test, Omni analyzed a video of five costumed sheep at the door. Its saved analysis identified pirate, vampire, witch, pumpkin, and ghost outfits. The sheep were a synthetic test rather than evidence from an actual Halloween night. Real groups, costumes, and night lighting still require testing. Detection can trigger automatic greetings, but costume identification and contextually appropriate replies may require additional training.
Further work includes improving the animation, completing performance testing with real visitors, and automating candy delivery. The original goal was to let the system run Halloween while the operator visited other homes with family members, but that goal had not yet been reached.
Building another interactive character
The MonsterHouse code is not packaged for release, but the NVIDIA models used by the project are available, and NVIDIA Nemotron Voice Agent provides much of the required infrastructure.
-
Start with the voice pipeline. Clone the NVIDIA Nemotron Voice Agent and run the Generic Assistant example. It provides a cascaded pipeline with Nemotron Speech ASR, an LLM, Magpie TTS, voice activity detection, and end-of-utterance handling.
-
Use Bolt E4B as the language model. The Voice Agent includes Bolt E4B profiles for cloud, server, and single-GPU deployments. The model can also be tested on build.nvidia.com. Conversational mode is intended for short, speech-ready replies, and reasoning can be disabled for simpler exchanges.
-
Define the personality. Begin with a system prompt and several connected example dialogues. Keep state, such as whether a visitor has already said “please,” in the application rather than relying on the model to remember it. When prompting is insufficient, a LoRA adapter can be trained. NVIDIA NeMo AutoModel includes a LoRA recipe for Bolt E4B.
-
Add a voice. Magpie TTS Multilingual is the Voice Agent default. For a custom character, Magpie TTS Zeroshot works from a short reference sample. Voices should only be used with the necessary permissions.
-
Add a face. NVIDIA Audio2Face-3D converts speech into facial animation. Clean speech can drive the animation, with effects added afterward.
-
Optionally add visual awareness. The Voice Agent’s Nemotron Omni Assistant Subagents example adds webcam understanding with Nemotron 3 Nano Omni. A lightweight people tracker can be added when visitor counts are needed.
Cached responses and saved audio for common lines can reduce perceived latency. A system with limited computing capacity can use Nemotron Speech ASR, a Python conversation router, and cached audio. Bolt E4B and Magpie can generate many responses offline, which can then be used during the live performance.