MonsterHouse is an interactive Halloween projection designed to make a house the center of the trick-or-treating experience. Instead of simply displaying decorations, the garage door listens to visitors, answers as a grumpy old pirate ship and gradually becomes friendlier when they use good manners.
The system is built around NVIDIA Nemotron Voice Agent. Nemotron 3 ASR converts visitors’ speech into text, while the 4-bit version of Nemotron 3.5 Bolt E4B generates the replies. A LoRA adapter gives the language model the personality of a cantankerous building made from the timbers of a sunken pirate ship.
Magpie TTS Zeroshot converts each response into speech using short reference samples recorded for different moods. The house can switch between seven emotional styles, ranging from grumpy and menacing to amused and delighted. Audio2Face-3D then turns the clean speech signal into mouth animation inside an Unreal Engine 5 scene, while a separate audio stream adds the deep, echoing effects heard by visitors.
The interaction follows a planned progression. At first, the house may dismiss trick-or-treaters with complaints, but persistence can win it over. Saying “please” can change the exchange immediately, while “thank you” earns a warmer farewell. Conversation state is tracked by the application so that the house remembers how an encounter has developed.
The creator tested more than a dozen language models before choosing Bolt E4B for live dialogue. A dataset containing more than 5,000 multi-turn conversations was used to train and assess LoRA adapters. The system also uses prompts and application logic to maintain the character’s tone, continuity and timing without making the house endlessly argumentative.
Speed was another important consideration. The pipeline can release text before a complete response has been generated, allowing the house to begin speaking while the rest of the sentence is still being produced. Prepared greetings and cached audio handle common encounters more quickly. The reported median delay from the end of a visitor’s speech to generated audio was 1.70 seconds with Bolt Q4 and TTS, compared with 0.72 seconds for cached responses. Speech-end detection currently waits for 300 milliseconds of silence.
The visual system also includes optional AI-based perception. Nemotron 3 Nano Omni can describe costumes from camera images, while YOLO and ByteTrack count and track people in view. In a synthetic test using five costumed sheep, the saved analysis identified pirate, vampire, witch, pumpkin and ghost outfits. The test was not evidence of performance with real Halloween visitors, and the project still requires testing under real driveway conditions and night lighting.
MonsterHouse runs on a single workstation, with the language and voice pipeline assigned to an NVIDIA RTX PRO 6000 Blackwell Max-Q and Unreal Engine graphics with Audio2Face-3D handled by an NVIDIA RTX PRO 4000 Blackwell. Splitting the workload between the GPUs is intended to keep the animation frame rate stable while the conversational pipeline is processing.
A custom control interface provides access to cached lines, opening moods, conversation flow, voice treatment, mouth animation, camera feeds and performance monitoring. It also includes live, demo and microphone modes. In microphone mode, a person can speak through the house directly, bypassing speech recognition, dialogue generation and speech synthesis while still driving the mouth and applying the voice effects.
The project’s code is not packaged for release, but the NVIDIA models used in the system are available. The suggested starting point is the NVIDIA Nemotron Voice Agent’s Generic Assistant example, which combines speech recognition, an LLM, Magpie TTS, voice activity detection and end-of-utterance handling. From there, developers can use Bolt E4B for local dialogue, add a personality through prompts or a LoRA adapter, give the character a voice with Magpie TTS Zeroshot and animate it with Audio2Face-3D.
The creator says the project still needs further animation work, complete testing with real visitors and an automated way to distribute candy. The longer-term goal is to let the house run Halloween independently, while cached responses could help similar installations operate on more limited hardware.
Sources: Hugging Face Blog.