AI NPC Experiments prototype media from the project.
Section 03
Project Overview
This project was built to integrate a production-ready Conversational AI SDK into Unity 6, transforming static NPCs into responsive, real-time conversational agents. The goal was to bridge cloud-based multi-modal AI with interactive 3D gameplay without sacrificing frame budgets or game feel.
My work focused on engineering the core runtime integration and event pipeline: managing low-latency WebRTC audio streaming, synchronizing multi-modal embodiment including viseme lip-sync, gaze, and facial emotions, and routing autonomous gameplay action commands directly into Unity gameplay systems.
Section 04
Engineering Challenge
The core challenge was coordinating high-frequency streaming network data with Unity's frame loop while maintaining strict performance budgets, zero-allocation memory constraints, and deterministic gameplay states.
01
Create a zero-allocation event dispatching pipeline to eliminate garbage collection spikes during continuous audio streaming.
02
Synchronize streaming TTS network audio with facial viseme blendshapes within a strict sub-300ms latency window.
03
Build a decoupled action execution pattern allowing generative AI to autonomously drive NavMesh movement, gaze, and animations.
04
Resolve conversational focus across multiple simultaneous NPCs without dialogue flickering or abrupt interruptions.
Section 05
System Architecture
The architecture follows strict Domain-Driven Design principles, cleanly decoupling low-level network streaming from high-level gameplay entities through an abstract domain event bus.
MODULE 01
Low-Allocation Event Bus (EventHub)
A thread-safe, copy-on-write event broker managing runtime domain events with O(1) lock-free dispatching and main-thread task marshaling.
MODULE 02
Real-Time Audio & Transport Layer
Integrates native LiveKit WebRTC streaming, microphone capture, hardware audio resampling, and bidirectional Opus channels.
MODULE 03
Embodiment & Affective Pipeline
Drives real-time facial blendshapes for lip-sync visemes, three-dimensional emotion values across valence, arousal, and dominance, and procedural eye gaze.
MODULE 04
Action Dispatcher & Archetypes
An extensible command executor supporting 21+ gameplay archetypes including NavMesh locomotion, gestures, object observation, and UnityEvents.
Section 06
Gameplay Flow
Conceptual conversational lifecycle based on the runtime integration and gameplay pipeline implemented for this system. It shows how voice input becomes synchronized NPC embodiment and gameplay action execution.
STEP 01
Player voice input
Microphone capture receives the player's conversational input in Unity.
↓→
STEP 02
WebRTC transport
LiveKit carries bidirectional audio through the real-time transport layer.
↓→
STEP 03
Domain event distribution
EventHub routes streaming and conversation events between runtime systems.
↓→
STEP 04
AI response stream
The Conversational AI runtime produces speech, emotion, and gameplay action data.
↓→
STEP 05
NPC embodiment
Audio timing anchors visemes while gaze and facial emotion update the character presentation.
↓→
STEP 06
Action execution
The action dispatcher invokes typed gameplay commands such as movement, gestures, and observation.
↓→
STEP 07
Gameplay feedback
Unity gameplay state and player context feed back into the next conversational interaction.
Section 07
Technical Implementation
I developed the runtime integration and gameplay systems required to make autonomous conversational agents responsive, performant, and extensible.
01
Copy-on-Write Event Broker
Implemented a low-allocation EventHub utilizing immutable subscriber arrays and weak-reference pruning, achieving zero runtime GC pressure during speech.
02
WebRTC Audio Streaming & Lip-Sync Anchors
Configured native LiveKit FFI streaming and tied blendshape viseme evaluation directly to AudioSource.timeSamples to eliminate audio-lip desynchronization.
03
Decoupled Action Executor Pattern
Authored the IConvaiActionExecutor command architecture with typed parameters and strict lifecycle completion guarantees covering success, failure reasons, and timeouts.
04
Spatial Conversation Targeting Heuristics
Engineered gaze-cone angle checks, proximity thresholds, and hold delays through switchDelaySeconds to reliably arbitrate player focus in multi-NPC scenes.
Section 09
Results & Key Learnings
Outcomes
✓Integrated a low-latency Conversational AI pipeline operating with < 0.2ms main-thread frame overhead in Unity 6.
✓Achieved sub-300ms turn response latency with zero main-thread GC allocations during active speech turns.
✓Implemented 21+ autonomous gameplay action archetypes, bridging generative AI decision-making with Unity physics and NavMesh locomotion.
Key Learnings
→How to design low-allocation Copy-on-Write data structures to handle high-frequency network data in Unity without triggering GC spikes.
→The importance of hardware timeline anchoring when synchronizing procedural animations with streaming network audio.
→How spatial hysteresis and hold timers prevent jitter when managing player attention across multiple game entities.
→Why strict command completion guarantees are essential when driving game behavior trees and state machines from external AI models.
