miniVoxSetu
Real-Time Multimodal Voice AI Agent for Indian Banking
- Engineered an ultra-low latency multimodal voice AI agent combining streaming STT/TTS, semantic analysis, acoustic emotion detection, and asynchronous processing; reduced end-to-end latency from 1,250ms to ~390ms.
- Built and evaluated a hybrid RAG engine combining BM25 and vector search with Reciprocal Rank Fusion (RRF); improved retrieval Hit@1 from 68.4% to 93.8% and achieved 100% Hit@3.
- Designed a 7-layer barge-in system using AudioWorklet VAD, client queue flushing, WebSocket interruption signals, generation-ID gating, and stale-task pruning with <280ms interrupt-to-silence latency.
- Built automated evaluation harness covering RAG retrieval, acoustic inference, latency benchmarking, and 1.000 F1 PII redaction on synthetic banking transcripts.
Python
FastAPI
WebSockets
Groq
Deepgram
Qdrant
PyTorch
AudioWorklet
Engineered Benchmarks
~390ms
End-to-end voice roundtrip latency
<280ms
Barge-in interrupt-to-silence latency
1.000 F1
PII security redaction on banking transcripts
Architecture: WebSocket Streaming + Qdrant Vector + BM25 RRF