Case Study
Paco: a voice AI that answers WhatsApp calls
Paco answers a normal WhatsApp voice call and holds a real-time conversation. It checks email, calendar, weather and the web while you talk, and it can call you back with a spoken briefing. I built it from an empty folder to a production deployment in a single day.
The problem
I wanted to talk to my assistant the way you talk to a person: by calling it. Not a chat window, not push-to-talk voice notes. A real WhatsApp voice call that gets answered, understood, and responded to with natural latency.
WhatsApp has no public API for answering voice calls, so the interesting engineering problem was building a reliable bridge between a WhatsApp call and a real-time speech model.
Architecture
An inbound WhatsApp call reaches the WaVoIP SDK running inside headless Chrome under Puppeteer. A Node.js bridge captures the call audio, resamples it between the 16 kHz telephony rate and the 24 kHz the model expects, and streams it to the OpenAI Realtime API over WebSocket. The model's spoken reply streams straight back to the caller.
Outbound works too: a POST to the bridge's HTTP API starts a WhatsApp call to my number and delivers a spoken briefing built from live data.
The build, hour by hour
The whole system went from nothing to production on 1 March 2026. The timeline comes straight from the project log:
Running it for real
Paco runs as a daemon on my own Ubuntu server behind Nginx, deployed with PM2 and monitored through a custom operations dashboard. Mid-call, the model calls out to tools for email, calendar, weather and web search, so answers reflect live data rather than training data.
Reliability, security and what's next
Reliability. The bridge handles the messy parts of live telephony: it resamples audio in both directions between the 16 kHz call and the 24 kHz the model expects, and it manages the WebSocket stream so speech flows without gaps. Running under PM2 means the process is supervised and restarts if it falls over, and the operations dashboard surfaces call state at a glance.
Security. Paco is self-hosted on my own Ubuntu server behind Nginx with TLS. Call handling stays on infrastructure I control, and the assistant reaches the outside world only through the specific tools it is given for email, calendar, weather and web search.
What I would improve next. Add call transcripts and structured logging for every session, a fallback voice path if the realtime stream drops mid-call, and per-caller authorisation so outbound briefings can be opened up safely beyond my own number.