It started as a phone call. It became the system a business runs on.
I led the AI layer — the voice architecture, the agent and tool system, retrieval and grounding, and the guardrails around what the agent is allowed to do. I also ran delivery across the wider platform as it grew from a single call flow into the product described below.
Not occasionally — constantly. After hours, during service, when everyone is already on the line. Every missed call is a booking that goes to whoever picks up next.
The first version of Gioconda did one thing. It answered the phone.
Everything that follows came from a pattern that repeated over and over: solve one problem well, and the next problem immediately becomes visible.
Speech to text → LLM → text to speech. Audio was transcribed, the model generated a reply, ElevenLabs spoke it. It worked. It just wasn't good enough for a real phone call. Latency stacked — each hop added its own delay and they compounded. Turn-taking was mechanical; the system had no sense of when someone had finished a thought versus paused mid-sentence.
ElevenLabs moved to websocket streaming; Hume AI was added for emotion analysis, then pulled a month later once it added complexity without moving the number that mattered. Neither fixed the underlying problem — latency was never coming from one slow component. It came from the shape of the system: three sequential handoffs, each waiting on the one before it.
“When optimising the parts stops moving the number, the problem is the architecture.”
The fix was to delete the pipeline. Moving to OpenAI Realtime collapsed the chain — the model receives audio and produces audio directly, no separate transcription or synthesis step. Understanding improved as a side effect, because there was no longer a lossy text bottleneck between what the caller said and what the model understood.
Below roughly half a second a caller experiences it as conversation. Above it, they experience it as waiting.
The trade-off: less granular control over voice selection than a dedicated TTS provider offers. Worth it — a perfect-sounding voice that replies three seconds late is worse than a good voice that replies instantly.
The backend parsed what the caller wanted, then decided which API to call. Each capability a hardcoded path. The AI produced text; the code decided what to do about it. This breaks the moment a caller does something unanticipated.
The model gets tools and calls them itself — check availability, create a booking, update customer details mid-conversation. Adding a capability became adding a tool, not rewriting the conversation flow.
Not every action is equally reversible. Reading a calendar is safe. Confirming a quote to a customer is not. So tools are classified — a subset marked sensitive requires explicit confirmation before the agent can execute it.
“The instinct with agents is to give them everything and trust the model. The correct instinct is to give them everything reversible, and put a gate in front of the rest.”
Asking every client to upload documents fails in practice — businesses don't have a tidy knowledge base. What they have is a website. So Gioconda grew a crawler: it merges multiple discovery sources, ranks and crawls pages in parallel, and converts them into clean, model-ready markdown rather than raw HTML noise. Progress streams live, because a silent multi-minute process feels broken even when it's working.
Then the interesting part. Once you've crawled a business's website, you don't just have documents — you have structure: locations, staff, services. That became one-click setup: point Gioconda at a URL, and it provisions the knowledge base and staff records automatically.
Locations were deliberately left out of that flow — the crawl needed to resolve them was slow enough to break the "one click" feeling, so it was dropped rather than allowed to make onboarding worse. Not every extractable field is worth extracting.
A booking agent that knows the business gets asked the obvious next question: how much will it cost? That produced the quote engine — a conversational path from enquiry to a real, priced quote, gated by per-agent feature flags.
Then quotes needed to be paid. Stripe Connect meant payment flows through the business's own account, not a platform wallet — payment links dispatched over SMS and email. The feature outgrew its name and became GioCommerce.
The phone only works when someone calls. So the platform grew outward — each addition the same shape as the last: solve the problem in front of you, then notice what that reveals.
The model tier moved from gpt-4o-realtime-preview to the smaller mini variant, then to gpt-realtime once generally available. A realtime cost manager wired directly into billing attributes spend as it happens.
“The expensive model is only worth it where it changes the outcome.”
The voice system migrated to WebRTC, which handles real-time audio far better under packet loss and jitter — the ordinary reality of a phone connection. It runs parallel to Socket.io so browser testing and live telephony share the same call logic.
A dedicated hardening programme covering reconnaissance defence, boot preflight checks, internal-UI gating, and stable error codes.
Two tenants' knowledge bases lived in a single Postgres row. Two processes read and wrote it concurrently — one silently overwrote the other. No error, no crash, nothing in the logs. The affected tenant's agent simply stopped answering from their own documents, and kept talking confidently while doing it.
Fixed with row-level locking (SELECT FOR UPDATE) so concurrent writes serialise. That class of bug never appears in testing — it needs real concurrent load to surface, and it fails silently when it does.
Answer the phone → make it fast → give it tools → decide where autonomy stops → teach it the business from its own website → let it quote and take payment → follow the customer onto every other channel → then make the whole thing cheap and reliable enough to run every day.