Talha Malik
0
AI
Case Study · FalconXoft

Gioconda AI

It started as a phone call. It became the system a business runs on.

My role

I led the AI layer — the voice architecture, the agent and tool system, retrieval and grounding, and the guardrails around what the agent is allowed to do. I also ran delivery across the wider platform as it grew from a single call flow into the product described below.

VOICE AGENTS ✦ MULTI-TENANT SAAS ✦ OPENAI REALTIME ✦ WEBRTC ✦ STRIPE CONNECT ✦ RAG ✦ VOICE AGENTS ✦ MULTI-TENANT SAAS ✦ OPENAI REALTIME ✦ WEBRTC ✦ STRIPE CONNECT ✦ RAG ✦
See it in action
Where it began

One problem: businesses miss calls.

Not occasionally — constantly. After hours, during service, when everyone is already on the line. Every missed call is a booking that goes to whoever picks up next.

The first version of Gioconda did one thing. It answered the phone.

Everything that follows came from a pattern that repeated over and over: solve one problem well, and the next problem immediately becomes visible.

Part One

Making the call work.

01The pipeline

Speech to text → LLM → text to speech. Audio was transcribed, the model generated a reply, ElevenLabs spoke it. It worked. It just wasn't good enough for a real phone call. Latency stacked — each hop added its own delay and they compounded. Turn-taking was mechanical; the system had no sense of when someone had finished a thought versus paused mid-sentence.

02Optimising the parts

ElevenLabs moved to websocket streaming; Hume AI was added for emotion analysis, then pulled a month later once it added complexity without moving the number that mattered. Neither fixed the underlying problem — latency was never coming from one slow component. It came from the shape of the system: three sequential handoffs, each waiting on the one before it.

“When optimising the parts stops moving the number, the problem is the architecture.”

03Speech to speech

The fix was to delete the pipeline. Moving to OpenAI Realtime collapsed the chain — the model receives audio and produces audio directly, no separate transcription or synthesis step. Understanding improved as a side effect, because there was no longer a lossy text bottleneck between what the caller said and what the model understood.

Under 500ms response time.

Below roughly half a second a caller experiences it as conversation. Above it, they experience it as waiting.

The trade-off: less granular control over voice selection than a dedicated TTS provider offers. Worth it — a perfect-sounding voice that replies three seconds late is worse than a good voice that replies instantly.

Part Two

Making it useful.

From calling APIs to giving the AI tools
Before

The backend parsed what the caller wanted, then decided which API to call. Each capability a hardcoded path. The AI produced text; the code decided what to do about it. This breaks the moment a caller does something unanticipated.

After

The model gets tools and calls them itself — check availability, create a booking, update customer details mid-conversation. Adding a capability became adding a tool, not rewriting the conversation flow.

Deciding where autonomy stops

Not every action is equally reversible. Reading a calendar is safe. Confirming a quote to a customer is not. So tools are classified — a subset marked sensitive requires explicit confirmation before the agent can execute it.

“The instinct with agents is to give them everything and trust the model. The correct instinct is to give them everything reversible, and put a gate in front of the rest.”

Part Three

Making it know things.

Asking every client to upload documents fails in practice — businesses don't have a tidy knowledge base. What they have is a website. So Gioconda grew a crawler: it merges multiple discovery sources, ranks and crawls pages in parallel, and converts them into clean, model-ready markdown rather than raw HTML noise. Progress streams live, because a silent multi-minute process feels broken even when it's working.

Then the interesting part. Once you've crawled a business's website, you don't just have documents — you have structure: locations, staff, services. That became one-click setup: point Gioconda at a URL, and it provisions the knowledge base and staff records automatically.

Locations were deliberately left out of that flow — the crawl needed to resolve them was slow enough to break the "one click" feeling, so it was dropped rather than allowed to make onboarding worse. Not every extractable field is worth extracting.

Part Four

From answering to selling.

A booking agent that knows the business gets asked the obvious next question: how much will it cost? That produced the quote engine — a conversational path from enquiry to a real, priced quote, gated by per-agent feature flags.

Then quotes needed to be paid. Stripe Connect meant payment flows through the business's own account, not a platform wallet — payment links dispatched over SMS and email. The feature outgrew its name and became GioCommerce.

Part Five

Beyond the phone.

The phone only works when someone calls. So the platform grew outward — each addition the same shape as the last: solve the problem in front of you, then notice what that reveals.

Sales outreach
Lead lists, custom fields, scheduling with start/end dates, calling windows, a conversational sales methodology engine.
SMS
Shared multi-provider number pool, inbound reply tracking, opt-in and consent disclosures wired into signup.
Email
Campaign wizard with merge tags, per-recipient CTA click tracking, overage billing beyond included sends.
WhatsApp
Business Account connection with verification, strict status flags, a sweep system for drifted accounts.
Mobile
Companion app with its own session auth, voice endpoints, and settings summary.
Jobs & automations
Scheduled and triggered work with status tracking and transport validation.
GioDiscover
Prospect discovery with quality scoring, landing page publishing, sequence analytics, reply knowledge modes, and human escalation.
Part Six

Making it survivable.

Cost

The model tier moved from gpt-4o-realtime-preview to the smaller mini variant, then to gpt-realtime once generally available. A realtime cost manager wired directly into billing attributes spend as it happens.

“The expensive model is only worth it where it changes the outcome.”

Transport

The voice system migrated to WebRTC, which handles real-time audio far better under packet loss and jitter — the ordinary reality of a phone connection. It runs parallel to Socket.io so browser testing and live telephony share the same call logic.

Security

A dedicated hardening programme covering reconnaissance defence, boot preflight checks, internal-UI gating, and stable error codes.

A bug that taught me something

Two tenants' knowledge bases lived in a single Postgres row. Two processes read and wrote it concurrently — one silently overwrote the other. No error, no crash, nothing in the logs. The affected tenant's agent simply stopped answering from their own documents, and kept talking confidently while doing it.

Fixed with row-level locking (SELECT FOR UPDATE) so concurrent writes serialise. That class of bug never appears in testing — it needs real concurrent load to surface, and it fails silently when it does.

Stack
Python · FastAPI · Node · React · OpenAI Realtime · WebRTC · Socket.IO · Twilio · Stripe Connect · PostgreSQL · Redis · Pinecone · Crawl4AI · Docker · Multi-LLM fallback · Python · FastAPI · Node · React · OpenAI Realtime · WebRTC · Socket.IO · Twilio · Stripe Connect · PostgreSQL · Redis · Pinecone · Crawl4AI · Docker · Multi-LLM fallback ·
The arc, in one line

Answer the phone → make it fast → give it tools → decide where autonomy stops → teach it the business from its own website → let it quote and take payment → follow the customer onto every other channel → then make the whole thing cheap and reliable enough to run every day.