Contact centres automate the easy eighty percent of contacts and route the rest to a human. That leaves agents handling only the hard calls — angry customers, tangled billing histories, churn risk, compliance exposure — while juggling a CRM, a knowledge base, and a policy manual, in real time, with a person waiting on the line. The tooling most agents get is a search box. A search box does not tell you that this customer disputed a charge six months ago, that they have just crossed from frustrated into a churn threat, or that leading with a goodwill offer right now will read as a bribe rather than an apology.
← Back to portfolio// project showcase
Real-time AI copilot for contact-centre agents
Agent Copilot
A working prototype of a real-time copilot for the calls that reach a human after automation has handled the easy ones — the complex, emotional, multi-system contacts. A scripted billing dispute plays out turn by turn, and on every turn the model reads the transcript so far and derives five panels live: how the customer is feeling, whether the agent is staying inside policy, a running brief on who they're talking to, the single most relevant knowledge-base article, and one concrete next move. When the call ends it drafts the after-call wrap-up. The copilot never speaks to the customer — it augments the agent. This is deliberately a capability proof rather than a product: one call, played cleanly, with real model reasoning behind every panel. It is not integrated with live systems and does not handle live audio, and the architecture is built so that those are swaps rather than rewrites.
// the story
Problem, approach, result
Next.js 14 (App Router, TypeScript) with a single server route that holds the Anthropic key and calls Claude Haiku 4.5. Every turn, the client posts the turns so far and the server runs five derivations in parallel, plus a sixth on the final turn to draft the wrap-up. The design constraint that makes it honest: the call script contains only the conversation. No sentiment scores, no article choices, no advice are pre-written — everything on screen is derived by the model from the transcript, which is why the panels are worth watching rather than a slideshow. Each derivation returns strict JSON, is parsed defensively, and has its own fallback, so a single bad response degrades one panel instead of the screen.
A prototype that demonstrates the whole pattern end to end: live multi-panel reasoning over a running transcript, retrieval that cannot hallucinate its own source text, per-panel graceful degradation, and a human kept firmly in the loop on both the customer-facing and the record-keeping side. It's the shape of a real deployment — swap the scripted transcript for streaming speech-to-text and the JSON fixtures for a real CRM and knowledge base, and the copilot logic is unchanged.
// architecture
How it fits together
The client plays a scripted call from data/call-script.json and, on each turn, POSTs the turns so far to /api/copilot. That route is the only place the API key lives; it loads the customer profile and knowledge-base articles from JSON fixtures and hands everything to deriveCopilotState(). That orchestrator renders the transcript to plain text once and fans out six Claude Haiku calls under a single Promise.all — sentiment, compliance, customer context, knowledge retrieval, next best action, and (only when the turn is flagged final) the after-call wrap-up. Each derivation owns its own system prompt, its own JSON contract, and its own try/catch fallback value. The seams are deliberate: the transcript source, the data fixtures, and the model calls are three independent layers, so replacing the first two doesn't touch the third.
// engineering deep-dives
Decisions I'm proudest of
The script contains only the conversation
This is the decision the whole prototype rests on. It would have been far easier — and completely worthless — to pre-write the sentiment scores, the article choices, and the advice into the call script and play them back on cue. Instead the script holds nothing but agent and customer turns. Every number, every flag, every suggested action on screen is derived live by the model from the transcript up to that moment. That's what makes it a copilot demo rather than an animation, and it's why an off-looking panel is something to acknowledge in a demo rather than hide: it's the model reasoning on real input, and in production those prompts get tuned against real call data.
Retrieval that cannot invent its own source
The knowledge panel is grounded by construction, not by asking nicely. The model is shown the article catalogue and is only permitted to return an id — kb-003, and nothing else. The application then looks that id up in the real record and renders the stored tag, title, and body. The model chooses; it never authors. So the article text on the agent's screen physically cannot drift from the knowledge base, and an unrecognised id falls back to a known article rather than rendering something invented. It's the same fact-versus-commentary discipline as ScamCheck's escalate-only scoring and Medical Aid Navigator's deterministic emergency layer: give the model the judgement call, never the source of truth.
Parallel by default, and Haiku on purpose
All six derivations fire concurrently under one Promise.all, so a turn costs roughly the latency of the slowest panel rather than the sum of six. The model choice is part of the same argument, and it's written into the code as a comment rather than left implicit: Haiku is chosen because on the live path latency matters more than depth. A more capable model that answers after the moment has passed is worse than useless on a call — the agent has already spoken. That's a real engineering trade, made explicitly and documented where the next reader will find it.
Every panel degrades on its own
An LLM returning malformed JSON mid-call is a certainty, not a risk, so no panel is allowed to take the screen down with it. Each derivation strips stray code fences, parses defensively, and returns a sensible fallback on failure: sentiment falls back to a neutral read, context falls back to the customer's stored profile and history, knowledge falls back to a real article. Enum fields are validated against a whitelist rather than trusted — an unrecognised tone becomes neutral, an unrecognised compliance level becomes info — and unbounded arrays are capped before they reach the layout. The result is a copilot that can have a bad turn without having a bad call.
The human stays in the loop, on both sides
Two boundaries, drawn deliberately. The copilot never speaks to the customer: it has no channel to them, it only ever advises the agent, and the agent decides what to do with the advice. And the after-call wrap-up — outcome classification, factual summary, action tags — is drafted for agent review before submission, not auto-filed. Getting the summary from a blank box to an editable draft is where the time actually goes; auto-filing it would hand an unreviewed model output to the compliance record. The compliance panel points the same direction: it monitors the agent's conduct against authorisation limits and disclosures, which is the thing a QA reviewer would flag, and it's instructed to report that the agent is acting correctly rather than manufacture an issue to look useful.
// tech & skills
Built with
- Next.js 14 (App Router)
- TypeScript
- Anthropic Claude (Haiku 4.5)
- Anthropic SDK
- Parallel LLM orchestration
- Structured JSON contracts + defensive parsing
- Grounded retrieval (id-only selection)
- Prompt engineering as guardrails
- Graceful degradation / per-panel fallbacks
- Human-in-the-loop design
- Vercel serverless
// engineering process
How it was built
Designed and built solo as a bounded capability proof: one scripted call, played cleanly, with live model reasoning behind every panel. The scope exclusions are the point — no live audio, no real system integrations, no multi-call handling — and the code is layered so each of those is a swap rather than a rewrite. lib/copilot.ts holds all five prompts plus the wrap-up in a single reviewable file, each with its JSON contract and fallback beside it, and the comments record why each choice was made rather than restating what the code does.
// proof points
Why it matters
Real-time AI applied where it earns its keep: the hard calls that reach a human after automation, augmenting an expensive skilled agent rather than replacing them.
Grounding enforced structurally, not by prompt alone — the model selects an article id and the application supplies the text, so displayed knowledge cannot be fabricated.
Production instincts in a prototype: concurrent derivation, a documented latency-over-depth model choice, per-panel fallbacks, whitelist-validated enums, and an after-call draft that a human reviews before it becomes a record.
Honest scoping: what it proves and what it deliberately doesn't do are both stated up front, and the architecture is layered so the missing pieces are swaps rather than rewrites.
// roadmap
What I'd build next
- Swap the scripted transcript for a streaming speech-to-text feed — the copilot already consumes only "turns so far", so the derivation logic doesn't change.
- Replace the JSON fixtures with real CRM and knowledge-base integrations behind the same interfaces.
- Stream each panel's result as it lands instead of awaiting all six, so the fastest panels paint first.
- Build an eval set from real call transcripts and tune the five prompts against it, with regression tests on the compliance panel in particular.