Engineering Atlas › Building AI Systems

Design an AI Support Assistant

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope: voice calls, proactive outreach, and changes to legal terms or account ownership.

Back-of-envelope

API

POST /conversations                  -> { conversation_id }
POST /conversations/{id}/messages    { text, client_msg_id }
  -> stream of events: token | citation | action_proposed | action_done | handoff
POST /conversations/{id}/handoff     { reason }
GET  /conversations/{id}             -> transcript, status, actions

Tools (server-side, customer_id from the session, never from the model):
  search_help(query, product?)                 -> passages with article ids
  get_orders(status?, limit<=10)               -> orders
  get_invoice(invoice_id)                      -> invoice
  issue_refund(order_id, amount, reason_enum)  -> refund (idempotency key = conversation_id:turn)
  update_setting(name_enum, value)             -> ok

Data model

High-level design

flowchart LR
  C["Customer chat"] --> API["Conversation service"]
  API --> ORCH["Orchestrator: workflow + bounded agent"]
  ORCH --> GW["LLM gateway: routing, limits, caching, tracing"]
  ORCH --> RET["Retrieval: hybrid search + rerank"]
  ORCH --> TOOLS["Tool service: orders, invoices, refunds"]
  TOOLS --> CORE["Core systems: orders, billing"]
  ORCH --> POL["Policy: limits, approvals, escalation"]
  POL --> HQ["Human queue with summary"]
  API --> STORE["Conversations, messages, actions"]
  GW --> OBS["Traces, cost, online evals"]

Write path (customer message): the conversation service stores the message (deduplicated by client_msg_id), then the orchestrator classifies intent with a small model and runs the matching workflow: questions go to retrieval plus answer generation with citations; account questions call read tools; refund requests propose an issue_refund action that the policy engine approves automatically within limits or routes to a human. Tokens stream back to the client as they are generated. Read path (agent console): human agents see the transcript, retrieved sources, actions and an AI-written summary when a conversation is handed off.

Deep dives

1. Grounded answers

Options: answer from the model's own knowledge; long context with all articles; retrieval. Choice: hybrid retrieval (BM25 plus vectors, fused, reranked) over chunks with title and section context (Anthropic measured top-20 retrieval failures falling from 5.7 percent to 3.7 with contextual embeddings, 2.9 with contextual BM25 added, and 1.9 with reranking, and found 20 passed chunks beat 5 or 10 (Anthropic)), filtered by product, locale and visibility; the answer must cite passages and say "I'm not sure" when they do not cover the question. A faithfulness check (does each claim map to a cited passage?) runs on a sample online and on every case offline. Retrieval recall at 10 is tracked separately from answer quality.

2. Safe actions

Options: let the agent call any API; human approval for everything; policy-based autonomy. Choice: narrow tools with customer identity from the session, enums for reasons and settings, idempotency keys per conversation turn, and a policy engine: refunds up to 50 dollars on the customer's own orders within 60 days are automatic, larger ones or VIP accounts need approval, and a daily cap per customer limits abuse. Every action is recorded in actions with arguments and approver. This is the OWASP "excessive agency" checklist in practice: minimal functionality, minimal permissions, the user's own context, human approval for high-impact actions, and checks enforced in the downstream system rather than by asking the model (OWASP LLM06). The idempotency key follows the Stripe pattern: the provider saves the first result per key, rejects a retry whose parameters differ, accepts keys up to 255 characters, and may prune them after 24 hours (Stripe docs), so keep your own actions row as the durable record.

3. Knowing when to hand off

Options: only on request; on fixed rules; on model judgment. Choice: combine signals: explicit request, low retrieval confidence or "not sure" answers twice in a row, negative sentiment after 3 or more turns, sensitive intents (legal threats, account takeover, safety), tool errors, or an agent loop. Handoff passes a structured summary (issue, what was tried, customer's goal) so the human can continue immediately. The handoff rate is tracked by reason; a rising "low confidence" rate points at missing help content.

Down the stack

Layer 1The contract

Customers are promised accurate answers grounded in current policy, actions only on their own account, a human whenever they need one, and responses that start within about a second. The business is promised a measured resolution rate, bounded refund exposure and cost per resolved conversation.

Layer 2The mechanism

This follows Anthropic's own advice to start with the simplest design, use workflows for predictable paths, and add an agent only where steps cannot be predicted, with stopping conditions such as a maximum iteration count (Anthropic). The orchestrator is mostly a workflow: classify, retrieve or look up, generate, check, respond, with a bounded agent loop (8 steps, 0.20 dollars) for multi-step account issues. The gateway applies prompt caching to the system prompt and tool definitions, routes classification to a small model and generation to a larger one, enforces per-tenant limits, and logs tokens, cost and versions per turn. Tool calls validate against schemas, check ownership in the tool service, and use the actions table's unique idempotency key to stop duplicates when a turn is retried.

Layer 3The machine

Latency is dominated by model time: about 0.3 seconds for classification on a small model, 0.1 seconds for retrieval in Postgres, about 0.8 seconds to first token for generation with a cached prefix, then about 4 seconds of decode for 300 tokens at roughly 75 tokens per second, which streaming hides. Streaming uses server-sent events over long-lived HTTP connections; the conversation service runs on an event-driven server so thousands of open streams cost little memory. Retrieval at 30,000 chunks fits in memory, so index lookups are sub-millisecond and full-text queries a few milliseconds.

Why it's built this way

  1. Most support questions are answered by existing content, so retrieval quality drives resolution.
  2. Actions carry money and privacy risk, so identity, limits and idempotency sit outside the model.
  3. Humans are expensive but essential, so escalation is designed, measured and smooth.
  4. Most traffic follows a few paths, so workflows handle them cheaply and an agent handles the rest.
  5. Quality and cost drift with every change, so evals and per-version metrics gate releases.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Workflow with bounded agentPredictable, cheap, testableLess flexible on odd casesMost support traffic
Fully agentic assistantHandles anything in principleCost, variance, harder safetyInternal tools with expert users
Automatic refunds under a limitFast resolutionBounded loss from mistakes or abuseSmall, reversible amounts
Human approval for all actionsMaximum controlSlow, expensiveEarly launch, high-value accounts
Single large modelSimpleCost and latencyLow volume
Small model for routing, larger for answersCheaper, fasterTwo models to evaluateProduction volume

Leadership lens

Check yourself

  1. Why is resolution rate usually more important than model cost for a support assistant?

    Show answer

    A human-handled conversation costs far more than the model calls, so improving resolution saves more money than shaving model cost, as long as quality stays high.

  2. Why should customer_id never be a tool argument chosen by the model?

    Show answer

    So the model cannot be tricked or mistaken into accessing another customer's data; identity must come from the authenticated session.

  3. What makes a good handoff?

    Show answer

    It happens on the right signals (request, low confidence, sentiment, sensitive topics, loops) and includes a summary so the customer does not repeat themselves.

  4. How do you stop a retried turn from refunding twice?

    Show answer

    An idempotency key per conversation turn, enforced by a unique constraint and passed to the payment provider.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →