Engineering Atlas › Building AI Systems

Observability and Cost Control for AI Features

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

What to record per call

For every model call: trace and span IDs, feature, team, tenant, user (hashed), model and version, prompt template version, input, output and cached token counts, time to first token, total latency, stop reason, tool calls made, retries, cost computed from the price table, and a link to the stored prompt and response (with personal data redacted, and retention limits). OpenTelemetry defines generative-AI semantic conventions with attributes such as gen_ai.request.model and gen_ai.usage.input_tokens, so this data fits standard tracing pipelines. Two cautions. The conventions are still at Development status and have moved to their own repository, and older names have changed (the provider is now gen_ai.provider.name, replacing gen_ai.system), so pin the version you instrument against (OpenTelemetry GenAI spans). And OpenTelemetry says gen_ai.usage.input_tokens should include cached tokens, whereas Anthropic's input_tokens counts only tokens after the last cache breakpoint; total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens (Anthropic prompt caching). Mixing the two double-counts or under-counts cost. Langfuse makes the same point: usage buckets must not overlap, so cached tokens get their own key and their own price, and provider counts that already include them must be converted first (Langfuse). Store large prompt and response bodies as events or in a separate store, not as span attributes, because backends struggle with big attributes (OpenTelemetry blog).

The dashboards that matter

Start from Google's four golden signals, which apply unchanged: latency (tracked separately for successes and failures), traffic, errors (including implicit ones, such as a 200 response with wrong content) and saturation (Google SRE book). The implicit-error case is the AI-specific trap, which is why quality needs its own signal below.

Cost per outcome

Cost per call hides the real economics. A support assistant that costs 2 cents per message but needs 12 messages and escalates 30 percent of conversations to a human costs far more per resolved ticket than one that costs 5 cents per message, needs 4 and escalates 10 percent. Define the outcome (resolved ticket, accepted suggestion, completed task) and divide total cost, including retries and human handling, by outcomes.

Anomalies and guardrails on spend

Cost incidents look like other incidents: a sudden step. Anthropic's rate-limit docs note that a spend cap, when reached, stops all API use until the next month and returns a 429 with no retry-after, so a cap is a last-resort stop and not a substitute for per-feature budgets (Anthropic rate limits). Common causes are an agent loop, a prompt change that included a large document in every request, a retry storm, a misconfigured batch job, or a user abusing a free tier. Detect them with per-feature daily or hourly baselines and alerts on large multiples, and limit blast radius with budgets per feature and tenant that downgrade, queue or block when exceeded. Alerts tell you after money is spent; enforcement stops the spending, so use both. If you reconcile against the provider's bill, Anthropic's Usage and Cost Admin API reports token usage by model, workspace, API key and service tier in 1-minute to 1-day buckets, and data usually appears within about 5 minutes (Anthropic Usage and Cost API); it needs an admin credential, not an ordinary workspace key.

Quality in production

Offline evals catch regressions before release; online signals catch what offline sets miss. Sample a fraction of traffic for automated grading, track explicit feedback, and watch behavioural proxies such as rephrased questions, abandoned sessions or copy-to-clipboard rates. Always break metrics down by prompt and model version, because the most common cause of a quality change is a change you made. Sometimes it is not: in 2025 Anthropic's own quality drops came from serving bugs that its evals and dashboards did not flag, and users' reports were the first signal (Anthropic postmortem). Chip Huyen's book treats user feedback design, and extracting feedback from conversation signals, as a core part of the production architecture, next to monitoring and observability.

The same shapes as any system

This is the standard observability discipline (RED metrics, traces, logs, SLOs) with cost as a first-class metric and quality as a sampled measurement. The anomaly detection is the same as for error rates; the attribution is the same as cloud cost allocation in FinOps practice.

Down the stack

Layer 1The contract

Each AI feature has SLOs for latency and availability, a quality target measured online, and a cost budget per outcome. The platform promises that every call is attributed, that dashboards update within minutes, and that anomalies page the owning team.

Layer 2The mechanism

The model gateway or client library emits spans and metrics for each call, computing cost from token counts and a versioned price table (since prices change). Metrics go to a time-series store with labels for feature, model and version; high-cardinality labels such as tenant are kept in logs or traces or aggregated carefully to control cost. Prompt and response bodies go to a separate store with redaction and short retention, linked by trace ID. Anomaly detection compares current hourly cost per feature with a rolling baseline, alerting when it exceeds a multiple such as 3 times. Online grading jobs read sampled traces and write scores back keyed by trace ID.

Layer 3The machine

Storing full prompts and responses is expensive in itself: an assistant with 10 million calls a day averaging 8 KB of prompt and response text produces about 80 GB a day before compression. Text compresses well (often 5 to 10 times), but retention, indexing for search and redaction processing add up, which is why teams sample bodies, keep metrics for everything, and set short retention for raw text. Sampling should be deliberate: keep every error, every slow request and every low-scored one, and a fraction of the rest (Observability Engineering, Ch 17, "Cheap and Accurate Enough: Sampling"). Metric cardinality is the other machine-level cost: a label such as tenant ID with 50,000 values multiplied by model and feature creates millions of time series, which can overwhelm a metrics backend.

Why it's built this way

  1. Model calls are a large, variable cost and a new failure mode, so they need first-class telemetry.
  2. Cost without attribution has no owner, and owners are what make cost go down.
  3. Per-call cost misleads; per-outcome cost reflects real value.
  4. Spend can spike quickly, so it needs the same alerting as errors.
  5. Quality drifts with changes you make, so it must be measured by version in production.

Diagram

flowchart LR
  F["Feature call"] --> GW["Gateway or client"]
  GW --> SP["Span: model, version, tokens, latency, stop reason, cost"]
  GW --> BODY["Redacted prompt and response (sampled, short retention)"]
  SP --> MET["Metrics by feature, model, version"]
  SP --> TR["Traces"]
  BODY --> GR["Online grading on sample"]
  GR --> Q["Quality by version"]
  MET --> AN{"Cost above 3x baseline?"}
  AN -->|yes| PAGE["Page owning team, apply budget"]

Worked example

A company's AI features cost about 60,000 dollars a month. On a Tuesday, the hourly cost of the "contract review" feature jumps from about 90 dollars to 640 dollars. The anomaly alert (more than 3 times the 7-day hourly baseline) pages the owning team within 20 minutes.

The dashboard breaks it down: request count is flat, but input tokens per request rose from 9,000 to 61,000 after a 10:05 deploy. The trace for a sample request shows the new prompt template includes the full contract history instead of the current version. The team rolls back the template; cost returns to baseline within the hour. Total extra spend: about 1,300 dollars, instead of an estimated 150,000 over a month.

The same week, they add cost per outcome: contract review costs 0.42 dollars per review accepted without edits by lawyers. An experiment with a smaller model drops cost per call by 70 percent but cost per accepted review only by 20 percent, because lawyers edit more of its outputs and the edits trigger re-runs. They keep the larger model for complex contracts and route simple ones to the smaller model.

In the wild

Trade-offs

PracticeStrengthsWeaknessesChoose when
Metrics for every callComplete cost and latency viewCardinality managementAlways
Full prompt and response loggingDeep debuggingStorage, privacySampled, redacted, short retention
Per-tenant attributionFair billing, abuse detectionHigh cardinalityMulti-tenant products
Cost per outcomeReal economicsNeeds outcome trackingChoosing models and designs
Automated online gradingContinuous quality signalGrader cost and calibrationUser-facing features at scale

Pitfalls in production

Leadership lens

Check yourself

  1. Why can a cheaper model raise total cost?

    Show answer

    If it fails more often, causing retries, longer conversations or more human escalations, cost per successful outcome can rise even as cost per call falls.

  2. What are typical causes of a sudden AI cost spike?

    Show answer

    Agent loops, prompt changes that add large content, retry storms, misconfigured batch jobs, and abusive or unexpected usage.

  3. Why tag metrics with prompt and model versions?

    Show answer

    Most quality and cost changes come from changes you made, and version tags let you attribute shifts to a specific release.

  4. Why not log every prompt and response forever?

    Show answer

    It is expensive and holds personal or sensitive data; sampling, redaction and short retention keep debugging value with less risk.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →