Engineering Atlas › Building AI Systems

Structured Output and Tool Use

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why structure

Prose is for people. Code needs fields it can trust: {"category": "billing", "priority": 2}. Asking for "JSON please" in the prompt works most of the time, which in production means it fails some of the time. Providers now offer stronger mechanisms: tool and function calling with schemas, and constrained decoding that only allows output matching a schema. Anthropic's structured outputs (output_config.format, plus strict: true on tools) are generally available (Anthropic structured outputs), and OpenAI distinguishes plain JSON mode, which only guarantees valid JSON, from Structured Outputs with strict: true, which guarantees the schema (OpenAI structured outputs). The old trick of prefilling the assistant turn with { no longer works on Claude 4.6 and later, which return a 400 for a prefilled last message (Anthropic errors). Even then, you validate, because a value can be well-formed and still wrong.

The tool-use loop

  1. Your request includes tool definitions: name, description, and input_schema (JSON Schema for the arguments).
  2. The model decides whether to call a tool. If it does, the response contains a tool-use block with an ID, the tool name and arguments, and the stop reason is tool_use (Anthropic tool use). With the default tool_choice of auto the model may also answer directly. Forcing a call with tool_choice of any or tool is rejected with a 400 on the newest Claude models, so use strict schemas with auto instead (Anthropic errors).
  3. Your code validates the arguments, checks permissions, runs the tool, and sends back a tool-result block with the same ID, containing the output or an error.
  4. The model reads the result and either calls another tool or answers.

This loop is the core of every agent. The model never executes anything; your code does, so your code is where safety lives.

Designing good tools

Validation and repair

Validate every tool call and every structured response against its schema, even with constrained decoding, because the guarantee has holes. A refusal (stop_reason: "refusal") can return output that does not match, a max_tokens stop can truncate the JSON, and Claude may change the capitalisation of enum values (Anthropic structured outputs). OpenAI likewise returns a refusal item in place of schema-shaped output and can stop incomplete at the output-token limit (OpenAI structured outputs). If it fails, return the validation errors as the tool result (or a follow-up message) and let the model try again, at most once or twice. If it still fails, fall back: a default, a human, or a clear error. Log every failure; rising rates usually mean a prompt or model change broke something.

Side effects

Tools that change things (create a refund, send an email, update a record) need the same care as any API: idempotency keys derived from the conversation and step, permission checks based on the end user, not the model's claims, and approval for high-risk actions. A retried model call can repeat a tool call, so design for duplicates.

The same shapes as any system

Tool calls are remote procedure calls with an unusual client, and retries follow the usual rule: only repeat an operation that is idempotent (Azure Retry pattern). Every API design rule applies: validate input, authorise, make writes idempotent, return clear errors. The trust-but-verify shape is the heart of it.

Down the stack

Layer 1The contract

A tool-using feature promises that only valid, authorised tool calls execute, that side effects happen at most once per intent, and that invalid model output never reaches downstream systems unvalidated. Each tool has a written contract: inputs, outputs, errors and side effects.

Layer 2The mechanism

Tool definitions are sent as part of the prompt, so they consume input tokens and influence behaviour like any instruction. The model was trained to emit tool calls in a specific structured format, which the API parses into tool-use blocks. Validators such as JSON Schema libraries, Pydantic in Python or Zod in TypeScript turn the schema into code that returns precise errors. With constrained decoding, the server restricts which tokens can be sampled at each step to those that keep the output valid against a grammar derived from the schema, which removes syntax errors but not wrong values. Supported schema features are narrower than full JSON Schema. Anthropic's strict mode requires additionalProperties: false, and does not support minimum, maximum, minLength, maxLength or recursive schemas; there are also complexity limits (20 strict tools and 24 optional parameters per request) and extra latency on the first request with a new schema while the grammar compiles, cached for 24 hours after last use (Anthropic structured outputs). OpenAI's strict mode similarly wants every property in required and additionalProperties false, with a slower first request per schema. So range checks such as "limit between 1 and 20" still belong in your own validator.

Layer 3The machine

Constrained decoding works at the sampling step: before choosing the next token, the server masks the logits of every token that would make the output invalid, so the model can only pick from allowed ones. Building that mask requires tracking a parser state for the schema as tokens stream, which adds a small amount of CPU work per token. Without it, a model generating JSON is just predicting likely characters; a missing quote or comma is a sampling accident, which is why validation is still needed when constraints are not available.

Why it's built this way

  1. Code needs structured, trustworthy data, but models produce text.
  2. Schemas give the model a clear target and your code a precise check.
  3. The model cannot be trusted to call tools safely, so execution and authorisation stay in your code.
  4. Failures will happen, so validation plus bounded repair plus fallback is the robust loop.
  5. Tool calls can repeat, so side effects need idempotency like any API.

Diagram

sequenceDiagram
  participant App
  participant Model
  participant Tool as Tool (your code)
  App->>Model: messages and tool schemas
  Model-->>App: tool_use: search_orders(customer_id)
  App->>App: validate args, check permission
  App->>Tool: run with idempotency key
  Tool-->>App: compact result
  App->>Model: tool_result
  Model-->>App: final answer or next tool call

Worked example

An e-commerce support assistant has one tool, run_sql(query), and a prompt explaining the schema. It works in demos. In production, 3 percent of calls produce SQL errors, some queries scan the entire orders table, and a red-team test gets it to read another customer's orders.

The team replaces it with three tools: get_order(order_id), list_orders(status, limit) and create_return(order_id, item_ids, reason). Schemas require order_id to match ^O-[0-9]{8}$, status to be an enum of five values, limit between 1 and 20, and reason to be an enum. The customer ID is not an argument at all: the server takes it from the authenticated session. create_return takes an idempotency key built from the conversation ID and step number.

On 2,000 replayed conversations, tool-call validation failures drop from 3 percent to 0.4 percent, and each failure returns an error message the model uses to correct itself on the next try; after one retry, 0.05 percent still fail and go to a human. The cross-customer read becomes impossible, because the model never chooses whose data to read.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
"Return JSON" in the promptSimple, any modelOccasional invalid outputPrototypes, with validation
Tool calling with schemasReliable structure, multi-stepNeeds a loop in your codeMost production features
Constrained decodingNo syntax errorsNot available everywhere; values can still be wrongStrict formats, extraction
Generic tools (run_sql, run_shell)FlexibleHard to validate and secureSandboxed developer tools only
Narrow, typed toolsAccurate and safeMore tools to buildUser-facing and production agents

Pitfalls in production

Leadership lens

Check yourself

  1. Why is a tool call's arguments untrusted input even though your own model produced them?

    Show answer

    The model can be wrong, can be manipulated by prompt injection, and can hallucinate values, so arguments must be validated and authorised like any external request.

  2. What should happen when a tool call fails schema validation?

    Show answer

    Return the specific validation errors to the model and allow a bounded retry, then fall back to a default, a human or an error if it still fails.

  3. Why take the customer ID from the session instead of a tool argument?

    Show answer

    So the model cannot be tricked or mistaken into accessing another customer's data; identity comes from authentication, not from generated text.

  4. What does constrained decoding guarantee, and what does it not?

    Show answer

    It guarantees syntactically valid output matching the schema; it does not guarantee that the values are correct.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →