Guardrails for autonomous agents

An agent that can call tools is a program whose control flow is decided at runtime by a language model. That is a useful capability and an uncomfortable thing to point at production data. The teams who ship agents successfully are not the ones with the best prompts. They are the ones who assumed the model would do something stupid on run 400 and built so it would not matter.

Scope the tools, not the prompt

The most common design error is exposing a general capability and describing its limits in English. run_sql(query: string) with "only read from the analytics schema, never write" in the system message is not a constraint. It is a suggestion, and it will be ignored under prompt injection or an unlucky sample.

Tools should be as narrow as a well-designed API endpoint:

• One verb, one resource. refund_order, not manage_order(action).
• Enumerations instead of free text wherever the value set is known.
• Validate arguments against a schema, then re-check authorization server-side against the run's identity, as you would for a browser request.
• Scope every query by tenant inside the handler. The model never supplies the tenant id.
• Separate read tools from write tools so budgets and approvals can treat them differently.

// src/agent/tools/refund-order.ts
import { z } from "zod";
import { defineTool } from "@/agent/kernel";

const Input = z.object({
  orderId: z.string().uuid(),
  amountCents: z.number().int().positive().max(50_000),
  reason: z.enum(["damaged", "not_received", "duplicate_charge", "goodwill"]),
});

export const refundOrder = defineTool({
  name: "refund_order",
  description: "Refund part or all of a single order that is already paid.",
  input: Input,
  effect: "irreversible",           // drives the approval gate
  costCents: (i) => i.amountCents,  // drives the spend budget
  async execute(input, ctx) {
    const order = await db.order.findFirst({
      where: { id: input.orderId, orgId: ctx.orgId }, // orgId never comes from the model
    });
    if (!order || order.status !== "paid") {
      return { ok: false as const, error: "order_not_refundable" };
    }
    if (input.amountCents > order.totalCents - order.refundedCents) {
      return { ok: false as const, error: "amount_exceeds_remaining" };
    }

    const refund = await stripe.refunds.create(
      { payment_intent: order.paymentIntentId, amount: input.amountCents },
      { idempotencyKey: `refund:${ctx.runId}:${ctx.stepId}` },
    );
    return { ok: true as const, refundId: refund.id };
  },
});

Failures come back as structured results rather than thrown exceptions. An agent that reads order_not_refundable will try something else. An agent that hits a 500 just calls the same tool again.

Approval gates on anything irreversible

Classify every tool by effect and let the class drive policy, rather than making a judgement call per tool:

• Reversible: reads, drafts, internal notes, anything in a sandbox. Runs freely.
• Externally visible: sending an email, posting to a channel, changing a public status. Free below a threshold, gated above it.
• Irreversible: money movement, deletion, production writes, anything a customer experiences as final. Always gated, or gated above a value you can defend later.

The gate has to be a real pause, which means the run has to be durable. An agent holding an open HTTP connection while a human decides breaks the first time the process restarts. We persist run state, emit an approval request containing the proposed call and the steps that led to it, and resume from the stored state when a decision arrives. An unanswered gate after 24 hours fails the run rather than sitting open forever.

Show the approver the arguments and the last few steps, because "send_email" alone tells them nothing and they will approve on reflex. Record the decision, the approver and the timestamp: the first serious incident review will ask exactly who approved what.

Budgets: steps, tokens, money, wall clock

Every run gets hard ceilings enforced by the loop, not by the model:

• Step budget. 12 to 20 tool calls covers almost every task we ship. Hitting the ceiling is a signal that the task was wrong for an agent, or that the tools are too fine-grained.
• Token budget per run, so a context growing with every observation cannot quietly cost twenty times the estimate.
• Spend budget in real currency, covering model cost and the value of actions taken. A refund agent with a 5,000 cent daily ceiling per operator cannot become an incident.
• Wall clock. Kill a run at ten minutes and surface it rather than letting it retry into next week.
• Loop detection. Hash the tool name plus normalized arguments. The same hash three times means the agent is stuck, and stopping there saves money and gives a clearer trace.

Trace every run, replay any run

You cannot debug an agent from application logs. What you need is the ordered record of every step: messages sent, model and version, sampling parameters, tool called, arguments, raw result, token counts, latency. Append-only rows, keyed by run.

create table agent_runs (
  id                 uuid primary key,
  org_id             uuid        not null,
  goal               text        not null,
  status             text        not null,  -- running|awaiting_approval|done|failed
  step_budget        int         not null,
  spend_budget_cents int         not null,
  spend_cents        int         not null default 0,
  started_at         timestamptz not null default now(),
  ended_at           timestamptz
);

create table agent_steps (
  run_id        uuid        not null references agent_runs(id),
  step_no       int         not null,
  kind          text        not null,      -- model_call|tool_call|approval|error
  model         text,
  tool_name     text,
  input         jsonb       not null,
  output        jsonb,
  prompt_tokens int,
  output_tokens int,
  cost_cents    int         not null default 0,
  latency_ms    int,
  created_at    timestamptz not null default now(),
  primary key (run_id, step_no)
);

create index agent_steps_tool_idx on agent_steps (tool_name, created_at desc);

That table gives you replay for free. Feed the recorded steps back through the loop with each tool stubbed by its stored output, and you can test a prompt change against 200 real historical runs before it touches traffic. It is the closest thing an agent system has to a regression test, and it turns "the new prompt feels better" into a diff you can read. Sample and label too: a weekly review of 20 random runs, scored by a human, gives you a quality number that survives contact with reality.

Retries and idempotency

Agent loops retry constantly. The model retries because a result looked wrong, the framework retries on a timeout, the queue redelivers because the worker died holding the job. Every tool call therefore executes at least once and possibly several times.

The fix is the one payments taught us. Derive a deterministic key from the run and step, pass it to any downstream that supports idempotency, and keep a local table of executed keys and results for those that do not. On a repeat, return the stored result instead of acting again. runId:stepId is stable across retries of the same step and different for a genuinely new decision.

Distinguish retryable from terminal errors in the tool contract. A 429 or a connection reset is retryable with backoff. A validation failure, an authorization failure or "order already refunded" is terminal: hand it back to the model and let it choose differently.

When an agent is the wrong tool

This is the section the vendor demos leave out. A loop that plans its own steps is the right shape when the task genuinely branches, the input is unstructured, and the paths are too numerous to enumerate: triaging inbound support mail, investigating a failure across several systems, drafting from messy source material.

It is the wrong shape when the process is already known. If you can draw the flowchart, build the flowchart. A queue, a rules table and three well-tested functions will be faster, cheaper by an order of magnitude, deterministic, and debuggable by anyone on the team at 2am. We have replaced two "agents" with a state machine plus one narrow model call for classification, and both got more accurate and roughly 30 times cheaper.

The test we apply: if a competent new hire could follow written instructions and do this without judgement calls, it is a workflow, not an agent. Use the model for the step that needs judgement and ordinary code for the rest.

How we work

We ship agents with the boring parts first: typed tools with server-side authorization, an effect classification per tool, budgets enforced in the loop, a run table with full traces, and a replay harness before the first production run. Approval policy is written down with the client, tool by tool, thresholds attached. And we say plainly when part of a workflow does not need an agent, because a small model call inside a well-understood pipeline is usually the version still working a year later.