How to Build a Reliable System Around an Unreliable AI Agent

Your code should, for any task with a known shape. Let the model pick the next step only when you genuinely cannot enumerate the steps in advance, which is rarer than most builds assume. Anthropic drew this line in 2024 and it has held up. "Workflows are systems where LLMs and tools are orchestrated through predefined code paths," the post reads. "Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage." "Workflows offer predictability and consistency for well-defined tasks, whereas agents are the better option when flexibility and model-driven decision-making are needed." - Erik Schluntz and Barry Zhang, Anthropic, "Building effective agents", 19 December 2024 The arithmetic is the reason. A step that is right 95 per cent of the time, chained twenty times with no checks in between, gives you 36 per cent end to end. Every step you move from the model into code is a step that stops multiplying. Anthropic's own post names the risk directly: autonomous agents bring "higher costs, and the potential for compounding errors". This is also where the money goes. Gartner predicted on 25 June 2025 that over 40 per cent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Inadequate risk controls is an architecture problem, not a model problem.

Force the shape at decode time instead of asking for it in the prompt. Constrained decoding compiles your JSON schema into a grammar and blocks any token that would break it, so a malformed response is not merely unlikely, it is unreachable. The measured gap is large. When OpenAI shipped Structured Outputs on 6 August 2024, it reported that gpt-4o-2024-08-06 scored 100 per cent on its complex JSON-schema-following eval, against under 40 per cent for gpt-4-0613 on the same eval, and put prompt-only format reliability at 35.9 per cent. Anthropic's Claude platform now ships the same mechanism in two modes: a JSON output format, and strict tool use where the tool call arguments must match your input schema. Anthropic's documentation notes the compiled grammar is cached for 24 hours from last use, with extra latency on the first request that uses a new schema. Outside the vendors, Outlines, XGrammar, Guidance, LM Format Enforcer and llama.cpp's GBNF do the same job on open models. XGrammar, released by the MLC team on 22 November 2024, reports mask computation under 40 microseconds per token for JSON schemas, roughly 100 times faster than earlier approaches, which retires the old argument that constraining is too slow. One honest counterpoint. Tam et al., "Let Me Speak Freely?" (Appier, EMNLP 2024 Industry Track, August 2024) measured reasoning accuracy falling under tight format restrictions, while classification held up or improved under JSON mode. dottxt's rebuttal "Say What You Mean" argued the paper conflated JSON mode with true constrained generation. The practical resolution is simple: give the model a free-text reasoning field first in key order, then the structured fields it has to commit to.

Reject it. A validator is a gate that returns pass or fail, not a judge that scores quality. If it produces a number between one and ten, it is an eval, and it belongs in a different part of your system. Schema conformance is the floor. The schema says invoice_total is a number. Your validator says invoice_total must equal the sum of the line items, customer_id must exist in your database, and the appointment must fall in the future and inside business hours. Those are plain code: deterministic, unit-testable, and they catch the failures a schema never will. A perfectly-shaped JSON object with a fabricated customer ID passes every structured output check ever built. Instructor, maintained by Jason Liu and Ivan Leo, packages the loop most teams end up writing anyway: pass a Pydantic model, validate the response, feed the validation error back and retry. It reports over 3 million monthly downloads on PyPI in 2026. Zod does the same job in TypeScript. The library matters less than the discipline of parse, validate, reject. Billy, another buildDay registrant, said he was "trying to build self validating statistical workflows". Right instinct, one correction: the validation must not be self-validation by the model. Guy, registering for the same event, named his blocker as "Achieving robust code aligned with my requirements/prompting." The move there is to stop expressing requirements as prompt text and start expressing them as assertions. A prompt is a request. A validator is a rule.

Model the work as explicit states with explicit transitions, and let the model fill slots inside a state rather than choose which state comes next. This is the single highest-value structural change most builds can make. Take a quoting flow. The states are intake, enriched, priced, awaiting_approval, sent and closed. Your code owns which state follows which, and what must be true before a transition fires. The model does the fuzzy work inside a state: read the inbound email, extract the job type, draft a line item description. The flow only moves from enriched to priced if every required slot exists, has the right type, and passes your business rules. Ankit, on a buildDay sign-up form, described being stuck on "Executing complex tasks that contain variables." A slotted state machine is the direct answer. The variables stop being free text buried in a prompt and become named, typed, validated fields on a state object that you can log, diff and replay. You get four things from this that a free-roaming agent cannot give you. Work is retryable at the state rather than from the beginning. It is resumable after a crash. It is inspectable, because you can look at any run and see exactly which state it died in. And the blast radius of a bad generation is one slot, not the whole run. LangGraph gives you this with a graph plus checkpointing. Temporal, Inngest and Restate give you durable execution for the same shape. A status column on a database table and a worker process gives you most of it with no new dependency. The pattern is what matters, not the framework.

Give every side effect an idempotency key derived from the work, not from the attempt. A duplicate call then returns the original result instead of doing the thing again. Stripe has run this pattern for years and the mechanics are worth copying exactly. Stripe saves the status code and body of the first request made for a given idempotency key, returns that same result for later requests using the key, and prunes keys after they are at least 24 hours old. Errors get replayed too, which is the part people leave out and then regret. For an agent, derive the key from the state machine rather than from the model. Something like quote-1841-send-email-v1: same job, same key, one email, no matter how many times the step is retried or how many workers pick it up. Never let the model generate the key, because a non-deterministic component producing your uniqueness token defeats the whole mechanism. Then add the outbox pattern. Write the intended action into your own table inside the same transaction that advances the state, and have a separate worker perform it. A crash between deciding and doing can no longer produce either a lost action or a duplicate one. The rule underneath all of it: an agent should never call a side-effecting API directly. It should request an action, and a deterministic executor should perform that action exactly once.

Retry when the failure is mechanical and bounded. Escalate when the failure is semantic, or when you have spent your budget. Deciding this once, in code, is what stops an agent looping at 3am. Cap the retries and cap the spend. Three attempts with jittered backoff covers nearly every transient failure, and an unbounded retry loop on a task that is genuinely impossible is how a project burns a month of budget overnight. Log the escalation with the full state object attached, so the human who picks it up does not have to reconstruct what happened. Fahri, registering for a buildDay, listed what he was stuck on as "Production quality checlists" [sic]. That table is most of one. The rest is the boundary table further down this page.

On the irreversible actions, and nowhere else. A gate on everything gets clicked through inside a week and stops being a control at all. The cost of getting this wrong is documented. Over 17 and 18 July 2025, Replit's coding agent deleted a production database during an explicit code freeze, then told the user a rollback would not work. The user recovered the data manually anyway. "Replit agent in development deleted data from the production database. Unacceptable and should never be possible." - Amjad Masad, CEO, Replit, quoted in Fortune, 23 July 2025 Replit's response was architectural rather than prompt-level: automatic separation between development and production databases, better rollback, and a planning-only mode that lets the agent work without touching a live codebase. That is the whole lesson in one incident. The agent had been told not to act. Being told is not a control. Not holding the credential is a control. Design the gate as part of the state machine. The agent proposes an action, the run parks in awaiting_approval, the human sees a plain-language diff of exactly what will happen, and the approval carries the idempotency key so a double-click cannot fire it twice. LangGraph implements this directly with interrupts and checkpointing. A row in a table and a message in Slack does the same job. What matters is that the approval is a state transition your code owns, not a sentence in a system prompt.

Anything in the top half of this table that you hand to the model is a place where a bad generation turns into a bad outcome. That is the whole test. Run the audit on your own build in about ten minutes. List every action the system can take, mark each one reversible or irreversible, and check that no irreversible action can fire without passing a validator your code owns. Then check that no model output reaches a side effect without a schema, a validator and an idempotency key in between. None of this tells you how often the model is right. That is a different question with different tools: fixed test cases, graders and thresholds wired into your deploy. Our guide on AI evals covers that side. Architecture and evals answer different questions, and teams that skip the architecture end up measuring a system that was never contained in the first place.

Does forcing JSON output make the model worse at reasoning? It can, if you force the answer field first. Tam et al. (Appier, EMNLP 2024) measured reasoning accuracy falling under tight format restrictions. The fix is key order: put a free-text reasoning field before the structured fields, so the model thinks before it commits. How is this different from running evals? Evals measure how often the output is right, after the fact, using test cases and graders. Architecture decides what happens when it is wrong. You need both, but containment comes first, because an eval on an uncontained system only tells you how often you got lucky. How many retries should an AI step get before it escalates? Two for validation failures, three with jittered backoff for transient network errors, then escalate with the full state object attached. Unbounded retries on a genuinely impossible task are how an agent burns a budget overnight. Do I need LangGraph or Temporal to build this? No. A status column on a database table and a worker process gives you most of a state machine. Frameworks help with checkpointing, replay and durable execution once runs get long, but the pattern works with plain code and a database. Can I just tell the agent in the prompt not to do dangerous things? No. Replit's agent deleted a production database during an explicit code freeze in July 2025. A prompt is a request, not a permission boundary. If an action would be unacceptable, the agent should not hold the credential that performs it.