The Agent Reliability Calculator: What 98% Per Step Really Means

Most people hear "each step is 98% reliable" and assume the whole system is 98% reliable too. It is not, and the gap is not small. When steps run in sequence and any single failure breaks the chain, the system's overall success rate is the product of every step's individual success rate, not the average. This is a basic reliability engineering principle used for decades in hardware and aerospace systems, and it applies just as directly to a chain of AI agent calls, a multi-step automation, or any pipeline where step two depends on step one having worked. The reason it feels wrong is that our intuition treats "98% reliable" as a fixed property of the system, like a grade. It is not. It is a per-step probability, and probabilities compound. Every additional step you add to a chain is another chance for the whole thing to fail, even if each individual step is genuinely good.

Here is the calculation in full, so you can check it yourself with any calculator. Start with one step at 98% reliable, written as 0.98. If everything depended on just that one step, you would succeed 98% of the time, exactly as expected. - Step 1: 0.98 (98.0% success) - Step 2: 0.98 x 0.98 = 0.9604 (96.0% success) - Step 3: 0.9604 x 0.98 = 0.941192 (94.1% success) - Step 4: 0.941192 x 0.98 = 0.922368 (92.2% success) - Step 5: 0.922368 x 0.98 = 0.903921 (90.4% success) Each step you add multiplies the running total by 0.98 again, and each multiplication shaves a bit more off. By the fifth step, the system that everyone still calls "98% reliable" actually succeeds about 90.4% of the time end to end. That is roughly 1 in 10 runs failing somewhere along the chain, not 1 in 50. Keep adding steps and it gets worse fast. At ten steps of 98% each, the system is down to roughly 81.7% reliable. At twenty steps, it is under 67%. This is why a multi-agent pipeline that looks solid in a three-step demo can become genuinely unreliable once it is doing the eight or ten steps a real production workflow actually needs: fetch data, validate it, call an API, parse the response, update a record, trigger a notification, log the result, and confirm completion. Every one of those is a step, and every step takes its cut.

A system that looks reliable from the outside might just be trying and trying again until it works, and every attempt costs money. Suppose a single step only succeeds 30% of the time on one attempt, written as a base success rate of 0.3. That sounds unusably unreliable. But if the system is allowed to retry automatically, the maths changes quickly, because the probability of eventual success after N attempts is 1 minus the probability of failing every single time. The failure probability on each attempt is 0.7 (1 minus 0.3). So the chance of failing N times in a row is 0.7 raised to the power of N, and the chance of succeeding at least once is 1 minus that number. - 1 attempt: 1 - 0.7^1 = 0.300 (30.0% eventual success) - 3 attempts: 1 - 0.7^3 = 0.657 (65.7% eventual success) - 5 attempts: 1 - 0.7^5 = 0.832 (83.2% eventual success) - 8 attempts: 1 - 0.7^8 = 0.942 (94.2% eventual success) - 11 attempts: 1 - 0.7^11 = 0.980 (98.0% eventual success) It takes about 11 attempts for a step that only succeeds 30% of the time on its own to reach a 98% chance of eventual success. That is the punchline worth sitting with: a system reporting "98% reliable" to a dashboard might be doing so honestly, while quietly burning 11 attempts for every successful outcome. Each of those attempts is a real API call, a real model invocation, or a real compute cycle, and each one has a dollar cost attached. If a business is paying per token or per API call, an agent step with a 30% base success rate is not a cheap step that occasionally needs a nudge, it is roughly 11 times more expensive than the headline success rate suggests, because 10 of those 11 attempts were failures that got quietly retried and never shown to anyone.

A demo usually runs the happy path once, with clean inputs and no load, which tells you almost nothing about reliability at scale. A demo proves a system can succeed. It does not prove how often it succeeds, because a demo is typically a single run, with hand-picked inputs, on a good day, watched closely by the person who built it. None of the multiplication or retry economics described above shows up in a five-minute walkthrough. Production is different in three ways that matter for reliability maths. First, volume: a system running a thousand times a day will hit its true failure rate on a predictable schedule, not just occasionally. Second, input variety: real users send messy, unexpected, and adversarial inputs that a demo script never does, which pushes individual step reliability down from whatever it measured in testing. Third, compounding: production workflows tend to have more steps than a demo, because real business processes have edge cases, approval steps, and integrations that a proof of concept skips. The practical result is that a founder who watched a five-step agent demo succeed three times in a row has seen a system with, at best, weak evidence of being reliable. Three successes in a row is consistent with anything from 90% single-run reliability to well under 70%, and there is no way to tell which from three runs alone. The only way to know the real number is to measure it properly, at volume, over time, which is precisely what most teams skip before they call a system production ready.

Validation gates are checkpoints between agent steps that catch and correct errors before they reach the next step, which stops failures from compounding down the chain. The multiplicative failure problem exists because a single bad output at step two silently poisons steps three, four, and five. A gate breaks that chain. Instead of trusting that step two's output is correct and passing it straight through, a gate checks the output against expected shape, expected values, or a secondary verification call, and only lets it through if it passes. If it fails, the gate can retry that step, flag it for a human, or halt the pipeline before the error compounds. This is not a small addition. Designing a good validation gate means deciding what "correct" looks like for that specific step, building the check itself, deciding what happens on failure, and testing the gate against the same messy production inputs described above. That is genuine engineering time, and it is reasonable for it to show up as a real line item, not a favour thrown in for free. The payoff is that gates do not just add reliability arithmetically, they change the shape of the failure. Without gates, a five-step chain at 98% per step lands at roughly 90.4%, as shown earlier. With an effective gate after each step that catches, say, 80% of the errors that would otherwise pass through, the compounding effect is broken at the point of failure instead of carrying forward, and the system's real-world reliability moves back much closer to the 98% figure everyone assumed it already had. The gate is what makes the headline number true, not the model.

Average speed can look fine while the slowest runs feel completely broken, because an average buries exactly the outliers users notice most. Most teams monitor mean latency, the average time a step or a pipeline takes to complete. It is the easiest number to compute and the easiest one to put on a dashboard. It is also the wrong number for understanding what users actually experience, because a mean can look healthy while a meaningful slice of runs are badly degraded. This is why tail latency, usually measured as P95 (the 95th percentile) or P99 (the 99th percentile), matters more than the average for a production system. P95 answers a specific question: how slow is the run that is slower than 95% of all other runs, meaning it is in the worst 5%? If a system's mean latency is a comfortable two seconds but its P95 is eighteen seconds, then one user in twenty is waiting eighteen seconds or more, every time, and the dashboard showing "2 seconds average" never reveals that. One in twenty is not a rare edge case. For a system handling a thousand requests a day, that is fifty people a day having an experience the team is not measuring and therefore does not know is broken. For a multi-step agent chain, tail latency compounds the same way reliability does: a slow step anywhere in the chain drags the whole run's completion time out, and the more steps there are, the more chances there are for one of them to land in its own slow tail. Measuring only the mean is measuring the system's best behaviour and calling it typical.

Is 0.98 to the power of 5 really about 90%? Yes. 0.98 x 0.98 x 0.98 x 0.98 x 0.98 equals approximately 0.903921, or 90.4%. Each multiplication compounds the previous result, so five steps at 98% each land at roughly 90.4% overall, not 98%. Why does chaining agents reduce reliability so much? Because success rates multiply when steps depend on each other in sequence. If step two only runs correctly when step one succeeded, the chance both succeed is the product of their individual chances, not the smaller of the two or their average. More dependent steps means more multiplication, and multiplication of numbers under 1.0 always produces a smaller number. How many retries does it take to reach 98% success from a 30% base rate? About 11 attempts. The formula is 1 minus the failure rate raised to the power of the number of attempts. With a 30% base success rate, the failure rate per attempt is 70%, and 1 minus 0.7 to the power of 11 works out to approximately 0.98, or 98%. What is a validation gate in an AI agent pipeline? A checkpoint placed between two agent steps that verifies the output of the first step before it is passed to the second. It can check format, values, or run a secondary verification call, and it can retry, flag, or halt the pipeline if the check fails. This is what stops one bad step from silently corrupting every step after it. Why should I care about P95 latency if my average speed looks fine? Because the average describes typical behaviour, and P95 describes what your worst-affected users actually experience. A healthy-looking average can coexist with one in twenty users waiting far longer than expected, and that group never shows up in a mean-only dashboard. For a business depending on the system feeling reliable, the P95 number is closer to the real answer than the average is.