AI Agents in Production: What Works, What Doesn't, and Who's Actually Shipping

An AI agent that aces a demo task often fails at the actual job, because a task comes with context handed to it and a job requires the agent to gather that context itself. This distinction shows up repeatedly when you compare different types of AI benchmarks against each other. Some benchmarks give a model everything it needs upfront: the files, the requirements, the acceptance criteria, laid out cleanly before the model does anything. Models tend to score well on these because the hard part of the job, working out what actually needs doing, has already been done for them. Other benchmarks instead hand the model something closer to a real brief: a client's description of what they want, a handful of files, and an instruction to figure the rest out. One widely-discussed benchmark of this second kind reportedly found that AI agents completed only a small single-digit percentage of realistic freelance-style projects at a quality a paying client would actually accept, a dramatically lower figure than benchmarks that supply full context in advance. That gap matters because it maps almost exactly onto what a business is actually buying when it hires someone, whether that is a person or a piece of software. Nobody pays a freelancer just to execute a task they have already fully specified. They pay them to work out what the task even is from a messy brief, ask the right questions, and then do the work. If an AI agent cannot reliably do that second part yet, the reasonable conclusion is not that agents are useless. It is that most of today's real value sits in the narrower set of jobs where someone else has already done the context-gathering, not in handing an agent an open-ended brief and walking away. - Tasks: context is supplied. Benchmarks here tend to show strong results. - Jobs: context has to be gathered by the agent itself. This is where completion rates reportedly fall sharply. - The practical question for your business is not "can AI do this task" but "can AI do this job, unsupervised, with the messy inputs I actually have". One widely-discussed benchmark reportedly found around a 2.5% completion rate for realistic freelance-style projects when models were given only a client brief and files, versus much higher scores (some above 60%) on benchmarks that hand over full context upfront (reported benchmark comparison, 2026).

The CEO of a well-known AI coding platform has stated publicly that outside two narrow areas, customer support and software development itself, most enterprise agentic AI is not yet reliably working. That is a notable admission, coming from someone whose company sells the tools that are supposed to make agentic AI work. His framing is worth sitting with. He has described most of what gets marketed as "agentic AI" today as closer to traditional software pipelines with an AI step or two embedded in them, rather than a genuinely autonomous system making its own judgment calls across a task. A lot of what looks like an agent from the outside is really a fixed sequence of steps, some of which happen to call a language model, wrapped in agent-shaped marketing. He has also pointed to why some domains resist agentic automation more than others. Models that get very good at maths or code tend to generalise reasonably well into adjacent technical domains, but that improvement drops off sharply in "squishy" areas like marketing or judgment-heavy client work, where there is no single correct answer to optimise toward. Without strong reasoning and something close to taste in those softer domains, deploying an agent to run them unsupervised remains genuinely difficult. This lines up with a related idea worth naming directly: a lot of low-quality AI output, sometimes called AI slop, comes from a model being asked to one-shot something with a lazy prompt and no real effort behind the scenes to shape the result. The fix is not a better model alone. It is a platform and process that puts real effort and judgment into the output, rather than assuming autonomy solves the problem on its own. - Two areas reportedly working reliably today: customer support and software development. - Most other "agentic" deployments described as closer to conventional pipelines with AI steps than true autonomous agents. - Squishy, judgment-heavy domains reportedly resist agentic automation more than technical ones, because there is no clean signal to optimise against.

Most enterprise agent deployments today operate in read-only mode, retrieving and summarising information, rather than being given write access to change anything in a production system. That gap is not really a technical limitation. It is a trust decision, and a reasonable one. Giving a non-deterministic system, one that can behave differently on the same input twice, permission to make changes in a live production system is a fundamentally bigger step than giving it permission to look things up. A read-only agent that gets something wrong produces a bad answer someone has to double-check. A write-enabled agent that gets something wrong can change a customer record, send an incorrect communication, or corrupt data that then has to be reconciled after the fact. This has a direct consequence for how much value organisations are seeing from their agent deployments so far. Read-mode agents behave a lot like an improved internal search tool: genuinely useful for finding information faster, but not the step-change productivity gain the more ambitious agentic pitches promise. The bigger productivity claims tend to depend on agents that can act, not just retrieve, and acting is exactly the part organisations are moving toward carefully. There is also a governance dimension that is easy to underestimate. An organisation might build and test an agent workflow in a matter of hours, but the internal sign-off to actually deploy it with broader permissions can take considerably longer, because someone has to be accountable if it goes wrong later. That asymmetry, fast to build, slow to trust with real permissions, is a large part of why so much agentic AI activity is still concentrated in the safer, read-only lane. - Read access: agents retrieve and summarise. Lower risk, and where most current deployments sit. - Write access: agents change production data or systems. Higher risk, adopted far more cautiously. - The productivity gains most businesses want tend to require write access, which is exactly the permission organisations are slowest to grant.

One industry estimate has suggested that only a small fraction of vendors marketing agentic AI capabilities are offering genuinely new agentic functionality, with much of the rest being existing chatbot or automation technology that has simply been rebranded. That estimate should be treated as a directional signal rather than a precise figure, but the underlying pattern it points to is worth taking seriously. The practical issue for a buyer is that "agentic" has become a marketing label almost as much as a technical description. A tool that automates a fixed sequence of steps and happens to use a language model for one of them can reasonably be called agentic under a loose enough definition, even though it behaves nothing like the flexible, judgment-making system the term implies to most people evaluating a purchase. Separately, a forecasting firm has been widely reported as predicting that a significant share of agentic AI projects will be cancelled within the next couple of years, largely due to cost and unclear return on investment. That prediction has circulated widely enough that it is worth flagging as a common industry expectation, though it should be read as a forecast rather than an established outcome, and the specific figure attached to it varies depending on the source repeating it. The honest takeaway from this scepticism is not that agentic AI is a dead end. It is that the category is genuinely crowded with relabelled older technology, and that due diligence on any specific vendor claim matters more than it would for a more settled technology category. Asking a vendor exactly what their system does when it encounters an input it has not seen before is usually a faster way to separate a real agent from a rebranded workflow than asking about the model underneath it. - A meaningful share of "agentic AI" vendors are reportedly offering rebranded chatbot or automation technology rather than something genuinely new. - Analysts have reportedly forecast a wave of agentic AI project cancellations over the next couple of years, largely on cost and ROI grounds. - Treat both figures as directional industry sentiment, not settled fact, and verify specific vendor claims against what the system actually does on unfamiliar input.

The sensible approach to agentic AI right now is to identify the narrow set of tasks where it demonstrably works for your specific business, rather than assuming the general capability the marketing describes. That is a smaller ambition than "deploy an autonomous agent across the business", and it is also the one that actually produces working software. In practice this means starting from a specific, bounded problem rather than a category. Customer support triage, first-draft code generation, and structured document summarisation are areas where agentic approaches have a stronger, more consistent track record. Open-ended client-facing judgment calls, anything touching production data with write access, and squishy domains without a clear right answer are areas where the track record is thinner and the failure modes are more expensive. A useful test before committing budget to any agentic feature is to ask what happens when the agent is wrong, not just whether it is usually right. If a wrong answer is annoying but recoverable, a narrower pilot makes sense. If a wrong answer touches money, customer data, or something irreversible, that is exactly the case for keeping a human in the loop and only expanding autonomy once the specific task has proven itself, repeatedly, in your own environment rather than in someone else's benchmark. - Start from a specific business problem, not a general "we should have an agent" mandate. - Favour tasks with a clear success criterion and cheap recovery from a wrong answer. - Pilot narrowly, measure against your own data, and expand permissions only once the specific task has proven itself repeatedly.

Do AI agents actually work in production today? Some do, reliably, in a narrower set of areas than the marketing suggests, most consistently in customer support and software development according to public comments from AI-tool company leadership. Outside those areas, results are mixed and depend heavily on how open-ended the task is. Why do AI agents perform so much better on benchmarks than in real client work? Many benchmarks hand the model context upfront, effectively doing the hardest part of the job for it. Real client work usually requires the agent to gather and interpret that context itself from a messy brief, which appears to be where completion rates drop the most. Why don't more companies give AI agents write access to their systems? Because a non-deterministic system with write access can make an unreviewed change to live data or customer-facing systems, which is a materially bigger risk than a read-only agent that only retrieves information. Most organisations are reportedly still in the read-only, retrieval-focused stage for that reason. Is "agentic AI" just a rebrand of older automation technology? Sometimes, according to industry estimates that have suggested a large share of vendors marketing agentic capabilities are offering existing chatbot or automation technology under a new label rather than something genuinely new. It is worth asking any vendor specifically what their system does on inputs it hasn't seen before. How should a small or mid-sized business decide whether to invest in an AI agent? Start with a specific, bounded task rather than a general agentic ambition, and test what happens when the agent is wrong before scaling it up. Tasks with a clear success criterion and a low-cost failure mode are the safest place to begin, and that is the approach we take when scoping agentic features at 13Labs.