A Real Prompt Injection Attempt Against a Live Australian System, and Why It Failed

Prompt injection happens when content an AI system was asked to read gets treated as an instruction it should obey. OWASP ranks it first on its Top 10 for LLM Applications 2025, published 12 March 2025, as LLM01. OWASP splits it two ways. A direct prompt injection is someone typing into the system and deliberately or accidentally changing how it behaves. An indirect prompt injection is the system reading content from somewhere else, a website, a file, an email, a support ticket, a calendar invite, where embedded instructions alter what it does. The reply we received is an indirect injection, and that distinction carries most of the risk. A direct attack requires the attacker to be a user of your system. An indirect one only requires getting text in front of it. They can fill in your website's contact form, post in a public channel, send an email nobody opens, open a pull request, or send a calendar invite you never accept. The mechanism underneath is uncomfortably simple. A language model receives one stream of tokens. Your system prompt, your retrieved data, and an attacker's email all arrive as text in the same context window, with no structural marker separating what to obey from what to merely read. The UK's National Cyber Security Centre put it plainly on 8 December 2025: "Under the hood of an LLM, there's no distinction made between 'data' or 'instructions'; there is only ever 'next token'." That is why the attempt against us opened with "This is the user". There is no field in a context window marked trusted, so the attacker was betting the system could not tell the difference. Against a lot of systems currently in production, that is a reasonable bet.

The attempt failed for three structural reasons, and none of them involved detecting it. Nothing in the system ever recognised an attack. The reply simply had nowhere to go. There was no inbound read path at all. The sending script makes zero calls to read the mailbox. No listing of messages, no fetching of a message, no reading of a thread, no polling of history. It builds a recipient list, sends, writes to a ledger, and exits. It cannot see a reply, so a reply cannot instruct it. This was the primary defence, and it exists because the job never required reading email rather than because anyone anticipated this. Recipients came from physical presence, not from email. The guest list is pulled from the event platform's admin API and filtered to people who were both approved and carried a check-in timestamp recorded at the venue. You had to turn up and be scanned in. An inbound email cannot add anyone to that list, because the list is not built from email at all. Even a system that did read the inbox would have had to cross a second boundary to act on what it read. The ledger is idempotent. Every successful send writes a row. On each run the script builds a set of addresses already sent to and skips them, and separately marks every code that has ever appeared in the ledger as consumed. Had the send somehow been triggered again for that address, the guard would have returned before sending, and no unused code would have been available to issue. The attack had to defeat all three. It could not defeat the first, because the first is an absence rather than a control. You cannot bypass a function nobody wrote.

This system could send email freely. It was not protected by a limited credential, it was protected by having no channel through which an outsider could reach it with an instruction. We want to correct that framing directly, because we got it wrong ourselves in an earlier write-up and described the credential as scoped to drafts. It was not. The script holds a refresh token minted with a Gmail scope that includes sending, and it also sends through a transactional email provider. Ask it to send a thousand emails and it sends a thousand emails. There was no sandbox, no draft-only mode, no approval queue between the code and the outbound message. So the accurate statement is that the system could send, it simply could not be told to. That is a more useful lesson than a restricted credential would have been, because restricted credentials are the first answer everyone reaches for and they are frequently impractical. Plenty of automations genuinely need to send, pay, post or write. What you can nearly always do instead is control what is permitted to talk to the thing holding that power. Stated as a rule: the component that reads untrusted input should not be the component that holds the credential for the irreversible action. Read on one side of a boundary, act on the other, and let deterministic code or a person carry the decision across. A genuinely restricted scope does exist elsewhere in our setup. The mailbox that received this reply is also read by AI assistants through a connector that can compose and save a draft but cannot send, so a human approves anything leaving. That is a different component from the credit sender, and the two patterns should not be confused.

An AI agent becomes exploitable when it holds three things at once: access to private data, exposure to untrusted content, and the ability to communicate externally. The independent researcher Simon Willison named this the lethal trifecta on 16 June 2025. Remove any one leg and the chain breaks. An agent that reads untrusted content and can send email but holds nothing private has nothing worth taking. An agent holding private data that can send but never reads anything external cannot be instructed. An agent that reads untrusted content and holds private data but has no outbound path has nowhere to put what it takes. Our system had leg one in a modest form, a list of who attended and their addresses, and leg three in full, the ability to send. It did not have leg two. That is the whole story. Here is the corollary that makes this hard, and it deserves stating plainly: all three legs at once is exactly what makes an agent useful. An assistant that reads your inbox, holds your customer records and replies on your behalf is valuable precisely because it does all three. Every capability you add to make the thing worth having moves it closer to the trifecta. This is not a defect awaiting a patch, it is a property of the shape. Willison is blunt about where the research stands: "Here's the really bad news: we still don't know how to 100% reliably prevent this from happening." And from his April 2025 piece on the CaMeL defence: "Sadly, there is no known reliable way to have an LLM follow instructions in one category of text while safely applying those instructions to another category of text." OWASP concedes the same, noting that its recommended controls reduce impact but that it is unclear whether fool-proof prevention exists given how the technology works.

The most common prompt injection in the wild is plain visible text carrying a social engineering frame, not invisible white-on-white code. Unit 42 at Palo Alto Networks measured plain text at 37.8 per cent of deliveries in research published 3 March 2026. That study was the first published telemetry of indirect prompt injections found on real websites rather than in a controlled test. Two findings run against the popular mental image. HTML attribute cloaking came second at 19.8 per cent and CSS rendering suppression third at 16.9 per cent, so the single most common approach is to write the instruction where anyone could read it and rely on nobody reading. And 85.2 per cent of the injections used a social engineering frame, typically an authority override along the lines of "this is a security update" or "this is a system message". Now look again at what landed in our inbox. Plain visible text with no obfuscation whatsoever, an authority frame in "This is the user", one sentence long. One unsolicited attempt against one small Australian system, sitting precisely inside the majority profile a global threat intelligence team measured across a dataset. The practical consequence is where you spend effort. Do not start by hunting for hidden characters. Start by assuming any free-text field a stranger can fill in, your contact form, your support inbox, a customer note, may contain an instruction addressed to your agent, written in ordinary English, looking completely unremarkable to a person. Two cautions on reading that research honestly. Unit 42 published twelve worked case studies, including the first observed real-world case of AI-based ad-review evasion, but states no total detection count, so twelve is not the number of attacks found. And what was documented is payloads planted at scale, not confirmed hijacks of production agents. Google Threat Intelligence described most of what it saw as experiments, pranks and search manipulation with a smaller malicious tail. The wave is real and growing. Mass compromise of live agents is not what this evidence shows.

Prompt injection has already succeeded against production systems at Microsoft, Google, Salesforce, Slack and GitHub, every case disclosed by named researchers with dates between August 2024 and September 2025. None of these were small teams with thin security budgets. | Incident | System | Disclosed | How the injection arrived | | - | - | - | - | | PromptArmor disclosure | Slack AI | 20 August 2024 | A message posted in a public channel, indexed for retrieval | | Invariant Labs demonstration | GitHub MCP server | May 2025 | A single issue filed on a public repository | | EchoLeak | Microsoft 365 Copilot | June 2025 | An email sitting unopened in the mailbox | | Invitation Is All You Need | Google Gemini | August 2025 | The title of a calendar invite | | ForcedLeak | Salesforce Agentforce | September 2025 | The description field of a website lead form | Slack AI could be made to leak data from private channels the attacker had no access to, by posting an instruction in a public channel that the assistant later retrieved. MITRE ATLAS records it as case study AML.CS0035. EchoLeak, found by Aim Security and tracked as CVE-2025-32711, was zero-click. NVD describes it as an AI command injection in M365 Copilot permitting information disclosure over a network. Microsoft rated it 9.3 critical while NVD's own assessment scores it 7.5 high, so check which figure you are being quoted. Microsoft patched it server-side and said there was no exploitation in the wild. The Gemini work by Ben Nassi, Stav Cohen and Or Yair showed fourteen attacks, several ending in control of real smart-home devices through a connected agent. Google fixed the issues and argued the researchers had altered default settings on who may add invites, which the researchers dispute in part. ForcedLeak deserves the most attention from Australian small businesses. Noma Security showed an injection hidden in a Web-to-Lead form submission, executed when staff later asked the AI agent to process that lead. That is a stranger filling in the contact form on your website, and it is the shape of the most common automation currently being sold to small businesses.

Sixty-nine per cent of Australian organisations are using autonomous AI agents while only 22 per cent have advanced agent governance models, according to Deloitte's State of AI in the Enterprise 2026. Seven in ten are running agents. Two in ten can govern them. We should be straight about a limit here. We could not find any Australian measurement of prompt injection or agent security incidents specifically. What exists is adoption and governance data, which supports the argument by inference rather than proving it. Anyone quoting you an Australian prompt injection statistic should be asked for the primary source. The international numbers are firmer. Google Threat Intelligence reported on 23 April 2026 that malicious-category prompt injections found on the web rose 32 per cent in relative terms between November 2025 and February 2026, measured by scanning between 2 and 3 billion pages per monthly Common Crawl archive. On organisational exposure, the Cloud Security Alliance and Token Security surveyed 418 organisations and published on 21 April 2026 that 65 per cent had an AI-agent-related incident in the previous twelve months, and that 82 per cent have unknown AI agents running somewhere in their infrastructure. The second figure is arguably worse. You cannot govern what you have never listed, and the likeliest place a small business finds an ungoverned agent is a connector somebody switched on inside a SaaS product. On whether better models fix it, the largest public test we know of is the Agent Red-Teaming Challenge run by Gray Swan AI with the UK AI Security Institute between 8 March and 6 April 2025. It gathered 1.8 million adversarial attempts against 22 frontier AI agents and produced roughly 62,000 successful breaks. The finding that matters is not any single success rate. Every tested behaviour was breached at least once. Not most. Every one.

Stop asking how to stop the injection and start asking what the blast radius is when it lands. Assume it arrives, then design so that its arrival is survivable. That is not defeatism, it is the working engineering position. Separate the reader from the actor. This is the rule our own system got right by accident and the one to adopt deliberately. The component ingesting untrusted content should not hold the credential for the irreversible action. The thing summarising your inbox is not the thing with the payments key, and the handoff between them is fixed code with a fixed contract, not free text the model can rewrite. Scope credentials to the smallest thing that works. The named root cause of the GitHub MCP incident was an overly broad personal access token. Every token your agent holds should answer the question of which specific action needs it and why nothing narrower would do. Require a person before anything irreversible. This is OWASP's fifth mitigation and the control most likely to be worth its friction. Money leaving, an email reaching a customer, a record being deleted, a public post: at those moments a human sign-off turns a total compromise into an odd-looking draft somebody declines. The person is not a good injection detector. The person is a circuit breaker. Constrain what the agent can do at all. The June 2025 paper "Design Patterns for Securing LLM Agents against Prompt Injections", written by fourteen researchers across IBM, Invariant Labs, ETH Zurich, Google and Microsoft, states the trade in its own abstract: "These design patterns constrain the actions of agents to explicitly prevent them from solving arbitrary tasks." Security here is bought with capability rather than added on top of it. Do not buy a percentage. Filtering helps and you should use it, but it does not close the gap. On the newer AgentDyn benchmark a leading filter still shows a 27.15 per cent attack success rate, and defences tuned on one benchmark do not transfer to another. Willison's objection to guardrails sold on a 95 per cent block rate is that in security 95 per cent is a failing grade, because an attacker who can retry keeps retrying until the remainder lands.

Australia's cyber agency has co-signed formal guidance on agentic AI. The Australian Signals Directorate's Australian Cyber Security Centre joined five partner agencies in publishing "Careful Adoption of Agentic Artificial Intelligence (AI) Services" on 1 May 2026. The co-sealing agencies are CISA and the NSA in the United States, the Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK alongside the ASD's ACSC. The document runs to roughly thirty pages with over a hundred recommendations, organised around five categories of risk: privilege, design and configuration, behavioural, structural, and accountability. We are describing its position in our own words rather than quoting it, deliberately, because we have not read the source PDF directly and would rather paraphrase accurately than put a secondhand sentence inside quotation marks. In substance it treats prompt injection as the most persistent and hardest to fix threat facing agentic systems, arising from a design constraint of language models that input sanitisation cannot fully resolve. It recommends adopting agentic AI incrementally, giving agents least privilege, distinct identity and monitoring, keeping people in oversight, and continuing to assess against a threat model that is still moving. Its broader posture is that until practices and standards mature, organisations should assume these systems may behave unexpectedly and should prioritise resilience, reversibility and containment ahead of efficiency gains. If you want the exact wording, get the document from cyber.gov.au rather than trusting any summary, ours included. What matters for an Australian business is that our national cyber agency has co-signed it, which makes not knowing agents carried this risk a hard position to hold in a post-incident review. On privacy, the OAIC published guidance on 21 October 2024 covering both the adoption of commercially available AI products and the development of generative models. Privacy obligations apply to personal information input into an AI system as well as to output containing personal information, and organisations are expected to do real due diligence before adoption, including how human oversight will be built into the process and who can reach the personal information going in or coming out.

Nobody outsmarted an attacker here. No detection fired, no alert was raised, and nobody spotted the reply in real time. The system carried on doing what it was written to do, which happened not to include reading email. The defence was structural and partly accidental. Two of the three properties, the absent read path and the check-in derived recipient list, exist because the job never needed anything else. Only the idempotent ledger was a deliberate safety decision, and it was made to avoid double-issuing codes rather than to stop an attacker. Two other limits are worth stating. The OAIC's October 2024 guidance predates agentic AI and does not address prompt injection, so treat any link between the two as our reasoning rather than regulator policy. And the ACSC Essential Eight does not cover AI either. It is an eight-control maturity model for endpoint and network hardening. Restricting administrative privileges maps conceptually onto least-privilege credentials for agents, and regular backups map onto the data destruction intent Unit 42 measured in 14.2 per cent of the attacks it studied, but anyone telling you the Essential Eight now covers AI agents is wrong. That honesty is the actual lesson. The systems surviving this class of attack are not the ones with the best detection. They are the ones where the attacker's message had nowhere to go, usually because somebody kept the design smaller than it could have been. Every capability you do not add is an attack you never have to defend. When you do add one, the question is not whether it could be abused, because it could, but what it would cost you when it is.

Was the attack sophisticated? No. It was one sentence of plain English with an authority claim and a request, and no obfuscation at all. That is why it is worth studying. Unit 42's March 2026 telemetry found plain visible text was the most common delivery method in the wild at 37.8 per cent, and 85.2 per cent of in-the-wild injections used a social engineering frame. The attempt against us was the majority case, not an outlier. Does a restricted or read-only credential solve this? It helps, but it is not what saved us and it is often impractical. Our sending system held a fully capable send credential, including a Gmail scope permitting send, and also sent through a transactional email provider. It could send freely. What it did not have was any channel through which someone outside could reach it with an instruction. Where you genuinely need send or payment capability, control what is allowed to talk to that component. Can I just filter for injection attempts? Filtering is a reasonable layer and OWASP recommends it, but treat it as reducing volume rather than closing the hole. Detection performance does not transfer between benchmarks, and on the AgentDyn benchmark a leading filter still shows a 27.15 per cent attack success rate. The argument against buying a block-rate percentage is that an attacker who can retry keeps retrying until the residual gap lands. Will a better model fix it? There is no evidence it does. The Gray Swan and UK AI Security Institute challenge collected 1.8 million adversarial attempts against 22 frontier agents in 2025, produced around 62,000 successful breaks, and found every tested behaviour was breached at least once. On the AgentDojo benchmark the model with the highest usefulness in benign conditions also recorded the highest targeted attack success rate. Better instruction-following cuts both ways. What is the single most useful change to make? Separate the component that reads untrusted input from the component holding the credential for the irreversible action, then put human approval in front of anything you cannot undo. If you do only one thing, do the separation. It is the property that made a live system with full send capability immune to an instruction posted directly into its own mailbox. How would I know if this happened to my system? Most likely you would not, which is the uncomfortable part. We know about this attempt only because a person read the mailbox. Cloud Security Alliance research published in April 2026 found 82 per cent of organisations have unknown AI agents running in their infrastructure and 65 per cent had an agent-related incident in twelve months. Start with a list of every agent and connector you run and what credentials each holds.