Using AI to Build a Test Suite for an App You Did Not Write

AI-written tests do catch real bugs, but far fewer than the volume of generated code suggests. Treat them as a draft you review, not a result you accept. The clearest measurement so far is ULT (UnLeakedTestbench), published on arXiv in August 2025. It asked language models to write unit tests for 3,909 real-world Python functions, chosen for high complexity and screened so the answers were not in training data. Averaged across the models tested, the generated tests reached 41.32% accuracy, 45.10% statement coverage and 40.21% mutation score. Mutation score is the number to care about. It measures how often a suite notices when the code is deliberately broken. Around 40% means most injected bugs walk straight past. Coverage scored higher than accuracy, which is the whole problem in one line: the tests ran the code without checking it. Now compare a setup where generation was aimed and reviewed. Meta's ACH system, presented as an industry paper at FSE 2025, generated deliberate faults and then tests to catch them across 10,795 Android Kotlin classes. In sessions with the Messenger and WhatsApp teams, engineers accepted 73% of the tests it produced and judged 36% of them relevant to the privacy concern being targeted. Those two studies measure different things, so the percentages are not directly comparable. The pattern still holds. Tests generated in bulk against whatever code exists are weak. Tests aimed at a named failure and reviewed by a person are worth having.

The most dangerous test an agent writes is one that locks in the bug it just wrote. The code returns the wrong number, the agent reads it, and writes a test asserting that wrong number is correct. Green forever. This happens because the same model produced both halves. When it writes the test it is predicting the most likely test given the code it just wrote, not checking that code against what you asked for. If your total skips GST, the test expects a total without GST. There is a second version of this, and it has been measured. ImpossibleBench, from Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini (October 2025), built coding tasks where the specification and the unit tests contradict each other, so passing is only possible by cheating. Their description of the risk is blunt: "an LLM agent with access to unit tests may delete failing tests rather than fix the underlying bug". On the harder multi-file variant built from SWE-bench, GPT-5 passed, and so by definition cheated, on 76% of those impossible tasks. Kent Beck described the same behaviour in a June 2025 Pragmatic Engineer interview, saying he had trouble stopping agents deleting tests in order to make them pass. The consequence is small and strict. If you did not see a test go red before it went green, you do not know it touches your code at all.

Test the three or four things that would genuinely hurt you if they broke, and ignore the rest. For a small app that list is short, and you can write it out before you write a single test. In order: - The path that touches money or data. Whatever action, if it silently did the wrong thing, would cost someone money or lose their work. Booking, payment, save, submit. - The write path. Form submits, the row lands in the database with the right values, and reads back after a refresh. Most silent failures in AI-built apps live here. - Access rules. One user cannot read or edit another user's rows. Write this as a real test with two users, not a manual check you did once and remember doing. - Whatever broke last week. Every real bug becomes a test the moment you fix it. This is the highest-value test you will ever write, because you have proof the failure is possible. Start with a few tests that drive the app the way a person does: open the page, fill the form, submit, check the result. Browser-driven tests read almost like plain English, so you can judge one without reading much code. Unit tests come after, for calculations you cannot afford to get wrong. Skip tests for layout and styling, tests that mock so much they only prove the mocks work, and anything written to raise a number.

Write the expected answer yourself and let the agent write everything else. The assertion is the one part of a test that has to come from you, because it carries what you wanted rather than what the code does. That is less work than it sounds. You do not need to write code, just one sentence with concrete values: given a $100 job with GST, the total shown is $110. Hand the agent that sentence and let it build the test file around it. A sequence that holds up: 1. Describe the behaviour in plain English, with real numbers, before any code exists. 2. Ask for a failing test only, then run it and watch it fail. Skipping this is how you end up with a test that was already passing and never touched your feature. 3. Ask for the implementation in a separate message. 4. Run the test again. If the agent edited the test too, reject the change and repeat step 3. Step four needs a rule you state out loud: do not modify test files while fixing code. Agents make failing tests pass the cheapest way available, and deleting the test is cheap. Put that in your project's agent instructions file so you are not retyping it every session. The wider finding from the DORA State of AI-assisted Software Development 2025 report applies here too: "AI's primary role is as an amplifier, magnifying an organization's existing strengths and weaknesses." A loose process just gets faster and looser.

Coverage tells you which code has never been run by a test. It does not tell you that code works. Use it to find blank spots, not as a score to chase. Google computes coverage across roughly a billion lines of code daily and reported a median of 78% in "Code Coverage at Google" (Ivankovic, Petrovic, Just and Fraser, ESEC/FSE 2019). A company with more testing resources than anyone sits below 100% on purpose: the last stretch costs more than it returns and breeds tests written only to satisfy the metric. For an app with a few hundred users, the useful target is not a percentage at all. It is a question: if every test passes, would you deploy on a Friday without clicking through anything? If not, name the thing you would click through and write that test next. Repeat until the answer is yes. That sizes the suite to your risk instead of someone else's number. A reasonable resting state for a small AI-built app: the main path a user takes, tested through the browser. Auth boundaries between two accounts, tested. Every calculation involving money or dates, unit tested. Every bug you have already hit, tested. Everything else untested, and that is fine. When a test does go red, the real skill is working out which side is wrong, the test or the code. Getting comfortable reading that failure is a large part of what we teach at buildAcademy.

Can I just ask my agent to add tests to my existing app? You can, and you will get a large suite that mostly describes current behaviour, bugs included. Better to name three things that must not break, ask for tests on those, and check each expected value yourself before keeping the file. Should I write tests before or after the feature? Before, for anything that matters. Writing the expected result first is the only reliable way to stop the agent asserting whatever its own code happens to do. For existing code, write the test from your description of correct behaviour, not from the code. How do I know whether a generated test is any good? Break the code on purpose and run it. Change a number, delete a line, flip a condition. If the test still passes, it is not testing anything. This takes about a minute and is the fastest quality check available to a non-technical builder. Do I need to learn a testing framework to do this? No. You need to read a test and say whether the expected value is right. The agent handles setup, syntax and tooling. Browser-based tests are the easiest place to start because they describe actions a person takes rather than internals. How many tests does a small app actually need? Fewer than you think. A handful covering the main user path, auth boundaries, and any money or date calculations will catch most of what would otherwise reach real users. After that, add one test per bug you actually hit, forever.