Judge an AI testing tool by what survives a manual check, not by how many bugs it reports.
I build an AI testing tool, so let me say the uncomfortable part first: tools like mine sell on volume. "Hundreds of tests in minutes." "Dozens of bugs on the first run." I get why. A big number looks great in a demo.
But think about who gets that number. A tester who receives 23 bug reports has to check 23 bug reports. If most of them are wrong, the tool didn't save anyone time. It just moved the work to someone else's desk.
So we started doing something simple. Every week we point our agent at a public demo app with only the URL. Then we check every single finding by hand on the live site, and we publish all of them, the real ones and the wrong ones. This post is about the second one of those, and what I'd now ask any vendor, us included.
The setup
The target was Toolshop (practicesoftwaretesting.com), the demo hardware store that a lot of testers learn Playwright and Cypress on. It's built for practice, so nobody's business is at risk. But it has real flows: sign-up, sign-in, search, product pages, a cart and checkout.
The agent got the URL and nothing else. No test account, no requirements, no hints.
In one 30-minute run it:
- explored 17 pages,
- mapped 16 user flows and 29 business rules,
- wrote 289 test cases,
- ran 71 of them before it hit the time limit,
- and flagged 23 issues.
Then we sat down with the 23.
The result: 8 real, 9 wrong, 6 we couldn't check
- 8 held up on the live site.
- 9 didn't. Some were design choices, some were demo shortcuts, and one was just our agent being wrong.
- 6 we couldn't check without creating an account, which our teardown rules don't allow.
8 out of 23 is about 35%. If you only count the ones we could check, it's 8 of 17, about 47%. Honestly, that second number is now the one I care about most. I call it the survival rate: how many findings survive a manual check.
It isn't a flattering number to publish about your own product. But I'd rather you hear it from me than find it out in week three of a trial.
What the agent found better than we would have
A rule that drifted in one place. Toolshop's sign-up page says a password must "Be at least 8 characters long." The form enforces 8. The API enforces 8. But type 7 characters and the error says "Password must be minimal 6 characters long."
We went into the source to be sure. The 6 lives in a separate translation file. Each piece looks correct on its own. A person testing the sign-up form sees "8" on the page, types a good password and moves on. The agent read the rule everywhere it appeared and noticed that one copy disagreed.
That's the kind of bug agents are genuinely good at: the same fact written down in several places, with one copy out of date.
Pages in states nobody visits on purpose. Open /checkout with an empty cart and nothing renders under the step bar. No "your cart is empty", no way back. The console tells you why: Cannot read properties of null (reading 'cart_items'). Mistype a product URL and you get a blank page with a "Related products" heading and no "not found" message.
People test the happy path because they know which path is the happy one. An agent exploring without a script doesn't know that, so it ends up in strange states all the time. Sometimes that's useless. Sometimes it's exactly the bug a customer hits.
Small, real inconsistencies. The product page prints the warranty as "2years", while the comparison page prints the same value as "2 years". "Forgot password" submits and shows nothing at all, no confirmation and no error. A "Show differences only" filter leaves rows that are identical for every product.
None of these is critical. All of them are things a customer notices and a team never does.
Where the agent was wrong
It treated a demo shortcut as a critical bug. It paid with the card number 1111-1111-1111-1111 and a made-up gift card, both were "accepted", and it rated that critical. In a real store it would be. Toolshop has no payment processor, so on this app it's expected. Same for an address lookup that invented a street for a fake postcode.
It guessed a business rule and got it wrong. It reported that the eco discount "never applied". When we read the code, the 5% discount applies only when more than half the items in the cart have a CO₂ rating of A or B. The agent's test cart didn't qualify. The rule isn't written anywhere on the site, so the agent made one up and then tested its own invention.
It reported design choices as bugs. A contact form that disappears after sending. An order confirmation that only shows the invoice number. Category pages without the price filter that the home page has. You might not like those choices, but they're choices.
One didn't reproduce. It said an empty sign-up form showed only 10 of its 12 errors. When we checked, all 12 were there.
The pattern is pretty clear. The agent was strongest where the app contradicts itself, and weakest where it had to guess what the business wants. That's the same gap you'd see between a tester in their first week and one who's been on the team for a year. The difference is all the knowledge nobody wrote down.
What we did about it: the agent now keeps its understanding of the app (pages, flows, rules) in a knowledge base that people can read and edit. If it has the discount rule wrong, you fix the rule once, and it stops guessing.
What 289 tests for 17 pages actually told us
The other number that bothered me was 289. That's 17 tests per page. It sounds like great coverage. Then I read them, and a lot of them were the same journey written in different ways.
Our agent was generating tests page by page. To test the checkout page, every test first has to get to checkout: find a product, add it to the cart, open the cart. Each test did that its own way. One searched for the product, one clicked a category, one opened the product URL directly. So the steps to reach the page were different in every test, and a failure in one told you very little about the others.
It also didn't match how people think. A product owner doesn't ask "does the checkout page work?" They ask "can a customer buy something?" That's a flow, not a page.
So we're changing how our agent writes tests: flow by flow. One flow, say "Ordering: pick a restaurant, add items, pay, see the order placed", becomes one group of tests that share the same path, with the same steps from page to page. Then a red result answers the question people actually ask. It's in progress, so I don't have numbers yet. I'll publish them when I do.
You don't need our tool to use the lesson. When any AI tool hands you a big test count, group the tests by user journey yourself. If 289 tests turn into 16 journeys with a pile of near-duplicates each, now you know what you really have.
A 6-step checklist to judge any AI testing tool
Use your own staging app, not the vendor's demo app. Give each tool one release cycle.
- Count the findings. Write down how many issues the tool reports.
- Check every finding by hand. Sort each one into real, wrong or can't check. Survival rate = real ÷ checked.
- Write down why each wrong finding was wrong. Demo shortcut, design choice, guessed rule, flaky step. If most are "guessed rule", the tool needs your business rules, so check whether it lets you give it them.
- Group the generated tests by user journey. How many journeys does it really cover? Does it reach each page the same way every time?
- Change the UI once. Rename a button or move a field. Count how many tests you have to fix by hand.
- Check the evidence. For each real bug, can a developer reproduce it from the report alone? Look for steps, screenshots and a recording.
Then compare tools on survival rate, journeys covered and repair effort. Ignore the raw test count.
What I'd ask a vendor (us included)
- What's your survival rate on an app like mine? Can you show me the wrong findings, not just the right ones?
- Where do I write down business rules so the agent stops guessing?
- What happens to my tests when the UI changes?
- Can I get the tests out as code? (Ours can't today. Ask anyway, because for some teams it's a deal-breaker.)
- Is it safe to point at production? (No, not for any agent that submits forms. Use staging.)
The short version
An AI testing agent is a fast, tireless new tester with zero context. It's very good at finding places where your app contradicts itself, and at wandering into states nobody tests on purpose. It's bad at knowing what your business intends. Judge it by the findings that survive a manual check, and group its tests by the journeys your users actually take.
This article was written with the help of AI. The numbers come from our own teardown and were checked by hand.
