Your AI penetration test is probably a vulnerability scan
Scanners have been rebranded far faster than they have been improved. The question worth asking is not whether AI was involved. It is who read what it produced, and whether they could tell a finding from a guess.
There is nothing wrong with a vulnerability scan. We run them, they are cheap, and they find real problems. What is wrong is selling one as a penetration test because a language model wrote the summary. What is worse is selling one to an organisation whose supplier questionnaire says penetration test, who now believes it has had one.
Three different things are being sold as AI penetration testing
The phrase covers three products that have almost nothing in common. Separating them is most of the work, because the price, the coverage and what you can honestly tell a customer afterwards are different in each case.
| What is actually happening | What you get | What it is worth |
|---|---|---|
| A scanner with a language model writing the report | The findings a scanner has produced for twenty years, in better prose | Genuine value at scanner prices. It is not a penetration test. |
| An autonomous agent chaining tool output on its own | Wider coverage than a scanner, with no accountable person behind the conclusions | Interesting, improving, and currently hard to sign off against a clause |
| A qualified tester using AI as a tool | A penetration test, with more ground covered than the same person unaided | A penetration test |
The first is by far the most common, and it is the one most likely to arrive with the word “AI” in the proposal and a four figure price next to it. The giveaway is usually the report: long, confident, evenly formatted, and with every finding described in the same voice whether it is a missing security header or a way into another customer’s data.
What a scanner cannot reach, whatever is driving it
The National Cyber Security Centre is direct about this in its guidance on scanning tools. On coverage, automated vulnerability scanning “cannot compare to manual processes such as penetration testing”. Its recommended division of labour is that scanning clears the “low hanging fruit” so that penetration testing can concentrate on “complicated security issues that are better suited to a human”.
That is not a complaint about tooling quality. It is a statement about what the category can and cannot do, and it holds whether the scanner is driven by a cron job or by an agent. The things that sit outside it are consistent:
- Business logic. A scanner does not know what your application is for. It cannot tell that a discount code field accepts negative numbers, that a refund can be issued twice, or that an order can be advanced to dispatched without being paid for. None of that is a vulnerability in any database. All of it costs money.
- Authorisation between users. Changing an order number in a URL and seeing somebody else’s invoice is the single most commercially damaging finding on most small web applications. Finding it requires two accounts, a sense of what the number means, and someone to notice that the name on the returned document is not the name they logged in as.
- Chains. A tester’s most valuable output is usually two or three findings that are individually unremarkable and jointly serious. An information leak that exposes usernames plus a login page without rate limiting is not two low findings. It is a way in.
- Anything behind a workflow. If reaching the interesting functionality takes a five step process with a document upload in the middle, a scanner reaches step one.
- What is only wrong in your context. Every estate has a configuration that is perfectly acceptable somewhere and dangerous here. Nothing automated knows which yours is.
The output is the easy part. Reading it is the job.
This is the part buyers underestimate, and it is where most of the damage is done. Pointing a tool at an estate is not difficult, and it is getting easier every quarter. Producing output has never been the constraint. A scan of a modest network will hand you several hundred findings before anybody has thought about any of them, and the entire value of the engagement is in deciding which three of those matter to you this month.
That decision goes wrong in three directions, and all three are expensive:
- A false positive is accepted. Your team spends a fortnight and a change window on a patch for something that was never there, which is how security work loses its budget.
- A real finding is dismissed. Somebody marks it as a false positive because they could not reproduce it, which is a different statement from it not being real.
- The severity is wrong for you. This is the most common and the least noticed. An outdated protocol on an internal print server arrives rated high because the tool rates it high everywhere. A way of reading another customer’s records arrives rated medium because the tool has no idea who your customers are. Acted on in order, that report gets the print server fixed and leaves the serious problem open.
Sorting that out requires somebody who can reproduce a finding by hand. Not read about it, not describe it, reproduce it. If the person handing you the report cannot do that, they are not in a position to tell you whether any individual line in it is true, and the confidence of the writing is not evidence either way. A generated report is fluent by construction. Fluency and accuracy are unrelated.
There is a quick test for this, and it costs you one meeting. Ask whoever is presenting the report to demonstrate one finding live, on your system, with you watching. A tester will pick the most interesting one and enjoy showing you. Somebody forwarding tool output will offer to send more documentation.
What happens when nobody checks the output
There is a well documented example of this running at scale. The curl project, which maintains software installed on billions of devices and runs a serious security team, publishes figures on what arrives through its vulnerability reporting. Its maintainer reported that by mid 2025 roughly 20 per cent of submissions were AI generated noise, while about 5 per cent of submissions that year turned out to be genuine vulnerabilities. In January 2026 the project closed its bug bounty programme, explicitly to remove the incentive for poorly researched submissions.
Consider what that means for you. That is a team of experienced maintainers, who wrote the code, triaging reports about software they know intimately, and the volume of plausible looking and ultimately wrong material was enough to make the programme not worth running. A report landing on the desk of an IT manager who did not write the application, about an estate they inherited, has considerably less defence than that.
CREST, which accredits penetration testing providers in the UK, now runs a specific accreditation for AI-enabled penetration testing. Its chief executive, Nick Benson, has put the reasoning plainly: unvalidated automated output “can generate false positives or miss logical flaws”, which is “why human oversight is still so important”. The accreditation examines transparency, operational controls and safety boundaries, and providers must hold the general penetration testing accreditation as well. The point of it is not that AI is used. It is that somebody is answerable for what the AI said.
Seven questions that separate a test from a scan
You do not need to be technical to ask these, and you do not need to understand the answers in detail. What you are listening for is whether the person can answer at all.
- Who is the named tester, and what do they hold? Not the company’s accreditation. The individual doing the work, and their certifications, which you can usually verify yourself.
- Will you demonstrate a finding live? The single most useful question on this list.
- Which findings were verified by hand, and how is that marked in the report? A good report distinguishes confirmed from reported. A generated one rarely does.
- What did you test that a scanner cannot see? If the answer is vague, you have your answer.
- How many days of human effort are in this? A two day engagement is a perfectly reasonable purchase. A two day engagement sold as a full application test is not.
- Who signs the report, and are they accountable for it? A named person with a professional body behind them has something to lose.
- Is the provider accredited for this work, and can I check it independently? CREST publishes its accredited suppliers. So does IASME. Verification takes about two minutes.
Our own answers to all seven are published on our credentials page, including the individual certifications and how to verify each one without asking us.
Where AI genuinely helps, in the hands of somebody who can check it
None of this is an argument against using AI in testing. We use it, we hold CREST’s AI-enabled penetration testing accreditation, and it has made the work better. The distinction is between a tool that extends a tester and a product that replaces one.
What it is genuinely good at: reading a large unfamiliar codebase in minutes rather than days, generating variations on a payload far faster than a person can type them, mapping an authentication flow nobody documented, noticing that the same mistake appears on two hundred endpoints when a human would have sampled twelve, and drafting the descriptive parts of a report so the tester spends their hours on the findings instead of the formatting.
What stays with the tester, every time: deciding what is in scope, confirming the engagement is authorised, choosing whether to exploit something and how far, recognising that three dull findings combine into one serious one, understanding what your organisation actually does so that severity means something, and signing the report. Those are judgement, not throughput, and judgement is the thing being quietly removed from the cheap version of this product.
We have written separately about what the CREST AI-enabled accreditation does and does not cover, which is worth reading if a provider has quoted it at you, and our AI penetration testing page sets out how we run it.
What we would actually tell you to buy
Buy scanning. It is cheap, it should be running continuously rather than annually, and it will find the unpatched and the misconfigured faster and more cheaply than any person. If somebody is selling you an AI product that does this well, that may be a good purchase at the right price.
Buy a penetration test when you need somebody to reason about your system: before a product launch, when a customer contract demands it, when you hold data whose exposure would end the relationship, and periodically because applications change. Expect to pay for days of a qualified person’s time, because that is what you are buying. What a penetration test costs sets out the UK ranges.
What we would not do is let anybody sell you the first as the second. If your contract, tender or insurance policy uses the words penetration test, somebody drafted that wording deliberately, and a generated report against a scanner’s output is unlikely to satisfy it if anybody looks closely. The time to discover that is not during a customer’s security review.
If you have a quote in front of you and you are not sure which of the three things you are being offered, send it to us. We will tell you what it looks like, including when the honest answer is that it is good value and you should take it.
Common questions
Is AI penetration testing a real thing, or just marketing?
How can I tell whether my AI pen test was really just a scan?
Will an AI penetration test satisfy my client’s contract or a tender?
Are findings from AI tools less reliable than findings from a person?
Does Solusec use AI in its penetration testing?
Automated testing is much cheaper. Is that not a good enough reason?
What is the single best question to ask a provider?
Can AI replace a penetration tester?
Should we stop running vulnerability scans, then?
Related
Ready to talk?
Scoping conversations are free and there is no sales team to get past. Tell us what you're dealing with and we'll tell you honestly what you need.