Your AI penetration test is probably a vulnerability scan

Scanners have been rebranded far faster than they have been improved. The question worth asking is not whether AI was involved. It is who read what it produced, and whether they could tell a finding from a guess.

There is nothing wrong with a vulnerability scan. We run them, they are cheap, and they find real problems. What is wrong is selling one as a penetration test because a language model wrote the summary. What is worse is selling one to an organisation whose supplier questionnaire says penetration test, who now believes it has had one.

Three different things are being sold as AI penetration testing

The phrase covers three products that have almost nothing in common. Separating them is most of the work, because the price, the coverage and what you can honestly tell a customer afterwards are different in each case.

What the phrase is being used to describe
What is actually happeningWhat you getWhat it is worth
A scanner with a language model writing the reportThe findings a scanner has produced for twenty years, in better proseGenuine value at scanner prices. It is not a penetration test.
An autonomous agent chaining tool output on its ownWider coverage than a scanner, with no accountable person behind the conclusionsInteresting, improving, and currently hard to sign off against a clause
A qualified tester using AI as a toolA penetration test, with more ground covered than the same person unaidedA penetration test

The first is by far the most common, and it is the one most likely to arrive with the word “AI” in the proposal and a four figure price next to it. The giveaway is usually the report: long, confident, evenly formatted, and with every finding described in the same voice whether it is a missing security header or a way into another customer’s data.

What a scanner cannot reach, whatever is driving it

The National Cyber Security Centre is direct about this in its guidance on scanning tools. On coverage, automated vulnerability scanning “cannot compare to manual processes such as penetration testing”. Its recommended division of labour is that scanning clears the “low hanging fruit” so that penetration testing can concentrate on “complicated security issues that are better suited to a human”.

That is not a complaint about tooling quality. It is a statement about what the category can and cannot do, and it holds whether the scanner is driven by a cron job or by an agent. The things that sit outside it are consistent:

  • Business logic. A scanner does not know what your application is for. It cannot tell that a discount code field accepts negative numbers, that a refund can be issued twice, or that an order can be advanced to dispatched without being paid for. None of that is a vulnerability in any database. All of it costs money.
  • Authorisation between users. Changing an order number in a URL and seeing somebody else’s invoice is the single most commercially damaging finding on most small web applications. Finding it requires two accounts, a sense of what the number means, and someone to notice that the name on the returned document is not the name they logged in as.
  • Chains. A tester’s most valuable output is usually two or three findings that are individually unremarkable and jointly serious. An information leak that exposes usernames plus a login page without rate limiting is not two low findings. It is a way in.
  • Anything behind a workflow. If reaching the interesting functionality takes a five step process with a document upload in the middle, a scanner reaches step one.
  • What is only wrong in your context. Every estate has a configuration that is perfectly acceptable somewhere and dangerous here. Nothing automated knows which yours is.

The output is the easy part. Reading it is the job.

This is the part buyers underestimate, and it is where most of the damage is done. Pointing a tool at an estate is not difficult, and it is getting easier every quarter. Producing output has never been the constraint. A scan of a modest network will hand you several hundred findings before anybody has thought about any of them, and the entire value of the engagement is in deciding which three of those matter to you this month.

That decision goes wrong in three directions, and all three are expensive:

  • A false positive is accepted. Your team spends a fortnight and a change window on a patch for something that was never there, which is how security work loses its budget.
  • A real finding is dismissed. Somebody marks it as a false positive because they could not reproduce it, which is a different statement from it not being real.
  • The severity is wrong for you. This is the most common and the least noticed. An outdated protocol on an internal print server arrives rated high because the tool rates it high everywhere. A way of reading another customer’s records arrives rated medium because the tool has no idea who your customers are. Acted on in order, that report gets the print server fixed and leaves the serious problem open.

Sorting that out requires somebody who can reproduce a finding by hand. Not read about it, not describe it, reproduce it. If the person handing you the report cannot do that, they are not in a position to tell you whether any individual line in it is true, and the confidence of the writing is not evidence either way. A generated report is fluent by construction. Fluency and accuracy are unrelated.

There is a quick test for this, and it costs you one meeting. Ask whoever is presenting the report to demonstrate one finding live, on your system, with you watching. A tester will pick the most interesting one and enjoy showing you. Somebody forwarding tool output will offer to send more documentation.

What happens when nobody checks the output

There is a well documented example of this running at scale. The curl project, which maintains software installed on billions of devices and runs a serious security team, publishes figures on what arrives through its vulnerability reporting. Its maintainer reported that by mid 2025 roughly 20 per cent of submissions were AI generated noise, while about 5 per cent of submissions that year turned out to be genuine vulnerabilities. In January 2026 the project closed its bug bounty programme, explicitly to remove the incentive for poorly researched submissions.

Consider what that means for you. That is a team of experienced maintainers, who wrote the code, triaging reports about software they know intimately, and the volume of plausible looking and ultimately wrong material was enough to make the programme not worth running. A report landing on the desk of an IT manager who did not write the application, about an estate they inherited, has considerably less defence than that.

CREST, which accredits penetration testing providers in the UK, now runs a specific accreditation for AI-enabled penetration testing. Its chief executive, Nick Benson, has put the reasoning plainly: unvalidated automated output “can generate false positives or miss logical flaws”, which is “why human oversight is still so important”. The accreditation examines transparency, operational controls and safety boundaries, and providers must hold the general penetration testing accreditation as well. The point of it is not that AI is used. It is that somebody is answerable for what the AI said.

Seven questions that separate a test from a scan

You do not need to be technical to ask these, and you do not need to understand the answers in detail. What you are listening for is whether the person can answer at all.

  1. Who is the named tester, and what do they hold? Not the company’s accreditation. The individual doing the work, and their certifications, which you can usually verify yourself.
  2. Will you demonstrate a finding live? The single most useful question on this list.
  3. Which findings were verified by hand, and how is that marked in the report? A good report distinguishes confirmed from reported. A generated one rarely does.
  4. What did you test that a scanner cannot see? If the answer is vague, you have your answer.
  5. How many days of human effort are in this? A two day engagement is a perfectly reasonable purchase. A two day engagement sold as a full application test is not.
  6. Who signs the report, and are they accountable for it? A named person with a professional body behind them has something to lose.
  7. Is the provider accredited for this work, and can I check it independently? CREST publishes its accredited suppliers. So does IASME. Verification takes about two minutes.

Our own answers to all seven are published on our credentials page, including the individual certifications and how to verify each one without asking us.

Where AI genuinely helps, in the hands of somebody who can check it

None of this is an argument against using AI in testing. We use it, we hold CREST’s AI-enabled penetration testing accreditation, and it has made the work better. The distinction is between a tool that extends a tester and a product that replaces one.

What it is genuinely good at: reading a large unfamiliar codebase in minutes rather than days, generating variations on a payload far faster than a person can type them, mapping an authentication flow nobody documented, noticing that the same mistake appears on two hundred endpoints when a human would have sampled twelve, and drafting the descriptive parts of a report so the tester spends their hours on the findings instead of the formatting.

What stays with the tester, every time: deciding what is in scope, confirming the engagement is authorised, choosing whether to exploit something and how far, recognising that three dull findings combine into one serious one, understanding what your organisation actually does so that severity means something, and signing the report. Those are judgement, not throughput, and judgement is the thing being quietly removed from the cheap version of this product.

We have written separately about what the CREST AI-enabled accreditation does and does not cover, which is worth reading if a provider has quoted it at you, and our AI penetration testing page sets out how we run it.

What we would actually tell you to buy

Buy scanning. It is cheap, it should be running continuously rather than annually, and it will find the unpatched and the misconfigured faster and more cheaply than any person. If somebody is selling you an AI product that does this well, that may be a good purchase at the right price.

Buy a penetration test when you need somebody to reason about your system: before a product launch, when a customer contract demands it, when you hold data whose exposure would end the relationship, and periodically because applications change. Expect to pay for days of a qualified person’s time, because that is what you are buying. What a penetration test costs sets out the UK ranges.

What we would not do is let anybody sell you the first as the second. If your contract, tender or insurance policy uses the words penetration test, somebody drafted that wording deliberately, and a generated report against a scanner’s output is unlikely to satisfy it if anybody looks closely. The time to discover that is not during a customer’s security review.

If you have a quote in front of you and you are not sure which of the three things you are being offered, send it to us. We will tell you what it looks like, including when the honest answer is that it is good value and you should take it.

Common questions

Is AI penetration testing a real thing, or just marketing?
Both, depending on who is selling it. A qualified tester using AI to cover more ground in the same days is real and is now accredited by CREST as a category. A vulnerability scanner with a language model writing the report is a scanner, whatever it is called. The two are sold under the same phrase at very different prices.
How can I tell whether my AI pen test was really just a scan?
Look for findings that required a human to think. Was anything found by combining two issues? Did anybody test whether one user can reach another user’s data? Is there a finding about how your business logic works rather than about software versions and configuration? If every finding could have come from a tool pointed at your addresses, it did. The other test is to ask the provider to demonstrate a finding live.
Will an AI penetration test satisfy my client’s contract or a tender?
Read the clause rather than guessing. Many name CREST or an equivalent accreditation, specify manual testing, or require a named qualified tester, and an automated report will not meet those. Some ask only for evidence of regular testing, which a scan may satisfy. The cost of getting this wrong lands during a customer’s security review, which is the worst moment to find out.
Are findings from AI tools less reliable than findings from a person?
Unvalidated, yes. CREST’s chief executive has said plainly that unvalidated automated output can generate false positives or miss logical flaws. Validated by somebody who can reproduce each one by hand, they are as reliable as any other finding, because at that point a person has confirmed it. The reliability comes from the validation step, not from the tool.
Does Solusec use AI in its penetration testing?
Yes, and we hold CREST’s AI-enabled penetration testing accreditation for it. It is used to cover more ground inside the same testing days: reading large codebases, generating payload variants, mapping undocumented authentication flows. Every finding that reaches your report has been reproduced by hand by the tester who signs it, and the scope, the exploitation decisions and the severity ratings are theirs.
Automated testing is much cheaper. Is that not a good enough reason?
It is a good reason to buy scanning, which you should be doing anyway and continuously. It is not a good reason to buy a scan instead of a test, because the two find different things. The cheap option is excellent at the problems a patching process should already be catching, and blind to the ones that put you in front of the Information Commissioner.
What is the single best question to ask a provider?
Ask them to demonstrate one finding live, on your system, while you watch. Somebody who tested your estate will pick their favourite and enjoy showing you. Somebody forwarding tool output will offer more documentation instead. It takes fifteen minutes and tells you more than any proposal.
Can AI replace a penetration tester?
Not at present, and the gap is in judgement rather than in capability. Deciding what matters to your organisation, recognising when three dull findings combine into a serious one, choosing how far to push an exploit on a live system, and being accountable for the conclusion are the parts a buyer is actually paying for. The tooling keeps improving and the honest position is that this will keep moving, but accountability has to sit with a person for a report to be worth anything.
Should we stop running vulnerability scans, then?
No, run more of them. The argument here is about what they are called, not about whether they are useful. Continuous scanning is one of the highest value things a small organisation can do, and the NCSC recommends exactly this division of labour: let scanning clear the routine problems so that testing can spend its time on the ones that need a human.

Related

Ready to talk?

Scoping conversations are free and there is no sales team to get past. Tell us what you're dealing with and we'll tell you honestly what you need.