What CREST AI-Enabled Penetration Testing actually covers

There are two entirely different things people mean when they say "AI penetration testing". One is using AI to help carry out a test of ordinary systems. The other is testing an application that itself uses AI. They are different purchases, with different deliverables and different skills behind them, and they sit in different places as far as accreditation goes. This matters to us more than to most people writing about it, because we hold the accreditation that is most often misread as covering both.

So this is partly an explainer and partly a self-imposed limit.

The two things, kept apart

The first is AI-assisted testing. The target is your ordinary estate: a web application, an external footprint, an internal network. Nothing in it uses AI. The AI sits on the tester's side, helping with reconnaissance, reading large volumes of source and configuration, drafting reproduction steps. What you receive is a normal penetration testing report. The AI changes how the work was done, not what was tested.

The second is testing an AI feature. Here the AI is the target: a language model feature in your product, a chatbot that can call your APIs, a pipeline that answers questions from your internal documents, or an agent that can act on a user's behalf. The flaws worth looking for here largely did not exist in your estate two years ago, and a conventional web application test will not find them, because it is not looking for them.

Which one the accreditation covers

CREST's AI-Enabled Penetration Testing accreditation covers how a provider uses AI responsibly within its own testing services. It is an assurance about delivery: governance of AI use, data handling, and where the human stays in control. It is formalised as Annex B to the existing penetration testing standard, and an annex to a delivery standard is about delivery. Applications opened on 28 July 2026, and CREST's own announcement said over fifty member companies had begun the process. Infosecurity Magazine reported on the first cohort of ten. We are one of them, with recognition dated 2 September 2026.

It does not accredit the testing of AI systems. CREST has said a separate accreditation, Security Testing of AI, will cover the evaluation of AI-enabled systems themselves. At the time of writing that accreditation does not exist, so nobody holds it.

The honest position is therefore narrower than the marketing around it. The accreditation says something real and independently audited about how a provider governs its use of AI in delivery. It says nothing about competence at testing AI systems. We hold it, and that is exactly what it means in our case. Any provider citing it as proof they can test your chatbot is stretching it, and if we ever do that, push back.

Without an accreditation, the evidence for the second thing comes from the ordinary places: individual qualifications, the scopes a tester has actually worked on, and findings you can read and judge for yourself. That is weaker assurance than a standard, and it is the only form available at present.

What AI-assisted testing actually gets you

AI is genuinely good at breadth and at reading. It will work through a large codebase faster than a person, cross-reference two hundred endpoints for the same pattern, pull the interesting lines out of a noisy configuration file, and draft reproduction steps so the tester can move to the next finding. Used well, it shifts hours out of the mechanical part of an engagement and into the adversarial part where serious flaws are found.

It is poor at judgement and it is confidently wrong. A model will produce a plausible finding for a vulnerability that is not present, with a severity rating attached. It will also miss what matters most, because what matters most is usually a chain: a low-severity information leak on the host that happens to hold a service account with far too much access. None of that chain is visible in a single file.

So a finding still has to be validated by hand, and the report should show the request and response that prove it. A finding nobody can demonstrate is a suggestion.

Then there is the data question, which buyers ask about least and should ask about most. During a test a provider handles your source code, your configuration, and at the end a document listing how to break into your systems. If that material passes through a third-party model, it has left your control and possibly theirs. Reasonable questions: which model, operated by whom, retained for how long, used for training or not, processed in which country. There is a contractual trap too, because the confidentiality terms you signed with the provider may promise more than their model provider's terms support. This is precisely the ground the AI-Enabled accreditation covers, which is why a narrow accreditation is still worth something.

What testing an AI feature actually gets you

The flaw classes are specific, and worth knowing by name even if you never buy the test.

• Direct prompt injection. A user types something that overrides the instructions the developer gave the model. The interesting cases are not rude answers, they are the ones where those instructions were the only thing preventing an action.
• Indirect prompt injection. The instructions are hidden in content the model reads rather than typed by the user: a CV with white text in it, read by a screening assistant, or a support ticket, a calendar invitation, a web page it was asked to summarise. The attacker never touches your interface, they leave instructions where your model will read them.
• Excessive agency. The model can do things: call an API, issue a refund, send mail, write to a record. The risk is rarely the model, it is the permissions of the account it acts through. A support assistant wired to a service account that can read every customer record will eventually read one it should not.
• Data leakage through retrieval. A pipeline built over a shared drive inherits everything in that drive, including the folder nobody meant to index. If authorisation lives in the prompt rather than the retrieval layer, the model is one well-phrased question away from answering from a document the user may not see. Access control belongs where documents are fetched, not in a politely worded prompt.

The AI feature also sits behind ordinary authentication, an ordinary API and an ordinary upload handler, and those still break in the ordinary ways. An AI application test that ignores the conventional layer underneath is doing half the job.

Questions that separate the two

1. Which of the two am I buying? Ask for it in the written scope, in those terms. A provider who cannot answer cleanly has not thought about it.
2. If it is testing my AI feature, which flaw classes are in scope? Direct and indirect injection, agency and tool permissions, retrieval-layer authorisation, and how each finding will be demonstrated.
3. If you cite the CREST AI-Enabled accreditation, which part covers testing AI systems? The accurate answer is that none of it does. This question separates careful providers from confident ones.
4. Where does my data go? Named models and operators, retention, training, geography, and whether your confidentiality terms survive the journey.
5. Who validates findings, and will the report show the evidence? You want a human name and a yes.
6. Does the model have tools, and what can they do? If the provider has not asked you this, they are not testing for excessive agency.

Where we would rather you spent the money

For most organisations neither of these is the urgent purchase, and it would suit us to pretend otherwise.

If your estate has never been tested, an ordinary penetration test will find more per pound than any AI-specific engagement. Exposed administrative interfaces, unpatched services, weak authentication and forgotten hosts are still where breaches come from, and are cheaper to find than prompt injection. If an AI feature has been bolted onto an application nobody has ever tested, the untested application is the larger problem.

It is also worth being calm about the risk. If your AI feature has no tool access and retrieves only from documents you publish anyway, the realistic worst case is an embarrassing screenshot rather than a breach. The moment it gains tool access or reads private documents that changes completely, and that is when the specific test earns its place. Our page on choosing a provider covers the general questions, and what a test costs gives the figures.

Common questions