AI & LLM Penetration Testing

Why this needs its own testing

An LLM application breaks a foundational assumption of application security: that instructions and data are separable. When a model reads a document, a web page or an email, it cannot reliably distinguish content it should process from instructions it should follow. Everything downstream of that is a consequence.

If you have built a retrieval-augmented chatbot over internal documents, or an agent that can call APIs on a user's behalf, you have created attack surface that conventional testing does not address.

Prompt injection

Direct injection, where a user manipulates the model's behaviour, and — more seriously — indirect injection, where instructions are planted in content the model ingests. A malicious instruction in a PDF, a support ticket, a calendar invite or a web page the agent retrieves, executing when the model processes it.

Data leakage and access control

Whether the model can be induced to reveal its system prompt, other users' data, or documents the requesting user should not be able to see. Retrieval systems very commonly inherit no access control at all — the vector store returns the most semantically relevant chunk regardless of who is asking. This is the single most common serious finding we see.

Agent and tool abuse

Where a model can call functions, send email, query databases or execute code, the question becomes what an attacker can make it do. Excessive agency, missing confirmation on consequential actions, and the classic chain: untrusted content enters, model follows embedded instruction, tool executes with the application's privileges.

Guardrail bypass and output handling

Testing whether safety filters survive encoding, indirection, role-play framing and multi-turn manipulation. And critically, what your application does with model output — LLM responses rendered unsanitised into a web page produce cross-site scripting the same as any other untrusted input.

The conventional application underneath

An AI feature sits inside a normal web application with normal authentication, normal APIs and normal access control. We test that too. It would be a poor outcome to hand over a sophisticated report on prompt injection while missing an IDOR in the conversation history endpoint.

Where this fits

Testing is aligned to the OWASP Top 10 for LLM Applications and informed by the NCSC's guidance on secure AI system development. If you are pursuing certification or answering client security questionnaires about your AI features, the report gives you defensible evidence that the system was assessed by a CREST accredited provider.

Common questions

We just use the OpenAI or Anthropic API — is that our responsibility?

Yes. The provider secures the model; you are responsible for what you connect it to, what data you put in its context, what tools you let it call and what you do with its output. Almost all real-world risk sits in your integration rather than in the model itself.

Is this different from red teaming the model?

Yes, and the distinction matters. Model red teaming assesses whether the model produces harmful content — that is the provider's problem. We assess whether your application can be manipulated into leaking data, taking unauthorised actions or bypassing your access control. That is your problem.

Can prompt injection be fixed?

Not eliminated with current architectures — you should be sceptical of anyone claiming otherwise. It can be contained: treat all model output as untrusted, enforce authorisation at the tool and data layer rather than through prompt instructions, require confirmation for consequential actions, and apply least privilege to everything the agent can reach. We report against that model.

Our AI feature is small. Is testing worth it?

If it can read data a user should not see, or take an action on their behalf, the size of the feature is irrelevant to the size of the breach. If it only rewrites marketing copy with no access to internal data, it is a lower priority and we will say so.