LLM Application Security Testing: What to Put in Scope and How to Set Up the Test
Say you run security for a company whose chatbot goes live next quarter. Someone on the product team has spent time trying to talk it into saying something it shouldn’t, and the results look fine. Then you’re asked to write the scope for an outside test. The temptation is to write “try to break the chatbot.” That sentence is a weak scope, because the interesting question isn’t what the chatbot can be made to say. It’s what the chatbot is able to fetch, and for whom.
This piece is a planning aid for that moment. It applies whoever ends up running the test.
Start with what the assistant can reach
Draw the assistant as a box and then draw every arrow leaving it. One goes to a customer database, another to a claims system, another to a document store or a payment API. Each arrow is a place where the assistant acts on behalf of a user, and each one is something a test should cover.
The picture has layers, and each needs its own question.
The application itself is the chat interface, the instructions the developers wrote for the model, and any filtering around inputs and outputs. Here the questions are about manipulation. OWASP’s Top 10 for LLM Applications names prompt injection, where input changes how the model behaves in ways its builders didn’t intend, including input that arrives in a file or web page the assistant reads, and sensitive information disclosure (OWASP GenAI Security Project, Top 10 for LLM Applications 2025). In plain words, that second one is protected information reaching someone who shouldn’t have it.
The retrieval layer matters if you run a RAG system, meaning the assistant looks up passages from your own documents before answering. The question to ask is whose permissions the lookup uses. If every document goes into one shared index, a question from a junior employee can pull back a passage from a board paper unless something filters results by who is asking. OWASP lists vector and embedding weaknesses as its own risk, and this is the kind of worry it describes.
The APIs and data behind the assistant are the last layer, and they get their own section because one of our projects turned on one.
A customer chatbot, an API and a policy number
Before an insurer’s customer chatbot went live, we were brought in to test it along with the APIs it calls. A tester signed in as one customer and mentioned another customer’s policy number while chatting. The chatbot replied with that other customer’s details. The API behind it had never checked that the number belonged to the person asking, and no error appeared anywhere along the way.
The insurer held the launch until the API was changed to confirm that a requested record belongs to the logged-in customer. Our retest covered two routes: the original chat, and the API with no chatbot in the way. The fix held on both. A longer account is on our AI security testing page.
What carries over to your scope is the ownership check. It’s the same one an API penetration test looks for. In general, a chatbot is another way of reaching your APIs, so write the scope around data: which records and actions can the assistant touch, and for each one, does the system behind it confirm that the person asking is allowed to see it?
How to set up the test
A few choices before testing starts decide how much it can find.
Test in staging where you can, and make sure staging connects the assistant to the same kinds of systems production does. A staging assistant wired to a mock that returns anything it’s asked for will prove nothing about access control. If you can only test in production, agree the limits up front, including which actions the tester must never trigger, such as sending a message to a real customer.
Create separate test accounts for different customers, each with its own records. This is advice for any test of this kind: you need account A to ask for something that belongs to account B, and you need to know in advance what B’s data looks like so you can recognize it when it comes back. Add accounts for each role the assistant serves, such as an agent or an administrator, if the assistant behaves differently for them.
Allow multi-turn conversations. A chatbot keeps context, so a multi-turn plan is worth asking about. The insurer’s gap showed up that way.
Tell the testers what the assistant is allowed to do. If it can only read, say so; if it can create a refund or update an address, say that too, and say whether those actions can run in the test environment. Anything that can change data needs a tester who knows where the safe edges are.
What a useful report and retest look like
You’ll know within a page whether a report is going to help. Look for a finding that shows the conversation or the request that produced it, with the proof recorded so an engineer who wasn’t there can repeat it. We write our reports that way. Look for the system where the fix belongs, because “the chatbot revealed data” can turn out to need a change in an API or a permission rule rather than in the chatbot. Ask the tester to say plainly what was not tested, so nobody reads silence as a pass.
The retest deserves the same care. In general, a retest that also calls the API directly tells you whether the fix is in the API, and one that only repeats the chat may not.
Whether an auditor or regulator accepts a report as evidence is their decision. What you can control is whether the file shows what was tested, what was found and whether the fix held.
Before you ask for quotes
Make a one-page inventory: the assistant, its connected systems, the document sources behind it, the user roles, the environment you’ll test in and the launch date. That page is the core of a scope. If you’d like a second pair of eyes on it, book a free 90-minute scoping session and bring it along; a retest is included in our work. If you already know the assistant is going to need outside testing, our AI security testing page describes what we cover.