AI Security Testing for Chatbots and LLM Apps: What the Assistant Can Reach, Not Just What It Says
Picture a team about to launch a chatbot that can look up a customer’s policy. The model has been reviewed, the prompts have been tried, and the launch date is public. What nobody on the team can answer yet is what happens when a customer asks it about somebody else’s policy. This page is for that team, or for anyone putting an LLM app, a chatbot or a RAG system in front of customers or staff.
A conversation that ended in someone else’s policy
A large insurer asked us to test its customer-facing chatbot and the APIs behind it before go-live. The window before the launch date was tight, and the testing ran in staging.
One of our testers logged in as a customer and, in an ordinary conversation, mentioned another customer’s policy number. The chatbot answered with that customer’s details. Behind the chatbot, the API had never checked that the number belonged to the person asking, so it simply returned the record.
You wouldn’t have seen it in a single request. It showed only after several turns of normal-looking conversation, and nothing threw an error. The chatbot was polite about it. A scanner has nothing to flag in that, and neither does a quick test.
The insurer postponed the launch until the fix was in and retested. The fix was an ownership check in the API: the record being requested has to belong to the customer who is logged in. When we retested, we repeated the original conversation with another customer’s number, and then skipped the chatbot and called the API directly with an ID from a different account. Both were refused. From finding to passed retest took a few days.
Afterward the insurer held the test report and the retest result, which showed the gap was closed before launch.
What we’d tell anyone in the same position: a chatbot is only as safe as the access checks in the systems it calls. Test whether it hands one customer’s data to another, and not only whether the prompt can be tricked.
What we test
We test the application around the model: the LLM app, the chatbot or the RAG system, and the APIs it calls. The place we start is reach. Which records can the assistant pull up for a user, and through which systems? For each of those we check that the system behind it confirms who is asking. That is the check the insurer’s API was missing.
You’ll know some of the vocabulary from OWASP’s list of LLM risks, which includes prompt injection and sensitive information disclosure. We prove each finding before we list it, and we record the proof so your engineers can repeat it.
What you get back
The insurer’s finding was a sequence of messages rather than a single request, so each finding in our report comes with the conversation that proves it and the detail needed to close the gap. The findings are mapped to the frameworks your auditor works from, such as ISO/IEC 27001:2022 and SOC 2, and to PCI DSS 4.0 where the assistant touches cardholder data. Whether an auditor accepts a report is the auditor’s call. For how an engagement runs from the first call to the report, see the penetration testing page.
Beyond the application
Testing the model itself, or agents that call tools on their own, is a different conversation. If that’s what keeps you up, bring it to the scoping session and we’ll tell you straight what we’d do and what we wouldn’t.
Two neighbors are easy to confuse with this service. A red team assessment is a goal-driven simulated attack on a whole organization over time, and AI systems aren’t its scope. And if the worry is your own staff pasting customer data into public AI tools, that’s what Nesgate is for: it masks sensitive data in ChatGPT, Gemini and Claude prompts, in the browser. It’s in early access with a 14-day trial.
Questions about AI security testing
How is this different from a normal penetration test? A web or API test follows screens and requests. An assistant adds a conversation on top. In the insurer’s case the weakness showed only several messages into a conversation, so a quick test would have missed it. We test the assistant and the systems underneath it together.
Do you test the model itself? The application around the model is our scope: LLM apps, chatbots, RAG systems and the APIs behind them. Questions about the model are for the scoping session.
Can we test before launch, in staging? Yes. The insurer’s test ran in staging ahead of go-live. Where we test is something we agree with you at the start.
Start with a scoping session
Tell us what the assistant can see and do, and when it has to go live. An NDA is available before you share any details. You’ll have a scoped proposal within 24 hours, and the report follows within 10 business days of testing, with a retest included. If your assistant sits in insurance or in technology and SaaS, those pages show the sectors we work in.
Book a free 90-minute scoping session · Request a scoped proposal