Automated retesting case study: 90 findings, 30 days and fixes that did not hold
When a developer says a vulnerability is fixed, most organizations take their word for it. This insurance and fintech group could not. Its external auditor would not accept a spreadsheet of self-reported fixes, and the audit was 30 days away.
Same attack, same evidence format, before and after kept side by side
Thirty days to prove 90 fixes
A penetration test had left the group with 90 findings. Developers were working through them and announcing fixes in chat messages. The auditor wanted independent proof that each weakness was really gone, and a list of “done” messages was not going to satisfy them.
How the retests worked
The idea is simple. When a finding is marked fixed, we run exactly the same attack again, in the same way and in the same environment, and record what happens. If the attack still works, the finding reopens.
We connected retesting to the client’s ticketing and code systems. When a ticket moved to “done”, or a developer tagged a code change, the matching retest ran automatically. Each finding got a before-and-after pair of evidence: the same payload, the same tool and the same environment, with screenshots and logs side by side.
Every result received one of four verdicts: closed, partly fixed, reopened, or covered only by a compensating control. A typical compensating control is a web application firewall (WAF, a filter that sits in front of an application). We always flagged that last verdict openly, because a filter can hide a flaw without removing it.
What the first retest showed
Fewer than half of the claimed fixes held.
About a third were only partly fixed. In many of those, the firewall blocked our exact test, but the application behind it was still vulnerable. Roughly a quarter had reopened, because the next deployment had undone the fix.
The individual cases were familiar to any tester. One cross-site scripting (XSS) filter blocked <script> but let <ScRiPt> through. An access-control flaw was fixed in the web application but not in the mobile API that did the same job. Another fix blocked an attack through one Envoy proxy header, while the same bug could still be reached through a different header. On 7 findings, staging had been fixed but production was still vulnerable.
None of this was dishonesty. The developers believed their fixes worked. They simply had no quick way to check.
The second round
The developers fixed what remained. The second retest confirmed every critical and high finding as closed, each with paired evidence, and the auditor accepted the evidence pack.
The retest cases stayed in place after the audit. They now run as permanent checks, and they have since caught 2 regressions automatically. Developer habits changed too. Once every “done” ticket was retested automatically, fewer tickets were closed too early.
Organizations heading into an audit with a long list of reported fixes and no independent proof are in the same spot this group was a month before its deadline.