Website Security Regression Checker Guide: Prove Old Fixes Stay Fixed

Fixing a security problem is only half the job. The other half is making sure it stays fixed. Websites change constantly: templates are edited, plugins are updated, server configurations are rewritten, and pages are redesigned. Any of those changes can quietly undo a previous fix, reintroducing a missing header, a lax cookie setting, or an unprotected form. The reintroduced flaw is called a regression, and it is particularly dangerous because everyone believes it was already dealt with, so nobody is looking for it.

The Website Security Regression Checker tool compares a site's current security state against a previous baseline of fixed findings, and flags anything that has reappeared. It checks the items you tell it to watch, against the results you recorded before; it does not discover entirely new classes of problems, which is the job of a full audit. That division of labour is the point: a full audit finds new problems, while a regression check proves old problems have not returned. Both are needed, and neither replaces the other.

A regression check needs a baseline, and the baseline needs to be specific. A note saying security was fixed in August is not a baseline. A list saying which findings were fixed, on which pages, with what evidence, is a baseline, because each entry can be re-tested and the result compared. The baseline is usually the fix list from the last audit report, with the evidence recorded at the time of fixing. When the regression check runs, each baseline item gets one of three verdicts: still fixed, regressed, or untestable because the page changed so much the original test no longer applies. The third verdict is important, because a page that changed beyond recognition needs a fresh audit of its own, not a shrug.

What to know before you start

Run the check after any change that could affect security: a template update, a server configuration change, a plugin or framework upgrade, a redesign, or a migration to a new host. The check is cheap compared with a full audit, which is why it can run often, and often is the point: a regression caught the day it is introduced is a small fix, while the same regression caught six months later has been exposed for six months and may have been noticed by others. Tie the check to the release process, so a release cannot go out with an unexamined regression check, in the same way it cannot go out with failing tests.

Work through the results in the order the baseline lists them, and for each regressed item, record three things: what changed, when it changed, and what will restore the fix. The cause is usually mundane: a new template that omitted a header the old template set, a configuration file overwritten during a migration, a plugin update that reset a setting to its default. Finding the cause matters because it tells you whether the regression is a one-off slip or a process gap that will recur. A template that lacks a header will lose that header on every page it renders, not just the one you noticed, so the fix belongs in the template, not on the single page.

Separate regressions from new findings when you report the result. A regression check that returns five items, of which three are regressions and two are new observations, should say exactly that. Mixing them hides the most important fact in the report: that three things previously fixed have come back. Report regressions first, with their original fix dates, because a regression is also a process signal. One regression is an accident; repeated regressions of the same kind are evidence that the fix was never built into the process that produces the site, and that is the deeper problem to solve.

A worked example you can follow

Take a sample baseline of 12 fixed findings from a previous audit: 6 headers, 3 cookie flags, 2 form protections, and 1 redirect rule. After a redesign, the regression check reports 10 still fixed and 2 regressed: one header and one cookie flag. The regression rate is 2 divided by 12, or about 17 percent, which is high enough to investigate the redesign process rather than merely re-applying two fixes. Suppose investigation shows the redesign introduced a new template for article pages that sets neither the missing header nor the cookie flag, while the old template set both. The article pages number 40 of the site's 60 pages, so the regression affects two-thirds of the site, not just the two sample pages the check examined.

The fix follows the cause. Update the new template to set the header and the cookie flag, then re-run the regression check across all article pages, not just the samples. If the check now reports 12 of 12 still fixed, the regression is closed and the template is corrected for future pages too. If it reports 11 of 12, one item remains, and the remaining item gets the same treatment. Count the result explicitly each time, because a regression check that reports a vague improvement instead of a count leaves room for a remaining regression to hide.

Consider a second sample: a baseline of 12 findings, a check reporting 12 still fixed, but 3 items marked untestable because their pages were removed in the redesign. The correct verdict is not 12 of 12 clean; it is 9 confirmed fixed, 3 untestable, and a question about whether the removed pages' functionality moved somewhere that needs its own check. Untestable is a verdict that demands follow-up, not a pass. Record it as such, and schedule the follow-up check for the replacement pages.

Common mistakes to avoid

The first mistake is running regression checks without a baseline, which turns them into unfocused mini-audits that find whatever they happen to notice. The second mistake is checking only the pages that changed, while a template or configuration change can affect pages that were never edited. Check the baseline items wherever they apply, not only where the change was made. The third mistake is re-applying a regressed fix on a single page when the cause is a template or configuration, which guarantees the regression will return with the next page built from the same template.

The fourth mistake is treating a clean regression check as a clean bill of health. A regression check only proves that old problems have not returned; it says nothing about new problems introduced by the change. Pair regression checks with periodic full audits, and run a full audit after major changes like a redesign or a platform migration, when new problems are most likely. The two checks answer different questions, and confusing them is how a new flaw hides behind a clean regression report.

Checking the result before you finish

A complete regression check ends with a dated report: the baseline used, the verdict for each item, the cause of each regression, the fix applied, and the re-check result. Keep it beside the audit report it follows, so the history of each finding, found, fixed, regressed, fixed again, is visible in one place. That history is also the evidence that turns a repeated regression into a process change: a header that regresses twice is a header that should be set by the server configuration or enforced by an automated test, not by a human remembering to set it.

For companion checks, the Website Security Audit tool produces the baseline a regression check needs, the Website Security Audit Report Builder tool structures the reports both checks produce, and the HTTP Security Headers Guide explains the headers most often regressed. Do not paste secrets into any checker: regression checks need observable behaviour, never credentials or tokens. A fix that stays fixed is worth more than a fix that must be rediscovered, and the regression check is how you prove the difference.