Write characterization tests before AI patches legacy code
In September 2026, Microsoft shipped a security update for Excel. There was no announcement that paste was going to stop working. There was no warning that AutoFill would break. It just happened, silently, across Excel 2016, Office 2019, Office 2021 LTSC, Office 2024 LTSC, and Microsoft 365 Apps.
The follow-up fix, KB5002665, eventually confirmed what users had already figured out from a forum thread that ran more than a hundred replies deep: workbooks containing conditional formatting caused paste operations to keep failing. Cells stuck mid-selection. Borders kept blinking. The Escape key stopped responding.
Nobody on that patch team decided to skip testing. There was no test to skip. The conditional formatting code path had never had one. A security hotfix is not the moment anyone stops to write the first test a decades-old module has ever seen.
This is not a Microsoft problem. This is what happens when an AI generates a diff in the time it takes you to read a CVE, and the code it’s touching has never been verified by anything except years of nobody complaining.
Why untested legacy code is a different kind of risk now
Michael Feathers defined legacy code in Working Effectively with Legacy Code as code without tests. Not old code. Not ugly code. Code you can’t safely change because nothing tells you when you broke it.
For most of software history, writing a patch was the slow part. Manual testing could roughly keep pace with human-speed edits. That’s no longer true. An AI assistant can draft a fix for a fifteen-year-old code path in the time it takes to read the CVE. The pace of change now outruns the pace of verification unless the test suite is automated and runs on every change.
A codebase with no seams makes this worse. When there’s no abstraction layer between the security fix and the surrounding logic, a narrow change to close one hole can reach into unrelated behavior. That’s how a clipboard regression ends up in a security patch. The fix touched the untested core, and the untested core touched everything.
️ How to patch legacy code safely with AI
Step 1: Classify the module as legacy before you touch it
If the module has no tests, treat it as legacy code under Feathers’ definition, regardless of when it was last edited. Stability without complaints is not the same as verified behavior. The Excel conditional formatting module had been shipping without issues for years. That was not a safety net.
Step 2: Write a characterization test before the AI changes a single line
A characterization test records what the module currently does, including the parts you find confusing or ugly. It doesn’t ask whether that behavior is correct. It asks whether the patch changed anything it wasn’t supposed to touch. That’s the more urgent question during an emergency fix.
Ask the AI to draft the test scaffolding for the exact module you’re about to patch. Then review every assertion yourself before committing. The AI drafts based on function names and signatures. You confirm that the assertions describe the behavior you actually want preserved, not the behavior the model assumed.
Step 3: Find the seam and route the fix through it
A seam, in Feathers’ terms, is a place where you can insert a change without editing the original untested code path. Find that point and route the AI’s patch through it. The fix should touch only the behavior the CVE is about. Nothing else.
Scope the change to the smallest edit that closes the actual security hole. Don’t authorize the AI to clean up nearby legacy code in the same commit. A security patch authorizes closing the hole and nothing else.
Step 4: Don’t let the AI change functional behavior without asking you first
If the smallest fix genuinely can’t avoid changing what the feature does, that decision belongs to a human, not to the model that drafted the diff. Make it stop and ask before it proceeds. This boundary matters most precisely when you’re under deadline pressure and least likely to enforce it.
Step 5: Run the full regression suite before and after
Run every regression test and any acceptance tests tied to the feature before and after the patch. Reject any AI-generated fix that doesn’t leave every existing test passing. When the AI reports that tests pass, run the suite yourself. Don’t accept the summary as proof.
Step 6: Add the new characterization tests to the pipeline permanently
Wire the new tests into CI so the same regression can never again depend on a user filing a forum post. A green dashboard built on vanity coverage numbers is not a safety net. Vanity coverage convinces a team it’s safe to ship fast, while the actual paths that break in production were never exercised by any test.
Step 7: Canary the rollout when the vulnerability isn’t already public
Ship to a small canary segment before a full rollout. A blind spot in decades-old code surfaces on a fraction of your users instead of all of them at once. Pair the canary with active monitoring. A staged rollout only limits damage if someone is watching it.
When the vulnerability is already public or actively exploited, skip the canary and ship to everyone. Leaving any segment unpatched in that situation hands those users to whoever is exploiting the hole. Lean on the characterization tests instead of a staged rollout to catch a regression.
What good AI prompting looks like here
The difference between a safe patch and a broken release often starts with the prompt. Here is the contrast the source author draws directly:
Bad prompt: Fix the security hole in the conditional formatting module and ship it today. This code has never had a test. It has worked fine for years without one. Ship it the same way.
Good prompt: Write characterization tests for the conditional formatting module. It never had tests. You must capture its real clipboard and AutoFill behavior. Show me the failing diff only after every existing and new test passes. I will also validate it manually. Scope the fix to the smallest change that closes the security hole. You aren’t authorized to change functional behavior. If you must, stop and ask me first.
The good prompt sets explicit exit criteria and draws a hard boundary around functional behavior. The bad prompt ships on faith.
⚠️ Common pitfalls
- Characterization tests preserve defects too. A bug baked into the legacy code gets immortalized right alongside everything that works. The test proves the patch didn’t change behavior, not that the behavior was correct to begin with.
- Retrofitting tests takes longer than the patch itself. That’s exactly why teams skip it under deadline pressure. Budget for it before the next CVE, not during one. Nobody writes careful tests while everyone is shouting in the incident channel.
- Rolling back restores the security hole. If the patch breaks production and you roll back, you’ve put the vulnerability back. A characterization test suite is what lets you ship a corrected patch fast instead of choosing between a broken feature and an open hole.
- AI-reported test results need independent verification. Run the suite yourself. Don’t treat a summary from the model as proof that the tests pass.
Why SQLite is the counter-example worth studying
SQLite has been running since August 2000. It sits inside an estimated one trillion database files worldwide and is likely the most widely deployed software library on the planet after zlib. Its own testing page states the project carries roughly 590 times as much test code as product code, with fuzzing that runs about a billion mutations a day, out-of-memory and I/O fault injection, crash recovery checks, and claimed 100% branch and MC/DC coverage on the core engine, verified under Valgrind and multiple sanitizers before every release.
Twenty-five years old and drowning in tests is a choice a team made and kept making. Twenty-five years old and never tested once is a different choice, made by omission and repeated every time nobody went back to fix it. The KB5002914 incident didn’t need a smarter model to avoid. It needed a characterization test on the conditional formatting code path, run automatically before the patch shipped to hundreds of millions of installs.
Users tolerate slow fixes. They don’t tolerate a security update that quietly breaks a feature they’ve used every day for years. A quick fix that breaks a feature everyone relies on isn’t quick. It’s a slower disaster with better timing.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
