AI writes your tests faster. It also breaks them faster.
Your AI assistant can write a 200-line Playwright test before lunch. That is not the problem. The problem is what happens when someone renames a CSS class at 2pm.
Coverage charts look healthier than ever. But the failure rate is scaling faster than the code. CodeRabbit analyzed 470 pull requests and found AI-co-authored code contains 1.7 times as many issues overall as human-written code, with logic and correctness errors 75% more common. Meanwhile, a Lightrun report found developers now spend an average of 38% of their week on debugging, verification, and troubleshooting. That is roughly two full working days gone.
Asad Khan, Co-Founder and CEO of TestMu AI, argues the instinct to blame AI coding tools misses the real issue. The architecture underneath the tests is what’s failing under velocity.
Why AI Accelerated the Brittleness Problem
Brittle tests are not new. They have been a QA problem for 15 years. What changed in 2026 is the speed at which teams accumulate them. Organizations that adopted AI-assisted test authoring went from zero coverage to thousands of tests in weeks.
That scale creates a fragility trap. A single UI refactor can break hundreds of tests. Engineers spend entire sprints rewriting selectors instead of shipping. The culprit is not the AI writing the tests. It is the selector-based architecture underneath them.
Here is what brittle looks like in practice: AI generates a test that finds a button by its ID. Someone redesigns the page and the ID changes. The button still works for real users. The test can not find it. Failure. On the flip side, a button hidden behind a pop-up or invisible in dark mode passes the test while real users can not interact with it. Your dashboard stays green. Customers leave.
Khan calls this the false-positive crisis. Most testing failures are not actual bugs. They are flakiness challenges: tests failing inconsistently because of timing issues, environment drift, and brittle UI selectors.
️ The Four-Layer Architecture
Fixing this means moving beyond selector-based architecture entirely. Khan lays out four layers that need to work together.
Layer 1: Intent-Based Testing
Instead of confirming specific elements, tests describe user intent. An agentic AI reads the instruction in plain language, for example “Fill in the email field and click submit,” then finds the right elements regardless of whether they have been renamed, moved, or restyled. The focus shifts from selectors to behavior.
Layer 2: Visual-Truth Validation
Tests run against what users actually see, not what the DOM reports. If a button is hidden behind an overlay, the test fails. If dark mode breaks contrast, the test catches it. This layer directly addresses the false-positive problem that DOM-based testing has produced for over a decade.
Layer 3: Self-Healing With Context
When the team migrates to a new design system, semantic understanding re-resolves the original intent against the updated DOM. The test does not break. It adapts based on contextual awareness of what changed and why.
Layer 4: Deterministic Execution
Generative AI is nondeterministic: the same input can produce a different output. That is acceptable during authoring. It is fatal during test execution, where you need the same test to produce the same result every single time. The fix is to use AI during authoring and analysis, then keep execution fully deterministic.
⚠️ The Trap Most Teams Fall Into
Teams usually adopt one layer, most often self-healing, and assume the problem is solved. It is not. Self-healing on top of a selector-based architecture just delays the failure. The maintenance burden gets papered over until the next refactor exposes it again.
Khan identifies three principles for teams trying to escape the cycle:
- Stop treating tests as code. Tests should describe behavior, not selectors. If your tests need updating every time a developer touches the UI, they are coupled to the wrong abstraction.
- Test what the user sees, not what the codebase says. DOM-based tests will keep producing false positives at scale. Visual-truth tests catch the bugs that actually affect users.
- Pick architecture over features. Self-healing, AI authoring, and visual validation only work if they are integrated. Adopting them as separate tools just moves the maintenance problem to a different layer.
When This Works
Teams that have adopted all four layers together have cut regression cycles from 10 hours to under one hour, according to Khan’s account of working with engineering organizations on this architecture.
The diagnostic question for this quarter: Is your quality infrastructure keeping pace with your development velocity? If a single sprint of UI refactoring can break hundreds of tests, the answer is no, and the fix is architectural, not a better tool.
When It Does Not
Adopting one or two layers in isolation will not close the gap. The four layers are designed to work as a system. Visual-truth validation without deterministic execution still produces unreliable results. Intent-based testing without self-healing still breaks when design systems change. The architecture requires all four working together before the brittleness cycle actually stops.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
