A flaky test is one that passes and fails on the same code, with no change in between. It is the most corrosive problem in a test suite, not because any single flake is expensive, but because flakiness teaches your team to stop trusting the tests. Once a red build "is probably just flaky," a real regression has somewhere to hide.
What flaky tests really cost
The costs are larger and quieter than most teams assume. A widely cited industrial study found flaky tests consuming around 2.5% of developers' productive time. For a team of 50 engineers, that is more than one full-time engineer's worth of work, spent re-running and babysitting tests.
Put money on it and the figures get attention. Estimates for a 20-engineer team land around six figures a year in wasted CI minutes and engineer hours, and a large end-to-end suite with a high flake rate can be far worse. Even Google, with deep testing infrastructure, has reported that roughly 16% of its tests show some flakiness. If they can't fully escape it, neither can you; the goal is to manage it, not pretend it away. Our flaky-test cost calculator turns your own numbers into an annual figure.
Why tests go flaky
Most flakiness traces back to a handful of causes:
- Timing and race conditions. Hard-coded waits and assumptions about load order. The test asserts before the app is ready.
- Shared state. One test leaves data or a session behind that changes how the next one runs. Order-dependence is a classic tell.
- Environment variability. Different CI runners, container images, or clock skew make a test behave differently run to run.
- Brittle locators. A selector tied to a CSS class or DOM path breaks on a cosmetic UI change, so the test fails even though nothing functional changed.
That last one is worth separating out, because it is the cause you can remove rather than merely manage.
How to fix flaky tests
A practical sequence that works:
- Measure the flake rate. Run your full suite around ten times against the same commit and count which tests give inconsistent results. Now you have a list, not a vibe.
- Quarantine first. Move the flaky tests out of the blocking pipeline immediately so they stop eroding trust while you fix them. Do not delete them; isolate them.
- Fix the root cause. Replace hard waits with condition-based waits, isolate test data so tests don't share state, and standardize the CI environment so runs are reproducible.
- Kill locator flakiness with stable locators. Target elements by role, accessible name, and text rather than brittle CSS paths, so a renamed class doesn't turn the suite red.
- Prevent recurrence. Track flake rate over time and treat a rising number as a real defect, not background noise.
The locator problem, solved
Two of the biggest flakiness sources, brittle locators and the maintenance they create, are exactly what self-healing test automation is built to remove. Instead of a test failing the instant a button's class changes, a self-healing runner re-resolves the same element through a cache, the DOM, the accessibility tree, and vision, and only heals when it can re-identify the element with confidence. Cosmetic change stops breaking tests; a genuine behavior change still fails, which is what you want.
BugBrain takes this further by generating coverage through exploration and running only the tests a change can affect, so the suite is both more stable and faster. You get the two things a flaky suite steals back: trust in a red build, and the hours you were spending to keep it green.
Fixing flaky tests is not glamorous work, but it is high leverage. A suite people believe in is worth far more than a bigger suite they have learned to ignore.