You run an ad experiment. Millions of people see it. The results come back, and you still can’t tell if the campaign made money. That is not a failure of execution. According to Yahoo’s own economists, it is the normal outcome.
What Yahoo Found
Yahoo ran 25 advertising experiments and found that most advertising lift tests lack the statistical power to distinguish profitable campaigns from unprofitable ones. The median confidence interval for ROI across those 25 experiments exceeded 100 percentage points. That range is wide enough to hide any meaningful signal underneath normal sales fluctuation.
The experiments reached millions of people. Scale did not fix the problem.

What to Do Instead
The research points to a structural fix: stop spreading test budgets thin across many channels. Instead, concentrate testing on the two or three channels that account for most of your actual spend, then run larger, longer, and repeated experiments on those.
Two process changes matter before any test launches:
- Define the minimum detectable effect upfront. If your test can’t reliably detect a meaningful return, you’re generating noise, not data.
- Report the confidence interval alongside the estimated lift. A single lift number without a range is a false precision that leads to bad budget decisions.
The Operator Takeaway
Treat underpowered test results as directional signals, not proof that a channel works or doesn’t. If your confidence interval is 100 percentage points wide, you have learned roughly nothing actionable. That’s the honest framing Yahoo’s research demands.
