Netflix: the night completion rate fell
A root-cause analysis of a metric that dropped without a deploy to blame.
Walks the metric drop down to a single segment and a single cause — and shows the discipline of ruling out the measurement artifact before touching the product.
The page
Completion rate is down 8% week-over-week. Go.
A senior leader pings you: title completion rate — the share of started titles watched to ~90% — fell 8% versus last week. No major release shipped. Find out why.
The amateur move is to start guessing at causes. The PM move is to first decide whether the number is even real, then narrow before theorizing. A drop is a search problem, not a brainstorm.
Step 1 — Is it real?
Rule out the measurement artifact first
- Did the metric definition or the logging change? An 8% 'drop' is often a tracking regression, not a behavior change — and it's the cheapest thing to rule out.
- Is the denominator stable? A spike in 'starts' (e.g. an autoplay change inflating starts) lowers completion rate without anyone watching less.
- Is it a data pipeline gap — a region or device whose events stopped landing? Check for suspicious zeroes before believing the average.
Step 2 — Narrow before theorizing
Segment until the drop has an address
- By time: a gradual slide vs. a cliff on one day points at very different causes.
- By platform / device / app version: a drop isolated to one client is an engineering bug, not a content story.
- By content: new releases vs. catalog, and by genre — a weak new-release slate drags the average without anything being 'broken'.
- By cohort: new vs. tenured users, and by region.
Step 3 — The likely shape
Where this kind of drop usually lives
Two endings are far more common than 'the product got worse for everyone'. One: the drop is concentrated in a single app version — a player or progress-tracking bug under-counting completions. That's a measurement story dressed as a product story, and it routes to engineering, not strategy.
Two: the drop is concentrated in new-release content — a soft slate this week means more people sampling and bouncing, mechanically lowering completion even though the product is unchanged. That's not a fire; it's seasonality the metric should be normalized against.
[PLACEHOLDER: if you want, anchor this to a specific worked example with invented-but-consistent numbers.]
Tradeoffs
What the RCA is really testing
Spend the first hour disproving the metric
Most 'urgent' metric drops are instrumentation. The PM who reflexively redesigns the product burns a sprint chasing a logging bug. Cheapest cause first.
Segment to a single address before proposing a fix
An 8% average can be a 40% drop in one cohort and nothing elsewhere. The fix for that is surgical; the fix for 'engagement' is a roadmap nobody can evaluate.