First confirm the drop is real: check tracking, the data pipeline and the metric’s definition, because a broken event or late data can look exactly like a fall in demand. Then find when it started, split orders into sessions × conversion, and break both down by platform, app version, city, channel, new versus returning users and payment mode to see where it sits. Only then test causes, internal or external, against the data.
1. Is it real? tracking, pipeline, definition, time zone
2. When? sudden or gradual; which day; the same week last year
3. Where? orders = sessions × conversion; by platform, version, city, channel
4. Why? internal: release, outage, price change; external: festival, competitor sale
5. So what? size the impact, fix or monitor, tell the people affected
Ask clarifying questions first, such as how the 12% was measured and against which baseline.
Dmitriev et al. (2017), A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments, KDD
More Cases and guesstimates questionsPractise it with Prism