P93Model EvaluationAdvancedInterpret16 min
Interpret Performance Drift Across Customer Cohorts
Interpret cohort-level performance changes without overclaiming a root cause.
#interpret#drift#cohorts
Scenario
A retention model's overall precision falls from 0.62 in Q1 to 0.58 in Q3. By cohort, enterprise precision is 0.64 to 0.63, while self-serve precision is 0.60 to 0.44. Self-serve share of scored accounts rises from 20% to 48%; label completion is also two weeks later in Q3.
Problem statement
Interpret the evidence and identify the next checks needed before naming the root cause.
Your Task
- Separate aggregate and cohort-level observations.
- Explain how cohort mix can affect the overall metric.
- List plausible causes and discriminating checks.
Cohort results
Q1: enterprise 0.64 (80% share), self-serve 0.60 (20% share)
Q3: enterprise 0.63 (52% share), self-serve 0.44 (48% share)
Overall precision: Q1 0.62, Q3 0.58
Related Concepts
Data driftCohort analysisMeasurement quality
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.