P91Feature EngineeringAdvancedDiagnose18 min
Diagnose Training-Serving Skew in a Feature Pipeline
Diagnose an offline-to-online feature mismatch without confusing it with ordinary overfitting.
#diagnose#training-serving skew#feature pipeline
Scenario
A churn model has validation AUROC 0.84 but falls to 0.62 after deployment. Training defines `spend_30d` as purchases in the 30 days ending at each monthly snapshot. Online inference calculates calendar-month spend at 09:00 on the first day of the month and replaces missing purchases with 0; late events can arrive after the score.
Problem statement
Diagnose the likely failure and specify what must be made consistent between training and serving.
Your Task
- Compare feature definitions and timestamps.
- Identify window and missing-value mismatches.
- Distinguish skew from model overfitting.
Feature definitions
Training: events in [snapshot - 30 days, snapshot), missing = null
Serving: calendar-month events through current date, missing = 0
Score time: first day, 09:00; late events arrive up to 24 hours later
Related Concepts
Feature parityPoint-in-time dataDeployment monitoring
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.