WORKBENCH NOTE
Use this when a model appears promising. It keeps evaluation tied to a real decision, protected evidence, and known limitations rather than one flattering score.
Before modeling
- Define the target, prediction time, and the action a prediction should support.
- Choose a metric intentionally and create a protected final test set.
- Check that inputs would be available at the time of prediction.
Baseline
- Establish a simple reference prediction or model that gives the project something meaningful to beat.
- State why the baseline is appropriate for the target and decision.
Validation
- Use a split that respects time, grouped observations, or other data structure where needed.
- Keep preprocessing and feature decisions inside the validation workflow.
- Do not repeatedly use the final test set to choose features, thresholds, or models.
Comparison and threshold
- Compare models consistently and inspect variation, not only an average score.
- For classification, inspect precision-recall trade-offs and align the threshold with business cost and capacity.
- For regression, inspect residual patterns and the scale of typical error.
Error analysis
- Review false positives, false negatives, residual patterns, and subgroup performance.
- Ask where the model fails, whether the failure is costly, and whether the evidence is representative.
Final test and interpretation
- Evaluate after major choices are complete and compare the result to the baseline.
- Report limitations; feature importance and a strong score do not prove causality.
- State what would need monitoring if the model met new data.
Reproducibility
- Save the pipeline, transformations, configuration or seed, and key assumptions.
- Make it possible for another person to recreate the evaluation without guessing.