P92Machine LearningIntermediateDesign14 min
Design a Model Selection Rule Across Validation Folds
Design a model-selection rule using average performance, fold variability, and complexity.
#design#cross-validation#model selection
Scenario
Five-fold validation results for a classification task are: Model A, regularized logistic regression: F1 0.74, 0.75, 0.73, 0.75, 0.74. Model B, boosted trees: 0.78, 0.69, 0.80, 0.70, 0.79. Model C, a large ensemble: 0.75, 0.76, 0.74, 0.75, 0.76, with substantially higher serving cost.
Problem statement
Design a defensible selection rule and apply it to this evidence.
Your Task
- Compare average and fold-to-fold variation.
- Consider complexity and meaningful differences.
- State what additional check would support the final choice.
Related Concepts
Cross-validationModel complexityStability
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.