Cross-validation
Estimate performance of a whole learning procedure
Cross-validation rotates held-out subsets through model fitting and evaluation. Ordinary folds suit exchangeable observations; grouped and temporal variants preserve relevant dependence. Nested cross-validation places selection inside each outer training fold so outer results assess the entire tuning procedure. Fold scores are related because training sets overlap, so they are not independent experimental replications.
Choose it when data are limited and a single split would be unstable, or when comparing tuned pipelines, provided the fold design represents the intended deployment and prevents information leakage.
Strengths
- Uses available observations for both fitting and evaluation
- Nested designs account for model selection during assessment
Limitations
- Repeated model fitting can be expensive
- Naive fold-level statistical tests understate dependence
Know the boundary
A cross-validation score used to select the winner is not an unbiased final estimate for that selected model.