Train, validation and test splits
Keep final evaluation outside every learning decision
Training data fit model parameters; validation data guide configuration and selection; test data assess the frozen procedure. Splits must respect the intended prediction setting, including time order, repeated people and related documents. Preprocessing, feature selection and oversampling are learned inside the training data. Percentages are planning choices, not universal requirements or safeguards against leakage.
Use a held-out test set when enough independent evaluation units are available and the target deployment population or future prediction period can be represented without leaking information into development.
Strengths
- Provides a clear final evaluation boundary
- Can mimic deployment through temporal or group separation
Limitations
- A small holdout produces unstable estimates
- Repeated test inspection turns the test set into development data
Know the boundary
Randomly shuffling correlated records does not create an independent test set.