Science Education Systems · Article 98. Maya, Jia Jun, Hana and Ethan are fictional learners used to make scientific reasoning visible. This article owns one distinct scientific job: cross-validation—estimating how a modelling pipeline generalises to unseen data while keeping training information separated from evaluation information. It does not replace model selection. It is the protocol that makes model comparison honest enough to trust.
The 50-second parent route
A model can look brilliant on the data it learned from and fail on new cases. Cross-validation asks the model to earn its score repeatedly on observations kept out of training.
The route is:
dataset → unit of independence → split rule → training fold → fitted preprocessing → fitted model → held-out prediction → metric → repeat across folds → aggregate → uncertainty → nested tuning if needed → final untouched test
The fastest diagnostic is to ask: Did any information from the held-out cases influence feature selection, scaling, imputation, hyperparameter tuning or model choice? If yes, the validation may be optimistic.
This article extends How Scientific Model Selection Works, How Scientific Sampling Works, How Scientific Time-Series Analysis Works and How Scientific Validation Works.
1. Cross-validation estimates performance on unseen data
The core idea is simple: train on one part of the data, test on another part not used for fitting, then rotate which observations are held out.
2. Training fit answers the wrong question for generalisation
Training error asks how well the model fits observations it has already seen. Cross-validation asks how well the fitted procedure predicts observations it did not use to learn.
3. Maya’s first error is random splitting without asking what is independent
She splits individual rows even though many rows come from the same student. Her repair is to hold each student together so participant-specific information cannot leak between training and test folds.
4. Jia Jun’s first error is scaling the whole dataset first
He standardises variables using the mean and variance of all observations before splitting. His repair is to fit scaling only on the training fold and apply that transformation to the held-out fold.
5. Hana’s first error is tuning and evaluating on the same folds
She picks the hyperparameter with the best cross-validation score and reports that same score as final performance. Her repair is nested cross-validation or a final untouched test set.
6. Ethan’s first error is treating one split as destiny
One train-test split gives a flattering score. His repair is to inspect variation across multiple folds or repeated cross-validation.
7. The first design question is the unit of independence
Rows are not always independent units. Trials belong to participants. Images belong to patients. Measurements belong to sites. Time points belong to one process.
8. Split the unit that will be new at deployment
If the future task is predicting a new patient, hold out whole patients. If the future task is predicting a later time for the same sensor, preserve temporal order instead.
9. Ordinary k-fold cross-validation divides data into k parts
Train on k−1 folds, evaluate on the remaining fold, then repeat until each fold has served as test data once.
10. Every observation receives one out-of-fold prediction in ordinary k-fold
This lets performance be aggregated across the full dataset while each prediction comes from a model that did not train on that observation.
11. Common values of k are pragmatic, not magical
Five-fold and ten-fold cross-validation are popular compromises between computational cost, bias and variance.
12. More folds use more training data per fit
As k increases, each model trains on a larger fraction of the dataset, often reducing pessimism in the estimate.
13. More folds can increase estimate variance
Highly overlapping training sets can make fold results dependent, and leave-one-out can be noisy for unstable models.
14. Leave-one-out cross-validation uses one observation per test fold
It fits n models for n observations. It can be computationally expensive and is not always statistically superior.
15. Repeated k-fold cross-validation changes the partition several times
Repeating splits reveals how much the score depends on random fold assignment.
16. Repeated CV is especially useful for small datasets
When each observation has large influence, different partitions can produce materially different rankings.
17. Stratified k-fold preserves class proportions approximately
For classification with imbalanced labels, stratification can ensure each fold contains a more comparable class mixture.
18. Stratification does not solve grouped dependence
Balancing labels while splitting one participant’s trials across folds still leaks participant information.
19. Group k-fold keeps related observations together
All measurements from one group—patient, school, site, device or family—remain in the same fold.
20. Leave-one-group-out tests transfer to new groups
Hold out one participant or site at a time. This is demanding but appropriate when deployment requires generalisation to unseen groups.
21. The grouping variable should match the intended use
Holding out samples but not laboratories will not test transfer across laboratories.
22. Hierarchical data can require multi-level thinking
Trials within people within clinics within countries create several dependence levels. The chosen split should reflect which level is expected to be new.
23. Time-series data should rarely be randomly shuffled
Random splitting allows future patterns to help predict the past, creating leakage and unrealistic evaluation.
24. Time-aware validation preserves order
Train on earlier observations and test on later observations.
25. Rolling-origin evaluation moves the forecast origin forward
Fit on an initial history, predict the next period, expand or slide the training window, and repeat.
26. Expanding windows mimic accumulating knowledge
Training data grows through time, as in many real monitoring systems.
27. Sliding windows mimic forgetting old regimes
Only the most recent observations train the model, which can be useful under drift.
28. Temporal gaps can prevent leakage through proximity
If adjacent observations are highly autocorrelated, a gap between training and test periods can produce a more realistic separation.
29. Spatial data can require blocked cross-validation
Nearby locations are correlated. Random point-wise splitting can let training and test folds share nearly identical environmental context.
30. Spatial blocking holds out regions
This tests whether the model transfers across space rather than merely interpolating between neighbouring samples.
31. Cross-validation estimates the whole pipeline
Scaling, imputation, feature selection, dimensionality reduction and model fitting all belong inside the training fold if their parameters are learned from data.
32. Preprocessing leakage can be subtle
Using global means for imputation reveals information about the distribution of held-out cases even without their labels.
33. Feature selection leakage is especially dangerous
Choosing variables based on correlation with the outcome using all data lets test labels influence the feature set.
34. PCA can leak too
If principal components are computed on all observations before cross-validation, held-out feature structure has shaped the representation.
35. Dimensionality reduction should be refit inside each fold
The transformation learned from training data is then applied unchanged to the held-out data.
36. Hyperparameter tuning creates another selection layer
Choose regularisation strength, tree depth, number of components or learning rate based on validation performance.
37. Reporting the tuning score is optimistic
The selected hyperparameter won partly because of favourable validation noise.
38. Nested cross-validation separates inner and outer jobs
The inner loop tunes hyperparameters. The outer loop estimates performance of the entire tuning procedure on data not used for selection.
39. Inner folds belong entirely inside the outer training set
The outer test fold remains untouched until the inner search has chosen the pipeline.
40. Nested CV evaluates a procedure, not one fixed model
Each outer fold can select a different hyperparameter. The score estimates how the selection process would perform on new data.
41. The final deployed model can then be refit on all development data
After estimating generalisation honestly, train the chosen procedure on the full available development dataset before deployment.
42. A final independent test set can still be valuable
Nested CV reduces selection bias, but a truly untouched external set provides another check, especially for high-stakes models.
43. External validation is stronger than internal resampling
Data from another site, time, instrument or population test whether generalisation survives real distribution change.
44. Cross-validation does not guarantee external validity
All folds can come from the same narrow population. Internal resampling cannot create diversity that the dataset lacks.
45. The metric should match the scientific decision
Accuracy, RMSE, MAE, log loss, calibration error, AUROC, sensitivity, specificity and F1 score answer different questions.
46. Class imbalance can make accuracy misleading
A rare-event classifier can achieve high accuracy by predicting the common class always.
47. Threshold-dependent metrics need threshold selection inside training
If the classification threshold is tuned on the test fold, leakage occurs.
48. Probability metrics evaluate more than final labels
Log loss and Brier score assess probabilistic predictions, including confidence.
49. Calibration should be cross-validated honestly
Calibrators fit from data are another learned component and should not see evaluation outcomes.
50. Fold-wise metrics can be averaged
But the mean alone hides variability.
51. Report dispersion across folds or repeats
Standard deviation, quantiles or confidence intervals can show whether performance is stable.
52. Fold scores are not fully independent
Training sets overlap, so naive statistical tests on fold scores can exaggerate certainty.
53. Comparing models needs paired evaluation
Use the same folds for competing models so performance differences are measured on identical held-out cases.
54. Small score differences can be meaningless
If ranking flips across repeats, selection should acknowledge uncertainty.
55. The one-standard-error rule can reduce complexity
Choose a simpler model whose CV score lies within one estimated standard error of the best.
56. Cross-validation can tune regularisation
Try several penalty strengths and select the one that performs best on inner validation folds.
57. Cross-validation can tune dimensionality
Choose the number of PCA components, selected features or latent dimensions based on held-out performance.
58. Cross-validation can tune model architecture
Tree depth, hidden-layer size and kernel parameters can all be selected through validation.
59. More hyperparameters increase search overfitting
The wider the search, the more opportunities to exploit validation noise.
60. Search spaces should be scientifically bounded
Do not test absurd values merely because computation makes it possible.
61. Grid search explores a predefined lattice
It is simple but inefficient in high-dimensional hyperparameter spaces.
62. Random search samples combinations
It can cover influential dimensions more efficiently when only a few hyperparameters matter strongly.
63. Bayesian optimisation adapts the search
It models performance across hyperparameter space and chooses promising next trials.
64. Adaptive search increases the need for nested evaluation
Because later trials use earlier validation results, the search is even more tightly tuned to the validation data.
65. Early stopping is a tuned decision too
Choosing the training epoch based on validation performance uses the validation set as a hyperparameter-selection signal.
66. Data augmentation must respect folds
Augmented versions of one original image should not appear in both training and test sets.
67. Duplicate records create hidden leakage
Near-identical samples across folds let the model memorise rather than generalise.
68. Family relationships can leak biological information
Genetic relatives across folds may make prediction easier than truly unrelated deployment cases.
69. Site identifiers can become shortcuts
A model may learn hospital-specific artefacts correlated with labels. Holding out sites can expose this failure.
70. Device artefacts can leak acquisition conditions
Image models may classify camera, scanner or laboratory rather than the phenomenon of interest.
71. Temporal duplicates can leak future labels
Repeated records from the same event across time can place nearly identical information in both folds.
72. Leakage can be label leakage
A predictor contains information created after the outcome occurred.
73. Leakage can be target proxy leakage
A field is almost another encoding of the label.
74. Leakage can be preprocessing leakage
Global scaling, imputation or feature selection uses evaluation data indirectly.
75. Leakage can be group leakage
Related observations from the same person or site are split across folds.
76. Leakage can be temporal leakage
Future information is used to predict the past.
77. Leakage can be benchmark contamination
Training data contain test examples or near-duplicates.
78. Benchmark contamination matters in AI evaluation
High performance can reflect memorisation rather than transferable capability.
79. Worked case: student-level prediction
Each student completes ten Science tasks. A naive random row split places eight tasks from one student in training and two in test.
80. The naive score answers an easier question
It tests prediction of new tasks from students the model already knows, not new students.
81. Group CV changes the question correctly
Hold out all tasks from some students. Now the model must transfer to unseen learners.
82. Worked case: medical imaging
Several images come from each patient. Random image-level CV can put one patient in both training and test folds.
83. Patient-level CV is the correct default for new-patient use
All images from a patient stay together.
84. Worked case: ecological sites
Measurements from nearby plots are spatially correlated. Random CV exaggerates transfer because test plots resemble adjacent training plots.
85. Spatial block CV gives a harsher but more realistic test
Hold out whole regions or blocks.
86. Worked case: time-series forecasting
Electricity demand from 2020–2026 is available. Random CV mixes future and past.
87. Rolling-origin CV respects deployment
Train through one date and forecast the next period repeatedly.
88. Worked case: feature selection
One thousand genes are screened using all labels, and the top twenty are cross-validated.
89. This is leakage even though the classifier never saw the test labels directly
The feature-selection step already used them.
90. Proper CV repeats feature selection inside every training fold
Each outer or inner split can choose different genes.
91. Primary Science can learn cross-validation through “build then test”
Use some observations to create a rule, then test the rule on examples held back from rule creation.
92. Primary 3 can use one holdout case
Infer which materials float from several examples, then test on a new object.
93. Primary 4 can rotate holdouts
Use four plant observations to make a rule and the fifth to test it, then rotate which observation is held out.
94. Primary 5 can compare rule stability
If the conclusion changes depending on which example is held out, the model is fragile.
95. Primary 6 can learn group-aware validation
Several measurements from one plant are related. Hold out a whole plant rather than one leaf when testing transfer to new plants.
96. Secondary Science can formalise k-fold CV
Students can calculate fold metrics and average out-of-fold error.
97. Secondary Science can learn nested CV
Separate hyperparameter tuning from evaluation explicitly.
98. Secondary Science can learn data leakage
Ask which transformations used information from the test fold.
99. Cross-validation and validation are different
Cross-validation is one resampling method for internal validation. Scientific validation is the broader question of fitness for purpose.
100. Cross-validation and model selection are different
Cross-validation estimates held-out performance; model selection uses that and other evidence to choose among candidates.
101. Cross-validation and bootstrapping are different
Cross-validation repeatedly holds out data for prediction assessment. Bootstrap resamples with replacement and is often used for uncertainty estimation, optimism correction or stability analysis.
102. Cross-validation and train-test split are related
A single split is the simplest holdout scheme. Cross-validation rotates multiple holdouts to use data more efficiently.
103. Cross-validation and external validation are different
CV reuses one dataset internally. External validation uses genuinely separate data from another source or time.
104. Cross-validation cannot detect what the dataset never contains
If all training data come from sunny conditions, no internal fold tests performance in storms.
105. Distribution shift is outside ordinary CV unless designed in
Grouped, temporal or domain-based splits can deliberately simulate some shifts.
106. Cross-validation should mirror deployment
The split strategy should reproduce the boundary between what the model will know and what will be new.
107. Random CV is appropriate only when random future cases are plausible
If future observations are exchangeable with current rows, ordinary k-fold may be reasonable.
108. Time, group and space break exchangeability
When observations carry structure, the validation design should carry it too.
109. AI benchmark splits are validation designs
Train, development and test sets define what counts as unseen capability.
110. Reusing a public benchmark test set erodes its value
Repeated model development against test performance turns the test set into a development target.
111. Hidden test sets protect benchmark integrity
They reduce direct tuning on evaluation labels, though repeated leaderboard submissions can still leak information.
112. Leaderboard overfitting is model selection against a benchmark
Teams iterate on small score differences until the public test set becomes part of development.
113. Private final evaluation can expose leaderboard overfitting
A second unseen test set provides a more honest check.
114. AI can help learners design validation
Useful prompts include: “Give me a dataset with participant leakage,” “Design a time-aware split,” “Show nested versus non-nested CV,” and “Find preprocessing steps that should live inside the pipeline.”
115. AI can produce leakage while appearing rigorous
Generated code may scale the full dataset before splitting or choose features globally. The workflow must be audited, not trusted because it uses familiar functions.
116. Parents can understand cross-validation through unseen work
A child can look strong on practised questions. The honest test is performance on comparable questions not used during teaching.
117. Tuition should protect true holdout tasks
Do not coach the exact diagnostic question and then count the retest as independent evidence of transfer.
118. The independence test is a cross-validation idea
Train using some examples. Measure performance on structurally comparable but unseen examples without tutor hints.
119. Repeated unseen tasks estimate stability
One successful transfer could be luck. Several independent held-out tasks provide stronger evidence.
120. Independent-attempt task 1: find the leakage
Given a pipeline—impute all data, scale all data, split, train—identify what must move inside the training fold.
121. Independent-attempt task 2: choose the split unit
For trials within patients, plots within sites and repeated readings from sensors, decide what should be held out.
122. Independent-attempt task 3: nested CV diagram
Draw outer training/test folds and inner tuning folds. Label which scores may be used for hyperparameter choice and which for final estimation.
123. Independent-attempt task 4: time-aware evaluation
Create monthly data and design an expanding-window forecast validation.
124. Independent-attempt task 5: repeated CV
Repeat five-fold CV several times. Compare mean score and ranking stability across models.
125. Diagnostic error: random split by default
Repair by identifying the deployment unit and dependence structure first.
126. Diagnostic error: preprocessing before CV
Repair by fitting every learned transformation inside training folds.
127. Diagnostic error: tuning score reported as final score
Repair with nested CV or an untouched final test set.
128. Diagnostic error: one lucky split
Repair with k-fold or repeated resampling where appropriate.
129. Diagnostic error: future-to-past leakage
Repair with chronological splits.
130. Diagnostic error: related observations split apart
Repair with group-aware folds.
131. Diagnostic error: benchmark reused indefinitely
Repair with hidden or fresh evaluation data.
132. Diagnostic error: average score without uncertainty
Repair by reporting fold or repeat variation and selection stability.
133. The evidence boundary
Cross-validation estimates generalisation only to data resembling the held-out units created by the split design. It cannot justify transfer to populations, times or environments absent from the dataset.
134. A compact cross-validation checklist
- What future case should the model generalise to?
- What is the true independent unit?
- Should splitting be random, stratified, grouped, spatial or temporal?
- Are duplicates or related observations kept together?
- Is preprocessing fitted only on training folds?
- Is imputation inside the pipeline?
- Is feature selection inside the pipeline?
- Is dimensionality reduction inside the pipeline?
- Are hyperparameters tuned inside an inner loop?
- Is final evaluation separated from tuning?
- Does the metric match the scientific decision?
- Is class imbalance handled appropriately?
- How variable are scores across folds or repeats?
- Could the dataset contain temporal, spatial or group leakage?
- Does the split mimic deployment?
- Is external validation still needed?
135. Frequently asked questions
What is cross-validation?
It is a resampling procedure that repeatedly trains a model on part of the data and evaluates it on held-out observations to estimate generalisation performance.
What is k-fold cross-validation?
The data are divided into k folds; each fold is held out once while the others are used for training.
What is nested cross-validation?
It uses an inner loop for tuning and an outer loop for unbiased evaluation of the whole model-selection procedure.
What is data leakage?
Leakage occurs when information unavailable at real prediction time influences training or model selection, making evaluation overly optimistic.
When should group cross-validation be used?
When several rows come from the same underlying unit and future deployment requires prediction on new units.
How does cross-validation help PSLE Science?
The formal machinery is advanced, but the habit is familiar: test a rule on examples that were not used to create or practise the rule.
How does it deepen in Secondary Science?
Students can reason about holdouts, repeated validation, grouping, time order, leakage and nested tuning more formally.
136. Continue the Science Education Systems series
- How Scientific Model Selection Works
- How Scientific Regularisation Works
- How Scientific Dimensionality Reduction Works
Conclusion: A model earns trust on data it was not allowed to learn from
Maya makes the split.
Jia Jun keeps preprocessing inside the fold.
Hana separates tuning from evaluation.
Ethan asks whether the split truly resembles deployment.
Science needs all four.
Protect the holdout.
respect groups.
respect time.
block leakage.
repeat enough to see instability.
Then let unseen data—not training applause—decide whether the model generalises.
