Cross-Validation: A More Honest Model Evaluation
Learn why a single train/test split is not enough and how k-fold cross-validation gives a more reliable estimate of how your model will perform in the real world.
You'll learn
Why splitting data once is not enough and how k-fold cross-validation gives a more honest estimate of model performance.
Use this when
You are comparing models or reporting performance metrics in a paper.
The Problem with a Single Split
When you split your dataset once into training and test sets, the performance you observe depends on which cases happened to land in the test set. A lucky split can produce AUC = 0.85; an unlucky split of the same model might yield AUC = 0.72. This variability is especially severe with small datasets.
📖 k-Fold cross-validation
In k-fold CV, the dataset is divided into k equal folds. The model is trained k times, each time using k−1 folds for training and 1 fold for testing. Performance is averaged across all k test folds. This gives a more stable estimate than a single split, and every observation is used for testing exactly once.
How to Choose k
| k value | Pros | Cons | Best for |
|---|---|---|---|
| 5-fold | Fast; reasonable variance | Higher bias than 10-fold | Large datasets, quick iteration |
| 10-fold | Standard; low bias, acceptable variance | Slower than 5-fold | Most datasets — the default choice |
| Leave-one-out (n-fold) | Lowest bias | Very slow; high variance | Very small datasets (< 50 cases) |
| Stratified k-fold | Preserves class ratio in each fold | Same cost as standard k-fold | Imbalanced datasets |
Nested Cross-Validation for Hyperparameter Tuning
If you use cross-validation to both tune hyperparameters and evaluate performance, you will overfit the evaluation — the model is optimized on the same folds used to measure it. The solution is nested cross-validation: an outer loop for performance estimation and an inner loop for hyperparameter selection.
⚠️ Reporting performance from the tuning loop inflates results
Always report test performance from the outer CV loop, not from the inner loop used for model selection. This is one of the most common sources of optimistic performance estimates in published ML studies.
Practice with your own dataset
View cross-validated performance metrics for your ML model.
- 1.Upload a dataset and go to Analyze → ML Prediction
- 2.Select your target and predictor variables
- 3.VibeResearch uses stratified k-fold CV internally and reports test-set performance
- 4.Review the AUC-ROC with CI from the held-out test fold
Trusted sources behind this lesson
Further reading
Read next
Preparing Time Variables for Trend Analysis
Learn how to format, validate, and aggregate date and time variables correctly — the essential first step before any time series or trend analysis.