Cross-Validation in Machine Learning: K-Fold Explained

Cross-validation estimates how well a machine learning model will perform on data it has never seen, by repeatedly training on part of the data and testing on the rest. It is the standard tool for comparing models, tuning hyperparameters, and catching overfitting before a model reaches production. The most widely used version is K-fold cross-validation.

This guide explains why a single train-test split is unreliable, how K-fold works step by step, how to choose K, and when to use stratified, repeated, leave-one-out, group, and time-series variants. You will run every technique in scikit-learn, see a data leakage experiment that turns a useless model into an apparent 81% accurate one, and learn the mistakes to avoid.

Why We Need Cross-Validation

A model that scores 100% on its own training data tells you nothing: it may have memorized the data. This is overfitting. To measure real performance you must evaluate on examples the model did not train on.

The simplest approach is the hold-out method: split the data once, for example 80% training and 20% testing. It has two problems:

• High variance. The score depends on which rows land in the test set. A lucky split looks better than the model deserves; an unlucky one looks worse. With small datasets this swing can be large.
• Wasted data. The 20% used for testing never helps the model learn.

Cross-validation fixes both. Every sample is used for testing exactly once and for training in the remaining rounds, and the final score is an average over several splits.

How K-Fold Cross-Validation Works

1. Shuffle the data (optional but usually wise) and split it into K equal parts called folds.
2. Round 1: hold out fold 1 as the validation set. Train on folds 2 to K. Record the score.
3. Round 2: hold out fold 2, train on the others, record the score.
4. Repeat until each fold has been the validation set once, giving K scores.
5. Average the K scores for the final estimate and report their standard deviation.

Each of five iterations uses a different fold as validation data and the remaining four folds for training.5-fold cross-validation on 150 samplesFold 1Fold 2Fold 3Fold 4Fold 5Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5score₁score₂score₃score₄score₅Training (120 samples)Validation (30 samples)Final = mean of 5 scores

The Formulas

CV score = (1/K) Σ_i=1..K s_i     std = √( (1/K) Σ_i (s_i − CV score)² )

• K is the number of folds.
• s_i is the score (such as accuracy) measured on fold i.
• The mean is the performance estimate; the standard deviation shows how much the score changes between folds.

Worked example. Ten samples with K = 5 give folds of 2 samples. Suppose the fold accuracies are 1.0, 0.5, 1.0, 1.0, and 0.5. The mean is (1.0 + 0.5 + 1.0 + 1.0 + 0.5) / 5 = 0.80. The deviations from the mean are +0.2, −0.3, +0.2, +0.2, −0.3; their squares sum to 0.30, so the variance is 0.06 and the standard deviation is 0.245. Report this as “0.80 ± 0.25”. The wide spread warns you that ten samples are too few to trust.

Train, Validation, and Test: Where Cross-Validation Fits

Cross-validation replaces the fixed validation set, the data you use to compare models and tune settings. It does not replace the test set. Keep a final test set locked away, and touch it once at the end. If you tune on the test set, its score becomes optimistic.

Test set held out first; cross-validation happens inside the development set.Hold out a test set, cross-validate the restFull dataset100% of the dataAfter the first splitDevelopment set (80%): model selection and tuningTest set (20%)Split into 5 folds, cross-validate hereUse once, at the very end

K-Fold Cross-Validation in Python

This tutorial uses the Iris dataset (150 flowers, 4 measurements, 3 species) with a KNN classifier.

Step 1: The fastest way, cross_val_score

from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
model = KNeighborsClassifier(n_neighbors=5)

scores = cross_val_score(model, X, y, cv=5)
print(scores.round(3))
print(f"Mean = {scores.mean():.3f}, std = {scores.std():.3f}")

Output:

[0.967 1.    0.933 0.967 1.   ]
Mean = 0.973, std = 0.025

cross_val_score clones the model for each fold, fits it on the training part, scores it on the validation part, and returns one score per fold. Passing an integer to cv uses stratified K-fold for classifiers and plain K-fold for regressors.

Step 2: Control the folds with KFold

from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=42)
for i, (train_idx, val_idx) in enumerate(kf.split(X), start=1):
    print(f"Fold {i}: train={len(train_idx)}, validation={len(val_idx)}, "
          f"first validation rows={val_idx[:5]}")

Output:

Fold 1: train=120, validation=30, first validation rows=[ 9 12 18 19 26]
Fold 2: train=120, validation=30, first validation rows=[ 0  4 10 11 15]
Fold 3: train=120, validation=30, first validation rows=[ 5  7 23 24 25]
Fold 4: train=120, validation=30, first validation rows=[2 3 6 8 13]
Fold 5: train=120, validation=30, first validation rows=[ 1 14 20 21 37]

split() yields index arrays, not data. shuffle=True randomizes the rows before splitting and random_state makes the result repeatable. Each fold validates on 30 rows and trains on 120, exactly as Figure 1 shows.

Step 3: Why shuffling matters

The Iris rows are sorted by species: rows 0 to 49 are one species, 50 to 99 another, 100 to 149 the third. Plain K-fold without shuffling cuts the data into consecutive blocks, so some folds hold almost a single class.

no_shuffle = cross_val_score(model, X, y, cv=KFold(5, shuffle=False))
shuffled   = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=42))
print("No shuffle:", no_shuffle.round(3), "mean =", no_shuffle.mean().round(3))
print("Shuffled:  ", shuffled.round(3), "mean =", shuffled.mean().round(3))

Output:

No shuffle: [1.    1.    0.833 0.933 0.8  ] mean = 0.933
Shuffled:   [1.    1.    0.967 0.933 0.967] mean = 0.973

Unshuffled folds 3 and 5 score 0.833 and 0.800, versus 0.967 for shuffled folds.Accuracy per fold, KNN on sorted Iris data1.00.90.80.7Fold 1Fold 2Fold 3Fold 4Fold 51.001.000.830.970.800.97No shuffle (mean 0.933)Shuffled (mean 0.973)Vertical axis starts at 0.7

The unshuffled estimate was too pessimistic by four percentage points, purely because of data order. If your data has any order (by date, by class, by source), shuffle, except for time series (see below).

Stratified K-Fold: Keep Class Proportions Stable

Stratified K-fold makes each fold contain roughly the same class percentages as the full dataset. It matters most with imbalanced classes. Take 100 samples where 10 are positive. Plain K-fold on data sorted with positives first puts all 10 positives into one fold, and the other four folds contain none, so their recall is undefined.

import numpy as np
from sklearn.model_selection import StratifiedKFold

y_imb = np.array([0]*90 + [1]*10)           # 10% positive class
X_imb = np.random.RandomState(0).randn(100, 3)

print("KFold positives per fold:      ",
      [int(y_imb[v].sum()) for _, v in KFold(5).split(X_imb)])
print("Stratified positives per fold: ",
      [int(y_imb[v].sum()) for _, v in StratifiedKFold(5).split(X_imb, y_imb)])

Output:

KFold positives per fold:       [0, 0, 0, 0, 10]
Stratified positives per fold:  [2, 2, 2, 2, 2]

Stratified folds keep 2 positives in every fold. Use StratifiedKFold(n_splits=5, shuffle=True, random_state=42) as your default for classification. When you pass cv=5 with a classifier, scikit-learn already uses stratification, though without shuffling.

How to Choose K

K controls a bias-variance trade-off in the estimate itself:

• Small K (such as 3): each model trains on less data (two thirds), so the score is pessimistically biased. It is fast: only 3 model fits.
• Large K (such as 20): each model trains on 95% of the data, so bias is low. But the validation folds are small, so fold scores swing more, and it costs 20 fits.

The experiment below runs stratified K-fold with logistic regression on the Breast Cancer dataset (569 samples) for several values of K.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Xb, yb = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

for k in [3, 5, 10, 20]:
    cv = StratifiedKFold(n_splits=k, shuffle=True, random_state=1)
    s = cross_val_score(pipe, Xb, yb, cv=cv)
    print(f"K={k:2d}  mean={s.mean():.4f}  std={s.std():.4f}")

Output:

K= 3  mean=0.9684  std=0.0042
K= 5  mean=0.9789  std=0.0131
K=10  mean=0.9789  std=0.0153
K=20  mean=0.9788  std=0.0236

The mean stabilizes at about 0.979 from K=5 while the fold-to-fold standard deviation grows with K.Breast Cancer, logistic regression: mean ± std across folds1.000.980.960.940.96840.97890.97890.9788K = 3K = 5K = 10K = 20std 0.0042std 0.0131std 0.0153std 0.0236Dots = mean accuracy, bars = ± one standard deviation. Vertical axis starts at 0.93.

Rules of thumb: K = 5 or K = 10 is the standard choice, and empirical studies find that both give a good balance of bias and variance. Use a larger K for small datasets (so each model trains on more data) and a smaller K when training is expensive.

Other Cross-Validation Strategies

Method	How it works	Use when
StratifiedKFold	K-fold preserving class ratios	Any classification, essential for imbalance
RepeatedStratifiedKFold	Runs K-fold several times with new shuffles	Small datasets where one K-fold run is noisy
LeaveOneOut	K = n: each sample is the validation set once	Tiny datasets (under about 100 samples), cheap models
ShuffleSplit	Random train/validation splits, repeated	Control over split size and number of repeats
GroupKFold	Keeps all rows of one group in the same fold	Several rows per patient, user, or device
TimeSeriesSplit	Trains on the past, validates on the future	Time-ordered data

from sklearn.model_selection import LeaveOneOut, RepeatedStratifiedKFold

loo = cross_val_score(model, X, y, cv=LeaveOneOut())
print("Leave-one-out:", loo.mean().round(3), f"({len(loo)} fits)")

rep = cross_val_score(model, X, y,
        cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0))
print("Repeated 5-fold x10:", rep.mean().round(3), "+/-", rep.std().round(3),
      f"({len(rep)} fits)")

Output:

Leave-one-out: 0.967 (150 fits)
Repeated 5-fold x10: 0.967 +/- 0.03 (50 fits)

Both estimates agree at 0.967. They are lower than the single shuffled run (0.973), which shows how much one lucky shuffle can flatter a result: the repeated estimate is the more trustworthy of the two.

Time-Series Data Needs a Different Split

Shuffled K-fold lets the model train on the future and predict the past, which is impossible in practice and inflates scores. TimeSeriesSplit always validates on data that comes after the training window.

Split 1 trains on 0-1 and validates on 2-3. Split 2 trains on 0-3 and validates on 4-5. Split 3 trains on 0-5 and validates on 6-7. Split 4 trains on 0-7 and validates on 8-9.Expanding window: always train on the pastt0t1t2t3t4t5t6t7t8Split 1Split 2Split 3Split 4Time steps 0 to 9, left to rightTrainValidateUnused

Cross-Validation for Hyperparameter Tuning

Tools such as GridSearchCV and RandomizedSearchCV run cross-validation for every candidate setting and pick the one with the best mean score (see the SVM tuning example and the KNN grid search). A caveat: the best score from a search is itself slightly optimistic, because you picked the maximum over many noisy estimates. For an honest estimate of the whole tuning procedure, use nested cross-validation: an outer loop evaluates performance, and an inner loop (the grid search) tunes parameters using only the outer loop’s training data.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

inner = GridSearchCV(make_pipeline(StandardScaler(), SVC()),
                     {"svc__C": [0.1, 1, 10]}, cv=3)
outer_scores = cross_val_score(inner, Xb, yb, cv=5)   # tuning happens inside each fold
print(outer_scores.mean().round(3))

Data Leakage Inside Cross-Validation

Data leakage happens when information from a validation fold influences training. The most common cause is fitting a preprocessing step (scaler, imputer, feature selector) on the whole dataset before cross-validating. The experiment below makes the damage visible. The data is pure noise: 100 samples, 2,000 random features, and random 0/1 labels. No model can beat 50% accuracy here.

from sklearn.feature_selection import SelectKBest
from sklearn.linear_model import LogisticRegression

rng = np.random.RandomState(0)
X_noise = rng.randn(100, 2000)
y_noise = rng.randint(0, 2, 100)

# WRONG: select the 20 "best" features using ALL labels, then cross-validate
X_sel = SelectKBest(k=20).fit_transform(X_noise, y_noise)
wrong = cross_val_score(LogisticRegression(), X_sel, y_noise, cv=5).mean()

# RIGHT: selection happens inside each training fold
right = cross_val_score(
    make_pipeline(SelectKBest(k=20), LogisticRegression()),
    X_noise, y_noise, cv=5).mean()

print("Leaky:   ", round(wrong, 2))
print("Correct: ", round(right, 2))

Output:

Leaky:    0.81
Correct:  0.43

Leaky selection: 0.81. Correct pipeline: 0.43. Chance level: 0.50.Random labels: what accuracy should we expect?00.51.0chance = 0.500.810.43Select features first(leaky)Select inside each fold(correct)

The leaky version looked at all labels while choosing features, so the validation folds were no longer unseen. The fix is always the same: wrap every learned preprocessing step in a Pipeline and cross-validate the pipeline, never the raw model on pre-processed data. The same rule applies to feature scaling.

Beyond Accuracy: Multiple Metrics and Predictions

from sklearn.model_selection import cross_validate

res = cross_validate(pipe, Xb, yb,
                     cv=StratifiedKFold(5, shuffle=True, random_state=1),
                     scoring=["accuracy", "f1", "roc_auc"])
for metric in ["accuracy", "f1", "roc_auc"]:
    print(f"{metric:9s} {res['test_' + metric].mean():.3f}")

Output:

accuracy  0.979
f1        0.983
roc_auc   0.995

cross_validate returns timing information and several metrics in one pass. Add return_train_score=True to compare training and validation scores; a large gap signals overfitting. cross_val_predict returns one out-of-fold prediction for every sample, which is useful for confusion matrices and error analysis, but do not average its output into a score by fold.

Common Mistakes and How to Fix Them

Scaling or selecting features before cross-validation

This is data leakage (Figure 6). Use a Pipeline.

Not shuffling ordered data

Figure 3 shows a 4-point accuracy loss on sorted data. Set shuffle=True with a fixed random_state.

Shuffling time series

This leaks the future into training. Use TimeSeriesSplit.

Splitting grouped data by row

If one patient has 10 rows, rows from that patient land in both training and validation, and the model recognizes the patient rather than the disease. Use GroupKFold with a groups array.

Ignoring class imbalance

Use StratifiedKFold and metrics such as F1, recall, or ROC AUC. Accuracy hides failure on rare classes.

Reporting only the mean

Always report the standard deviation too. “0.97 ± 0.01” and “0.97 ± 0.10” describe very different models.

Reusing the test set for tuning

Keep the final test set separate and use it once. Use nested cross-validation if you need an unbiased score for the full tuning procedure.

ValueError: n_splits=10 cannot be greater than the number of members in each class

A class has fewer samples than K. Lower K, merge rare classes, or collect more data.

ValueError: Cannot have number of splits n_splits=5 greater than the number of samples

The dataset is smaller than K. Use leave-one-out or a smaller K.

Using a very high K on expensive models

K = 20 means 20 full training runs. Use K = 5 while experimenting.

Real-World Applications

• Model selection: compare logistic regression, random forest, and gradient boosting on identical folds to choose a winner fairly.
• Hyperparameter tuning: every grid or random search relies on cross-validation.
• Healthcare: small clinical datasets use stratified, grouped (per patient) cross-validation.
• Finance and forecasting: walk-forward validation follows the same logic as TimeSeriesSplit.
• Machine learning competitions: a trusted local cross-validation score guides decisions instead of the public leaderboard.

Advantages and Disadvantages

Advantages

• More reliable and less variable than a single split.
• Uses all data for both training and validation.
• Shows stability through the spread of fold scores.
• Works with any model and any metric.

Disadvantages

• Costs K times the training time of a single split.
• Fold scores are not fully independent, because the training sets overlap, so the standard deviation underestimates the true uncertainty somewhat.
• Needs the right variant for grouped, time-ordered, or imbalanced data.
• Does not remove the need for a final untouched test set.

Related reading on BeingNeuron: Feature Scaling: Normalization vs Standardization, K-Nearest Neighbors (KNN) Algorithm Explained, Support Vector Machines: How They Work, and Gradient Boosting: How It Works.

Frequently Asked Questions

What is K-fold cross-validation?

It splits the data into K equal folds, trains K models where each uses a different fold as the validation set, and averages the K scores. Every sample is validated exactly once.

What is the best value of K?

K = 5 or K = 10 works well for most problems. Use higher K for small datasets and lower K for slow models.

What is the difference between K-fold and stratified K-fold?

Stratified K-fold keeps the class proportions in every fold. Use it for classification, especially with imbalanced data.

Should I shuffle data in K-fold cross-validation?

Yes for most data, particularly if rows are ordered. Do not shuffle time series.

Is cross-validation better than a train-test split?

For model selection and small datasets, yes: it gives a more stable estimate. You should still hold out a final test set.

What is leave-one-out cross-validation?

It is K-fold with K equal to the number of samples. Each sample is validated alone. It has low bias but is slow for large data and its estimates can have high variance.

Does cross-validation prevent overfitting?

No. It detects overfitting by measuring performance on unseen folds. You still need regularization, simpler models, or more data to reduce it.

How do I use cross-validation with a pipeline?

Pass the whole pipeline to cross_val_score. Preprocessing is then fitted inside each training fold, which prevents data leakage.

Can I use cross-validation for time series?

Yes, with TimeSeriesSplit or walk-forward validation, never with shuffled folds.

What is nested cross-validation?

An outer loop estimates performance while an inner loop tunes hyperparameters using only the outer training data. It gives an honest estimate of a tuned model.

Why is my cross-validation score higher than my test score?

Common causes are data leakage, tuning on many settings, or a test set that differs in distribution from the training data. Check that preprocessing sits inside a pipeline.

Conclusion

Cross-validation turns one uncertain test score into an average with a measured spread. The practical rules are short: hold out a test set first, cross-validate everything else, use stratified folds for classification, shuffle unless time matters, group by entity when rows are related, and keep every learned preprocessing step inside a pipeline. The leakage experiment shows the cost of breaking that last rule: 81% accuracy on pure noise.

With a reliable evaluation method in place, you can compare strong models fairly. Continue with Gradient Boosting, one of the most accurate methods for tabular data.