cross-validation

Training data is the part of your dataset that a model learns from. Testing data is a separate part that you hold back to check how well the model works on examples it has never seen. This difference is the core of fair model evaluation: a model that scores well on its own training data has told you very little, because it may simply have memorized the answers. This article explains training data vs testing data, adds the third piece (validation data) that many beginners miss, and shows you how to split data correctly. You will also learn about stratified splits, time-based splits, cross-validation, and data leakage, which is the most common way these splits go wrong. Table of contents Why you must split your data Training, validation, and test sets How to split a dataset Stratified splitting Time-based splitting Cross-validation Data leakage: the silent problem Python example Common mistakes Frequently asked questions Conclusion Why you must split your data Imagine a teacher who gives students a practice exam and then uses the exact same questions on the final test. Everyone would score well, but the scores would not show who understood the subject. Machine learning has the same problem. If you test a model on the examples it trained on, a high score may only prove that the model remembered them. What you actually want to know is how the model will perform on new data. Splitting the data lets you simulate that situation. The model learns from one portion, and you measure it on another portion that stayed hidden. The score on the hidden portion is your best estimate of real-world performance. This idea connects directly to overfitting, where a model learns noise in the training data instead of real patterns. You can read more in Overfitting vs Underfitting in Machine Learning. Training, validation, and test sets Most projects use three sets of data. Each one has a different job. Set Purpose When you use it Typical share Training set The model learns its parameters from these examples During training 60% to 80% Validation set You compare models and tune settings (hyperparameters) During development, many times 10% to 20% Test set You measure the final model’s performance once At the very end 10% to 20% Training data The training data teaches the model. The algorithm sees both the features and the correct labels, and it adjusts its parameters to reduce its errors on these examples. Validation data Validation data helps you make choices. Suppose you want to know whether a decision tree with depth 3 or depth 8 works better. You train both on the training set and compare them on the validation set. Because you use the validation set to make decisions, it slowly “leaks” into your choices, and its score becomes a little optimistic. Testing data The test set gives the final, honest score. You do not use it to choose models or settings. If you peek at it repeatedly and adjust your model each time, the test set stops being independent, and you lose your unbiased estimate. Some sources use the words “validation” and “test” loosely, or even swap them. When you read a tutorial, check how its author defines each term. In this series, “validation” always means the data used for tuning, and “test” always means the final check. How to split a dataset A common approach is a 60/20/20 or 70/15/15 split into training, validation, and test sets. For smaller datasets, many people use only a training and test split, such as 80/20, and use cross-validation on the training portion instead of a fixed validation set. In scikit-learn, the train_test_split function performs one split at a time. To make three sets, call it twice: first to remove the test set, then to split the remainder into training and validation sets. Always set random_state to a fixed number. This makes the split reproducible, so you and other readers get the same result each time you run the code. Shuffling is the default, and it protects you from ordered data, such as a file sorted by class. Keep the test set locked away from the start. A good habit is to create it first, before any exploration or cleaning that learns from the data. Stratified splitting A random split can produce uneven classes by chance. Imagine a dataset where only 5% of examples are fraud. A random test set of 100 rows could contain 2 fraud cases or 9. Your score would then depend heavily on luck. Stratified splitting fixes this. It keeps the same class proportions in every subset. If 5% of the full dataset is fraud, then about 5% of the training set, the validation set, and the test set will also be fraud. In scikit-learn, you pass stratify=y to train_test_split. Stratification applies to classification targets. For regression, the usual approach is a plain random split. Time-based splitting Some data has a time order: sales by day, sensor readings, or stock prices. A random split mixes the future with the past. The model would train on rows from next month and be tested on rows from last month, which never happens in real use. For time-ordered data, split by time. Train on the earliest period, validate on the next period, and test on the most recent period. Scikit-learn offers TimeSeriesSplit for cross-validation that respects order. Each fold trains on the past and tests on the period right after it. Cross-validation A single split can mislead you, especially on a small dataset. If you get an unlucky test set, your score may be too high or too low. Cross-validation reduces this risk by repeating the split several times. In k-fold cross-validation, you divide the training data into k equal parts called folds. You train k times. Each time, you hold out a different fold for validation and train on the other k minus 1 folds. You then average the k scores. Five or ten folds are common choices. You still keep a final