05 - training, validation, and testing

need for three

consider there are 10,000 detector measurements:

D = {(x, y)}_{i = 1}^{10000}

if training a model using all this, it may report an accuracy of $99.8 %$
a ridiculous model may simply store the mapping of each measurement with its label, giving $100 %$ accuracy
new data is needed to measure generalisation
the data can be split into training and testing sets
when testing, the test set is run on multiple models
the best performing one is chosen, and further customisations and selections are performed using the same test set
this risks overfitting the test set
humans are also a part of the optimisation loop
if performance of a model is measured using the test set, decisions are optimised against that
after enough iterations, the test set is no longer genuinely independent
thus, a validation set is also required
a typical split may be $70 : 15 : 15$
the training set is used to learn parameters:

θ^{*} = \arg min_{θ} L (D_{t r a i n}; θ)

the validation set is used during model development to help make modelling decisions, which might be to choose architecture, hyperparameters, regularisation, pre-processing, feature selection, or stopping point
the test set is used for the final unbiased evaluation
so the workflow becomes: build $\to$ tune $\to$ make decisions $\to$ freeze $\to$ open test set $\to$ final performance estimate
consider $\hat{y} = f (x; θ, λ)$ , where, $θ =$ parameters, and $λ =$ hyperparameters
the training data approximately determines $θ$
the validation performance helps choose $λ$
the test set evaluates the final combination

the workflow

TRAIN > learn parameters > MODEL > VALIDATION > choose hyper parameters > TRAIN AGAIN > VALIDATION AGAIN > ... > final model > TEST > report performance

validation overfitting exists too
eg: 10,000 different models/hyperparameter combinations are tried and the one with the highest validation score is selected, hence optimising against $D_{v a l i d a t i o n}$

data leakage

information getting into model training that would not legitimately be available when making real world predictions

eg: target hidden in a feature like medication_required in a diabetes prediction
eg: pre-processing before splitting
- when normalising a feature $x^{'} = (x - μ) / σ$ , $μ$ and $σ$ are calculated using train + validation + test, and then the data is split
- here, the transformation applied to the training data contains information about the distribution of the test set
- it should be: train > calculate $μ_{t}$ and $σ_{t}$ > fit scalar > individually transform train, validation and test
- generally, anything that learn something from data should be fitted only using the appropriate training partition
eg: there are 100 experiments, each generating 1,000 measurements, yielding 100,000 rows, split into 80,000 and 20,000 for training and testing
- this is a data leak as each run may have subtle unique characteristics, and the measurements from one run may be in both sets, making the test set less independent
- a better split would be experiments 1-80, 81-90 and 91-100 for train, validation and test, which is called a grouped split
eg: there are observations from 2022 to 2026
- a random split could result in observations from all years in both train and test sets
- the production should be PAST > train > predict FUTURE
- so a better split would be to do a temporal split by using 2022 to 2025 for training and 2026 for testing
a crucial limitation is that a test set does not prove universal model validity
eg: when a model is trained and tested using temperature data ranging from 15 to 25 degrees, it may not work well for 50

cross validation

cross validation is useful when there is not enough data for three sets
k-fold cross validation into $k$ equal groups and do multiple rounds of training using different groups for test and train
eg: $k = 5$ so there are folds 1 to 5
- round 1 train using 2 to 5, validate using 1, round 2 train using 1, 3 to 5, validate using 2, and so on
- there will be $s_{1}$ to $s_{5}$ , so calculate the average
- every observation gets used for both training and validation, but never simultaneously on the same fold while also giving a more robust estimate o performance
a final independent test set is still required

stratified splitting

consider data with $0.5 %$ failures
random splitting could produce $0.6 %$ and $0.1 %$ failure rates in the training and testing sets, which is undesirable
a stratified split attempts to preserve class proportions yielding approximately $0.5 %$ in both sets

important splitting patters

an ordinary random split is done when observations are reasonably independent and identically distributed
a stratified split is useful when class balance matters
a grouped split is useful when observations belong to related groups
a temporal split is useful when prediction future from past