06 - overfitting, underfitting, and regularisation

fitting

• if a model is too simple, it cannot capurue the underlying relationship
• if it is too flexible, it starts modelling the noise and quirks of the training data rather than the underlying relationship
• a complex model, eg. a high degree polynomial, has a much lower training error, but a much higher test error, which is called overfitting
• a simple model, eg. a straight line, has relatively high training and test errors, which is called underfitting

image: P. O'Driscoll, J. Lee, B. Fu

• increasing the amount of representative training data generally helps reduce overfitting
• this is as there are many plausible explanations, making it easier to fit noise in small dataset
• but more data isn't always useful, as it may not include the representativeness of data, leading to the distribution problem, eg: training data in temperature range 20-30 degrees, but production at 40

complexity

• complexity is not always jut the degree of the polynomial

• above is a shallow and a deep tree

• a deep tree can keep making increasingly specific decisions until it essentially memorises individual training examples

• for neural networks, increasing number of layers, neurons, or parameters generally gives the model more expressive capacity

bias and variance

• a high-bias model is too constrained to capture the true relationship, roughly corresponding to underfitting

• eg: modelling a nonlinear relationship using a straight line

• a high-variance model is excessively sensitive to the specific training sample, roughly corresponding to overfitting

• small changes in the training dataset can produce a very different model

• eg: a very flexible polynomial that changes dramatically when a few noisy measurements are added or removed

• it is not simply the statistical variance, but how sensitive the learned model is to the particular training sample

• suppose the true relationship is $y=f(x)+ϵ$, where $ϵ$ represents random measurement noise or inherently unpredictable variation
• there may be a floor below which prediction error cannot realistically fall, which is called irreducible noise/error

regularisation

• suppose the ordinary loss is $L_{\text{data}}$

• instead of minimising $L_{\text{data}}$, minimise $L_{\text{total}}=L_{\text{data}}+\lambda R(\theta )$, where $L_{\text{data}}$ is how badly the model fits the observations, $R(\theta )$ is the complexity penalty, and $\lambda$ is the regularisation strength

• here, the model is not just being rewarded for fitting the data, but also being penalised for undesiarble complexity

• a common choice is L2 regularisation, where large weights become more expensive:

• L1 regularisation adds the absolute values of the weights:

• this can drive some parameters exactly to zero

• regularisation is controlled by the hyperparameter $\lambda$

• $\lambda \to 0$ implies very little penalty, and the larger it is, the more constrained the model gets

dropout

• during training, dropout randomly disables some units, reducing overfitting
• it is a regularisation technique

early stopping

• suppose a neural network is trained for many epochs and $L_{\text{train}}$ and $L_{\text{validation}}$ are tracked
• $L_{\text{train}}$ keeps falling, but after some epochs, $L_{\text{validation}}$ improves till a certain epoch, and then gets worse
• this indicates overfitting has begun, so training can be stopped around the turning point

image: AIML.com