09 - training

till now, training has ben treated as a black box
training is simply a numeric optimisation of a loss function
suppose the function to be fitted is $\hat{y} = w x + b$ , where the parameters are $θ = (w, b)$
for any chosen $w, b$ , the model makes predictions ${\hat{y}}_{i}$
MSE can be used to measure how bad those predictions are:

L (w, b) = \frac{1}{N} \sum_{i = 1}^{N} [y - (w x_{i} + b)]^{2}

the loss is a function over parameter space:

(w, b) \to L

to train is to find:

(w^{*}, b^{*}) = \arg min_{w, b} L (w, b)

so it is simply an optimisation problem
consider there is a single parameter, $w$
start from a certain value of $w$ , and to find which direction to move to reach the minimum, use the gradient, $\frac{d L}{d W}$

\begin{matrix} \frac{d L}{d w} > 0 ⟹ loss increases as w increases ⟹ move w down \\ \frac{d L}{d w} < 0 ⟹ loss decreases as w increases ⟹ move w up \end{matrix}

this is the basic idea behind gradient descent:

w_{n e w} = w_{o l d} - η \frac{d L}{d w}

where, $η$ is the learning rate

for multiple parameters:

θ_{t + 1} = θ_{t} - η {\vec{\nabla}}_{θ} L

calculate loss
      ↓
calculate gradient
      ↓
move parameters downhill
      ↓
calculate new loss
      ↓
repeat

for a neural network, it may be $θ = (w_{2}, w_{2}, \dots w_{10^{6}})$
$L (θ)$ defines a very high-dimensional loss landscape
backpropagation efficiently calculates the gradients of the loss with respect to the neural-network parameters

input
  ↓
forward pass
  ↓
prediction
  ↓
loss
  ↓
backpropagation
  ↓
gradients
  ↓
optimizer
  ↓
updated weights

the learning rate controls the size of the updates
a small $η$ means tiny steps, and hence a slow convergence
conversely, a large $η$ means huge steps, and the optimiser can repeatedly jump across the minimum and might oscillate, fail to converge, or diverge
it can be set to a large-ish value in early training, and lowered later, which is called a learning-rate schedule

stochastic gradient descent

suppose there are a million observations
using batch gradient descent, ie. calculate the gradient using all observations before every parameter update, gets expensive

read 10,000,000 observations
          ↓
calculate gradient
          ↓
ONE parameter update
          ↓
read 10,000,000 observations again

the other extreme is to choose just one observation, calculate its gradient, immediately update the parameters, and then move to the another observation
this is stochastic gradient descent (SGD) in the strict sense
this is cheap per update, but individual gradients can be very noisy
the path through parameter space can be erratic

mini-batches

the practical compromise is usually taking a small group of observations and calculating the gradient from those, which is a mini-batch
this is often called SGD in everyday ML usage
eg: using a batch size of 100:

100 observations
       ↓
gradient
       ↓
update parameters

next 100
       ↓
gradient
       ↓
update parameters

next 100
       ↓
...

batch size is a hyperparameter
small batches:
- require less memory
- produce noisier gradient estimates
- give more updates per pass through the data
larger batches
- require more memory
- give more stable gradient estimates
- can exploit parallel hardware efficiently

epoch

an epoch means that the training process has gone through the entire training dataset approximately once

eg: $N = 10, 000$ and $B = 100$ , then one epoch contains $10, 000 / 100 = 100$ batches, so roughly 100 parameter updates
an iteration/set is typically one optimiser update
each epoch improves the parameters, but after a while overfitting begins, hence early stopping is needed

initialise parameters θ
         │
         ▼
      EPOCH 1
         │
         ├── batch 1
         │      ↓
         │   predict
         │      ↓
         │   calculate loss
         │      ↓
         │   calculate gradient
         │      ↓
         │   update θ
         │
         ├── batch 2
         │      ↓
         │     ...
         │
         └── final batch
                 │
                 ▼
         evaluate validation
                 │
                 ▼
              EPOCH 2
                 │
                ...

real neural-network loss landscapes can be extremely high-dimensional, with features like:
- flat regions
- saddle points
- different basins
- noisy gradients
thus, more sophisticated descent equations such as Adam, AdamW, etc are needed
it suffices for now to understand that optimiser is simply an algorithm controlling parameter updates

hyperparameter tuning

several hyper parameters have been accumulated:
- learning rate
- batch size
- number of epochs
- optimizer
- regularisation strength
- model architecture
- dropout rate
several training jobs can be run to know the best value of one hyperparameter
it is basically an experiment testing and choosing the most promising configuration from the validation performance of $λ_{learning rate}$
a grid search creates a combination grid of different parameters and tried every combination
eg: $η \in {0.1, 0.01, 0.001}$ and $B \in {32, 64, 128}$ , so a grid search tries every combination of $η, B$ , ie. 9 training runs
this is simple, but expensive for many hyperparameters
for large search spaces, trying randomly sampled configurations is more efficient, which is random search