H1 - model optimisation

quantisation

• if a model consists of enormous numbers of numerical parameters, it can be expensive

• the numerical precision used to represent them therefore matters

• a traditional representation is 32-bit floating point (FP32)

• if a model had one billion parameters stored at 32 bits each, ${10}^9\times 4\text{ bytes}\approx 4\text{ GB}$

• lower precision representations can be used

• it is useful for:

  • smaller model footprint
  • faster inference
  • lower serving cost

• the trade-off is deterioration in the numerical precision

• post-training quantisation is when the model is first trained, then converted/represented at lower precision for deployment

• quantisation-aware training (QAT) is when training accounts for the effects of lower-precision inference

other optimisation concepts