08 - regression metrics

for regression, there isn't a simple right/wrong but a residual/error:

e_{i} = y_{i} - {\hat{y}}_{i}

MAE

the mean absolute error (MAE) takes the absolute value of each error:

M A E + \frac{1}{n} \sum_{i = 1}^{n} | y_{i} - {\hat{y}}_{i} |

it is easily interpreted as an average deviation from the actual value

MSE

the mean squared error (MSE) takes the squares of the errors:

M S E = \frac{1}{n} \sum_{i = 1}^{n} (y_{i} - {\hat{y}}_{i})^{2}

it is less directly interpretable but punishes large errors heavily
MAE is more robust to outliers whereas MSE is more sensitive to large errors

RMSE

the root mean square error (RMSE) takes the square root of MSE:

R M S E = \sqrt{\frac{1}{n} \sum_{i} (y_{i} - {\hat{y}}_{i})^{2}}

now the units back to the original
it retains MSE's emphasis on larger errors while being easier to interpret

R-squared

$R^{2}$ measures how better a model is than simply predicting the mean:

R^{2} = 1 - \frac{\sum_{i} (y_{i} - {\hat{y}}_{i})^{2}}{\sum_{i} (y_{i} - {\bar{y}}_{i})^{2}}

a model could naively predict $\hat{y} = \bar{y}$ , so the $R^{2}$ value helps compare the model against that baseline
if $R^{2} = 1$ then the predictions are perfect
if $R^{2} = 0$ then the model is doing no better than simply predicting $\bar{y}$ for everything
if $R^{2} = - 0.5$ , it is performing worse than the mean-prediction baseline
note: $R^{2} = 0.92 ⟹ ⧸ accuracy = 92 %$

loss function vs evaluation metric

a loss function is what the algorithm optimises
an evaluation metric is what is used to judge the model performance
they can be the same, ie. training loss = MSE and evaluation loss = MSE
but they needn't be, eg. training optimises MSE, evaluation optimises MAE, RMSE, $R^{2}$