07 - classification metrics

simple metrics

suppose a model is predicting detector-failure for 1000 detector observations, wherein only 50 actually correspond to failures

	actually failure	actually normal
predicted failure	40	90
predicted normal	10	860

here true positive, $T P = 40$ , and true negative, $T N = 860$ , whereas false positive, $F P = 90$ , and false negative, $F N = 10$

accuracy

accuracy = \frac{T P + T N}{T P + T N + F P + F N}

in the above examply, it is $90 %$
it is hiding the important fact that there were 90 false alarms, and 10 missed real failures

recall/sensitivity

recall = \frac{T P}{T P + F N}

precision

precision = \frac{T P}{T P + F P}

here, it is $\sim 30.8 %$
consider an airport security system that flags almost all items, so it has $\sim 100 %$ recall but terrible precision
on the other hand, it may flag only items it is extremely confident about, so the precision will be excellent, while the recall suffers

specificity/TNR

specificity measures how many of the actual negatives were correctly identified:

specificity = \frac{T N}{T N + F P}

FPR

FPR = 1 - TNR = \frac{F P}{F P + T N}

now consider a dataset with $99.5 %$ normal measurements, but the model predicts everything as normal
here, the accuracy will be $99.5 %$ , but $T P = 0$ , hence recall $= 0$
so, accuracy can be almost useless for highly imbalanced classifications

the F1 score is the harmonic mean of precision and recall that balances these two scores:

F_{1} = 2 \frac{precision \times recall}{precision + recall} = \frac{2 T P}{2 T P + F P + F N}

note how TN does not directly appear in F1, which is useful when the negative class is extremely common
here, $F_{1} \approx 44.4 %$
the harmonic mean heavily penalises imbalance, hence that is used

a classifier often returns a probability, eg. $P (failure | x)$ , rather than a classification, so a threshold must be chosen
eg: $P \geq 0.5 ⟹$ failure, $P < 0.5 ⟹$ normal
if the threshold is lowered, more positives are retuned, increasing recall but decreasing precision
the converse is also true, giving the precision-recall tradeoff
which metric to prioritise is a domain question
if missing a positive is catastrophic, prioritise recall
if false alarms are more costly, prioritise precision

image: Evidently AI

by varying the classification threshold from 0 to 1, recall/TPR and FPR are calculated at each threshold, and TPR is plotted against FPR
a perfect classifier will be a straight line from $(0, 0) \to (0, 1) \to (1, 1)$
the area under the ROC curve (ROC AUC) summarises the ROC curve
$A U C = 1 ⟹$ excellent discrimination, $A U C = 0.5 ⟹$ random-level distribution
values below $0.5$ imply worse than random ranking

metrics can be approached using one-vs-rest reasoning
they can then be aggregated using:
- macro averaging
- micro averaging
- weighted averaging