Reading a Confusion Matrix and ROC Curve

Evaluate a classifier honestly with counts of the four outcome types and the threshold-sweep ROC curve.

The four outcomes

A binary classifier's predictions sort into true positives, false positives, true negatives, and false negatives. Arranged in a 2x2 confusion matrix, these counts expose behaviour that a single accuracy number hides - crucial when classes are imbalanced.

Derived metrics

• Precision = TP/(TP+FP): of predicted positives, how many are right.
• Recall (sensitivity) = TP/(TP+FN): of actual positives, how many were caught.
• Specificity = TN/(TN+FP): of actual negatives, how many were cleared.
• F1 = harmonic mean of precision and recall.

The ROC curve

A probabilistic classifier has a threshold. Sweep it from 0 to 1 and plot true-positive rate against false-positive rate: that is the ROC curve. The area under it (AUC) summarizes ranking quality independent of any single threshold - 0.5 is random, 1.0 is perfect.

import numpy as np
y=np.array([0,0,1,1,1,0,1,0])
score=np.array([.1,.4,.35,.8,.7,.5,.9,.2])
def rates(thr):
    pred=score>=thr
    tp=((pred==1)&(y==1)).sum(); fp=((pred==1)&(y==0)).sum()
    tpr=tp/(y==1).sum(); fpr=fp/(y==0).sum()
    return round(fpr,2),round(tpr,2)
for thr in [0.3,0.5,0.75]:
    print('thr',thr,'-> (FPR,TPR)',rates(thr))

Choosing a threshold

The right operating point depends on the cost of each error. In disruption prediction a missed event (false negative) is far worse than a false alarm, so you accept a higher false-positive rate for higher recall. The ROC curve lets you see and defend that trade-off rather than accepting whatever a default 0.5 threshold gives.