Classification Metrics

Classification metrics are statistical measures that compare ground-truth labels with model predictions to evaluate how well a binary classifier works. They take each observation’s actual class and predicted class or score as input, then output values such as accuracy, precision, and recall or curves such as ROC and PR.

The same model can be evaluated differently depending on which error matters more. If missing a positive is dangerous, examine recall first. If an incorrect positive decision is costly, examine precision first. Choosing a metric is therefore not merely a calculation problem. It is a decision about the costs of false positives and false negatives.

Split ten scores into positives and negatives

Suppose ten API requests receive incident-risk scores, and a score of 55 or higher is predicted to be an incident. The values below are synthetic data created to explain the concept.

data = [
    {"score": 42, "actual": "normal"},
    {"score": 48, "actual": "incident"},
    {"score": 50, "actual": "normal"},
    {"score": 53, "actual": "normal"},
    {"score": 57, "actual": "normal"},
    {"score": 58, "actual": "incident"},
    {"score": 62, "actual": "normal"},
    {"score": 65, "actual": "incident"},
    {"score": 72, "actual": "incident"},
    {"score": 80, "actual": "incident"},
]

An incident is the positive class, and a normal request is the negative class. Positive and negative do not mean good and bad. They are labels that designate the class being detected as positive.

A threshold of 55 produces four outcomes.

Actual	Predicted	Name	Count in this data
Incident	Incident	TP, true positive	4
Normal	Incident	FP, false positive	2
Incident	Normal	FN, false negative	1
Normal	Normal	TN, true negative	3

The 2×2 table containing these four counts is a confusion matrix. Libraries may place actual and predicted classes on different axes, so read the axis labels before interpreting the table.

The same four numbers answer different questions

Accuracy: how many predictions were correct overall?

$$\text{Accuracy} = \frac{TP+TN}{TP+FP+FN+TN} = \frac{4+3}{10} = 0.70$$

Accuracy is the proportion of all predictions that were correct. It can provide a quick summary when positive and negative examples are similarly common and the two errors have similar costs.

Accuracy can exaggerate performance when positives are rare. If 100 of 10,000 transactions are fraudulent, a classifier that predicts every transaction as normal has 99% accuracy while detecting no fraud. Do not choose a model using accuracy alone when the classes are highly imbalanced.

Precision: how many predicted positives were correct?

$$\text{Precision} = \frac{TP}{TP+FP} = \frac{4}{4+2} = 0.667$$

Precision is the proportion of predicted-positive items that are actually positive. It asks how much a positive prediction can be trusted. In medicine, it is also called positive predictive value, or PPV.

Precision matters when FP errors are expensive. Systems that send legitimate mail to spam or freeze legitimate accounts take consequential action on a positive decision, so they need high precision.

Recall: how many actual positives did we catch?

$$\text{Recall} = \frac{TP}{TP+FN} = \frac{4}{4+1} = 0.800$$

Recall is the proportion of actual positives identified as positive. Sensitivity and true positive rate, or TPR, are names for the same calculation.

Recall matters when FN errors are expensive. The first stage of disease screening or security-intrusion detection often needs to find a broad set of suspicious cases for further testing, so it may prioritize recall.

F1: combine precision and recall into one number

$$\text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}} {\text{Precision}+\text{Recall}} = \frac{2TP}{2TP+FP+FN} = \frac{8}{11} = 0.727$$

F1 is the harmonic mean of precision and recall. A low value on either side pulls it down strongly, so a model that raises only one of the two does not receive a high score.

F1 does not include TN in its formula and gives FP and FN equal weight. Do not conclude from F1 alone when correctly identifying negatives also matters or when the two errors have different costs. Variants such as $F_\beta$ give recall more weight, but directly incorporating real costs is clearer when that is possible.

Specificity and false positive rate: how well did we protect actual negatives?

$$\text{Specificity} = \frac{TN}{TN+FP} = \frac{3}{3+2} = 0.600$$

Specificity is the proportion of actual negatives correctly identified as negative. It is paired with sensitivity, which examines actual positives.

$$\text{FPR} = \frac{FP}{FP+TN} = 1-\text{Specificity} = 0.400$$

The false positive rate, or FPR, is the proportion of actual negatives incorrectly predicted as positive. When normal cases are very common, even a small FPR can create many errors in absolute terms. An FPR of 1% among one million normal requests per day means 10,000 false alarms each day.

Moving the threshold moves the metrics

Classification models often output a continuous score or probability for each observation. Comparing that value with a threshold produces the final positive or negative prediction.

Lowering the threshold sends more items to the positive class. FN usually falls and recall rises, while FP increases and precision falls. Raising the threshold usually moves them in the opposite direction. Precision may not change monotonically at every step in real data, but the tradeoff between the two errors remains.

Use the first tab of the explorer below to change the threshold. Thresholds 55 and 59 both produce an accuracy of 0.70. At 55, however, precision is 0.667 and recall is 0.800, while at 59 precision is 0.750 and recall is 0.600. Accuracy alone hides this difference.

Do not select the threshold that looks best on the test data and then report performance on that same test data as the final result. Use separate validation data or cross-validation to choose a threshold, then perform the final evaluation on data that was not used for training or selection.

ROC and PR curves sweep across every threshold

ROC and PR curves do not fix a single threshold. They change the threshold through its possible values and draw a performance trajectory. Both require ground-truth labels and continuous prediction scores for every observation. You cannot calculate these curves for data without ground truth.

ROC curve and AUC

The ROC, or receiver operating characteristic, curve places FPR on the x-axis and TPR, which is recall, on the y-axis. The upper-left point (0, 1) is ideal, and the expected result for a random ranking lies near the diagonal.

ROC AUC is the area under the curve. It can also be interpreted as the probability that a randomly chosen positive receives a higher score than a randomly chosen negative. It does not reveal precision at a specific threshold or the actual number of operational errors.

PR curve and average precision

The PR, or precision-recall, curve places recall on the x-axis and precision on the y-axis. When positives are rare, FP directly lowers precision, so the PR curve can expose operational weaknesses more clearly than ROC. The random baseline of a PR plot is not always 0.5. It is the positive prevalence in the evaluation data.

Average precision, or AP, summarizes the curve by weighting precision across increases in recall. Depending on the implementation, the area calculated by trapezoidal integration of a PR curve can differ from AP, so record the function name and library version in the report.

The ROC and PR tabs in the explorer let you change the threshold and class separation independently. The threshold moves the operating point along a curve. Separation changes the ordering of positive and negative scores, altering the curve itself and its AUC or AP.

Case: evaluating a variant candidate filter

Suppose a tumour-variant detection pipeline produces a score for each candidate variant. If a validated truth set exists, actual variants can be treated as positives and error signals as negatives to calculate classification metrics.

Confusion-matrix cell	Meaning in variant detection
TP	Retain a real variant as a candidate
FP	Retain a sequencing or alignment error as a candidate
FN	Miss a real variant during filtering
TN	Exclude an error signal

Relaxing a filter may preserve more real variants, but it may also admit more erroneous candidates. However, without a truth set, a larger candidate count alone cannot establish that recall increased. In that case, the analysis measures parameter sensitivity, meaning how results respond to a threshold, and should be distinguished from the classification metric called sensitivity.

In medical testing, PPV can change with prevalence even when sensitivity and specificity remain the same. Do not apply precision measured on a research dataset with artificially balanced positive and negative counts directly to the real patient population.

How is this different from false positives in statistical testing?

A classification FP is one actually negative observation predicted as positive. In statistical testing, a false positive or false discovery is a null hypothesis with no real effect that was selected as significant.

Both are incorrect positive decisions, but their denominators and control targets differ. A classifier’s FPR is FP/(FP+TN). The false discovery rate, or FDR, in multiple testing controls the long-run expected proportion of false discoveries among the selected findings. Statistical testing and multiple testing explains p-values and FDR.

What to record with evaluation results

• The definition of the positive class and the counts of positives and negatives
• How training, validation, and test data were separated
• Whether the model output is a probability or a ranking score
• The final threshold and the data used to select it
• The confusion matrix and decision-linked metrics such as precision and recall
• The library, version, and averaging method used to calculate ROC AUC or AP
• The real costs of FP and FN and the follow-up confirmation procedure

Official resources and original papers

• scikit-learn classification metrics documentation: definitions and APIs for confusion matrices, precision, recall, F1, ROC AUC, and AP
• scikit-learn Precision-Recall example: PR curves across thresholds and average precision calculations
• Davis and Goadrich, The Relationship Between Precision-Recall and ROC Curves: the relationship between ROC and PR space
• Saito and Rehmsmeier, PLOS ONE (2015): information revealed by PR curves for imbalanced binary classification