Statistical Testing and Multiple Testing

Statistical testing is a procedure for deciding whether an observed difference reflects an effect of the condition, or could arise from variation across repeated measurements alone. It considers both the size of a difference and whether that difference appears consistently.

Same difference, different confidence

Suppose a single test of a new server configuration reduces response time from 200 ms to 100 ms. The values differ by a factor of two, but you cannot tell whether that one request happened to be unusually fast.

If repeated measurements cluster at 190–210 ms for the old configuration and 90–110 ms for the new one, the difference is larger than the variation across measurements. If both configurations instead span 50–300 ms, confidence is lower even if the difference between their means stays the same.

Repeated measurements are needed to estimate this variation, or variance. How many repetitions are enough depends on the effect size, variability, measurement cost, and intended test.

What a p-value says

A p-value is the probability, under a model in which there is no real difference between conditions, of obtaining a result at least as extreme as the one observed.

A p-value is therefore not any of the following:

• The probability that the observed result occurred by chance
• The probability that there is no difference between conditions
• The probability that the study hypothesis is wrong

A small p-value does not require a large difference. With a very large number of measurements, even a small difference can be detected consistently. Conversely, an important difference can have a large p-value when the dataset is small or highly variable.

Consider effect size and uncertainty together

To assess a test result, consider not just the p-value but also the effect size, which describes how large the difference is. A reduction in response time from 200 ms to 199 ms may be highly statistically significant yet irrelevant in production.

When possible, also examine a confidence interval for the effect size. A point estimate shows the most plausible size, while the confidence interval shows the range of uncertainty supported by the current data.

Testing many items at once

When one metric with no real difference is tested at p < 0.05, about 5% of experiments can pass the threshold by chance in the long run. Testing 20,000 metrics in the same way produces many false positives. Here, a false positive is not one classifier misclassification. It is a hypothesis with no real effect that was selected as significant.

Multiple-testing correction addresses the false positives that accumulate as the number of tests grows. One common criterion is the false discovery rate (FDR), which limits the expected proportion of false discoveries among the selected results. A corrected p-value is commonly labeled adjusted p-value or padj.

An FDR of 5% does not mean that each selected item has exactly a 5% probability of being wrong. It is a criterion intended to control, on average across repeated uses of the same analysis procedure, the proportion of false discoveries within the selected set.

Example: testing expression differences across many genes

RNA-seq differential expression analysis tests thousands of genes at once, so it requires multiple-testing correction. Tools such as PyDESeq2 model variation in each gene’s raw counts across samples and test differences associated with a condition.

At minimum, examine the following three values together. Reading DESeq2 and PyDESeq2 result tables explains how to connect all six columns in a DESeq2-family result table.

• baseMean: how much of the gene was observed across all samples
• log2FoldChange: in which direction, and by what factor, the two conditions differ
• padj: whether statistical evidence remains after testing many genes together

For genes with very low counts, a difference of only a few reads can produce a large ratio. Analysts therefore sometimes filter low counts in advance or use the analysis tool’s independent filtering.

Example: testing enrichment of a gene set

Suppose 20 of 100 candidate genes belong to an immune pathway. The important question is not the number 20 by itself, but whether that pathway is unusually common in the candidate list relative to its proportion among all genes that could have been analyzed.

An enrichment test compares the candidate list with a background gene set. Results change depending on whether the background is the entire genome or only the genes that the experiment could actually measure. Because many pathways are tested at once, this stage also requires multiple-testing correction.

Summary

Statistical testing compares an observed difference with variation across repeated measurements. A p-value describes how extreme the data are under the null model; it does not replace the magnitude or importance of the effect. When many items are tested at once, examine corrected p-values and FDR, and assess effect size and uncertainty together.