VST: stabilizing variance

VST (variance stabilizing transformation) is a statistical transformation that brings variances that differ with the mean level onto a similar scale. It applies a transformation function f to data whose fluctuations grow in larger-value regions, making the range of variation appear relatively consistent across regions after transformation.

VST is not the name of one fixed formula. Depending on the probability distribution of the data and the relationship between its mean and variance, you select a different function such as a square-root, logarithmic, or Anscombe transformation.

The same relative fluctuation produces different variances

Suppose repeated measurements at two levels are:

Level	Repeated measurements	Mean	Sample standard deviation
Low level	8, 10, 12	10	2
High level	800, 1,000, 1,200	1,000	200

Both rows fluctuate by about 20% of their mean. In the original units, however, their standard deviations are 2 and 200, a hundredfold difference. This property, in which variance grows together with the mean, is called heteroscedasticity.

If these values are used directly in a scatter plot, distance calculation, or least-squares regression, the high-level observations can dominate the calculation. VST compresses distances more strongly in the high range, placing fluctuations in low and high ranges on scales that are easier to compare.

The synthetic dataset below makes six variables fluctuate by the same proportion around their respective means. Move the F6 mean, then switch between the raw-value and VST views to see how absolute variance grows for a variable with a larger mean and becomes more even after transformation. This example uses log2(x + 1), which is suitable for proportional error.

The principle for choosing a transformation function

Let the mean of a value X be $\mu$ , and suppose its variance changes with the mean according to $v(\mu)$ . After applying a transformation function $f$ , the delta method approximates the variance of the transformed value as:

\operatorname{Var}[f(X)] \approx \left(f'(\mu)\right)^2 v(\mu)

$f'(\mu)$ is the slope of the transformation function near the mean. To make the expression above approximately constant and independent of the mean $\mu$ , set the slope according to this relationship:

f'(\mu) \propto \frac{1}{\sqrt{v(\mu)}}

This means that where variance grows rapidly, the function’s slope is made smaller to compress the distances between values more strongly. If the form of $v(\mu)$ is known, integrating this slope gives a suitable transformation function.

The formula changes with the mean-variance relationship

Representative relationships and transformations include:

Property of the data	Mean-variance relationship	Common transformation
Integer event counts with a Poisson form	$\operatorname{Var}(X) \approx \mu$	Square-root or Anscombe transformation
Positive data where error is proportional to the value	$\operatorname{Var}(X) \propto \mu^2$	Log transformation
Proportions with a fixed number of trials	$\operatorname{Var}(X) \propto p(1-p)$	Arcsine square-root transformation
Positive data whose relationship is difficult to specify in advance	Search the data for transformation strength	Box-Cox family transformation

For example, if variance is proportional to the mean, then $v(\mu)=\mu$ , so the required slope is proportional to $1/\sqrt{\mu}$ . Integrating it produces a square-root transformation.

For better correction of small values from a Poisson distribution, you can use the Anscombe transformation instead of a simple square root.

f(x) = 2\sqrt{x + \frac{3}{8}}

It resembles a square-root transformation for large values, while the $3/8$ adjustment improves the approximation for small values.

What VST does and does not do

VST applies the same function f to every value before transformation. If the original data is a table, the numbers of rows and columns remain unchanged and only the value in each cell changes.

With a monotonically increasing transformation, the order of values is generally preserved. The largest value does not become the smallest. Differences, ratios, and units do change, so interpretations from the original units such as twice as large or an increase of 100 cannot be applied directly to transformed values.

VST does not automatically perform any of these tasks:

Make the mean of every variable zero
Make the variance of every variable exactly one
Remove outliers
Remove group differences
Fix an incorrect model or measurement error
Calculate effect sizes in the original, untransformed units

The goal is not to make every variance exactly equal. It is to reduce systematic dependence between the mean level and variance.

The relationship between log transformation and VST

A log transformation is one of the most familiar VST candidates.

import numpy as np

transformed = np.log(x)

For data where measurement error grows as a constant proportion of the original value rather than by a constant absolute amount, a logarithm can compress the high range strongly and even out variance. It cannot be applied directly to zero or negative values, however, and it does not stabilize variance in every dataset.

Adding a constant c, as in log(x + c), handles zero, but the chosen constant greatly changes the shape of the low-value region. Instead of selecting a logarithm simply because it is familiar, first inspect the mean-variance relationship.

How does it differ from normalization and z-scores?

The three operations have different purposes.

Operation	What it changes	Representative result
VST	Variance that changes with the mean level	Similar ranges of variation in low and high regions
Normalization	Differences in overall scale across observational units	Totals or reference sizes aligned across units
z-score standardization	Location and scale for each variable	Mean 0 and standard deviation 1 for each variable

After calculation, a z-score makes the variance of each variable one, but it does not model why the original data was heteroscedastic. VST differs because it selects a nonlinear function suited to the relationship between mean and variance.

Centering or standardization can be added after VST when needed. The appropriate order and combination depend on the input properties required by the next analysis.

How does it differ from Mahalanobis distance?

VST is preprocessing that changes the values in a matrix, while Mahalanobis distance is a method for measuring the distance between two rows. Both can reduce the problem of high-variance variables dominating a calculation, but they act at different stages and use different information.

Consider two variables with these means and standard deviations:

Variable	Mean and standard deviation
Low level	$10 \pm 2$
High level	$1{,}000 \pm 200$

With Euclidean distance on the original values, a difference of 200 at the high level contributes much more than a difference of 2 at the low level. VST compresses distances around 1,000 more strongly so fluctuations at the two levels become similar in size. Ordinary Euclidean distance or PCA can then be applied to the transformed values.

Mahalanobis distance does not directly compress the original values. Instead, it gives a smaller weight to differences in directions with greater ordinary observed variance, and it also accounts for correlations in how multiple variables move together. If the two rows are $x$ and $y$ , and the covariance matrix of the variables is $\Sigma$ , the distance is:

d_M(x,y)=\sqrt{(x-y)^T\Sigma^{-1}(x-y)}

The covariance matrix contains the variance of each variable on its diagonal and the correlated movement of pairs of variables off the diagonal. Mahalanobis distance therefore avoids counting the same information twice as strongly as Euclidean distance when it is repeated across two variables.

Distinction	VST	Mahalanobis distance
What it changes	The value in each cell	The formula for distance between two rows
Main problem it addresses	Heteroscedasticity, where variance grows with the mean	Scale differences across variables and correlations between variables
Required estimate	The relationship between mean and variance	The covariance matrix across all variables
Result	A transformed data matrix	Distances between rows

Mahalanobis distance does not remove the nonlinear relationship between mean and variance itself. It changes the weighting of differences based on covariance estimated from the current data. Conversely, VST generally does not incorporate correlations between variables when calculating distance.

In data such as RNA-seq, with 20,000 genes but only dozens of samples, it is difficult to estimate a $20{,}000 \times 20{,}000$ covariance matrix reliably. When there are more variables than samples, the covariance matrix also cannot be inverted directly. VST is therefore often applied before sample PCA or clustering. If Mahalanobis distance is needed, one option is to apply VST, reduce the dimensions with PCA, and estimate covariance in a PC space that is sufficiently small relative to the sample count.

Relationship to machine learning

VST is statistical preprocessing. It can be used before methods such as PCA, clustering, and regression that are sensitive to distance, variance, or linear relationships.

A result looking better after transformation does not by itself establish that the VST was appropriate. Estimating the transformation function using validation data creates data leakage. In a prediction task, determine the transformation from the training data and apply the same function to the validation and test data.

How to check whether a transformation is appropriate

Plot the mean or fitted value of the original values against the spread of residuals.
Check whether the spread widens as the mean grows.
Select a candidate transformation suited to the data-generating distribution or the mean-variance relationship.
Draw the same plot after transformation and check whether the spread is more consistent.
Check that the transformation has not introduced an important nonlinear relationship or made interpretation excessively difficult.
Record the transformation function, constants, and estimation method so the treatment can be reproduced.

If a model can handle the variance structure directly, VST is not necessarily required first. For example, weighted least squares or a suitable generalized linear model can model the mean-variance relationship in the original units. Transformation is not the objective. It is one of several possible solutions.