PCA: Principal Component Analysis

PCA (principal component analysis) is a statistical method that finds major directions of variation in data with many numerical features and reduces the number of axes. Rather than discarding original features one by one, it combines them to create new axes such as PC1 and PC2.

Each point in a PCA scatter plot is one original row. Nearby points have similar overall patterns across the input features, while distant points differ. PCA does not tell you why they are similar or different, so inspect metadata attached to each row, such as group, time point, and equipment.

Convert many numerical columns into PC1 and PC2

Suppose metrics collected from 100 servers are stored in this DataFrame.

server	cpu	memory	latency	error_rate	…
S1	42	68	103	0.01	…
S2	39	71	98	0.01	…
S3	91	84	430	0.12	…

Each row is one observation and each numerical column is one feature describing that observation. With 5,000 features, one server is a vector containing 5,000 numbers. Because humans cannot plot this space directly, PCA combines the original columns into new ones.

server	PC1	PC2
S1	-18.4	3.2
S2	-17.1	2.7
S3	21.6	-1.4

PC1 and PC2 are not originally measured features such as CPU or latency. They are new coordinates formed by multiplying several features by different weights and adding them. Using these two columns as the x- and y-axes lets you plot 100 rows on a plane.

The four steps PCA uses to choose axes

PCA creates new coordinate axes in this order.

1. Subtract each feature’s mean to move its centre to 0.
2. Find the direction in which the observations spread out the most and call it PC1.
3. Find the direction perpendicular to PC1 with the greatest remaining variation and call it PC2.
4. Project each observation onto (PC1, PC2) coordinates and draw it on the plane.

In the synthetic data below, each point is one observation, and the horizontal and vertical axes are two numerical features. As you rotate the candidate axis, each point is compressed onto it as an orange projected point. The direction in which those projected points spread out the most is PC1.

If the matrix is X and the matrix of feature weights is W, the new coordinates can be written as:

$$Z = X_{\text{centered}} W$$

The rows of Z are the original observations, and its columns are the new axes. An observation’s new coordinate is called a score, while the weight used by each feature to construct an axis is called a loading.

How to read points and distances

There are four things to inspect in a PCA scatter plot.

• One point: one row of the input matrix.
• Distance between points: how similar the overall numerical patterns are under the chosen features and preprocessing.
• Point colour and shape: metadata such as groups or measurement times overlaid later by a person.
• A distant point: something to investigate, which may be a genuinely unusual observation or a measurement or labelling problem.

PCA is an unsupervised analysis that creates axes without looking at group labels. If two colour-coded groups separate, PCA did not learn to classify the groups. The major direction of variation found without labels happened to correspond well to those groups.

What is 40% in PC1 40%?

The percentage beside an axis is the explained variance ratio. PC1 40% means that PC1 alone contains 40% of the total variation among observations in the input matrix.

It does not mean:

• 40% accuracy in classifying rows
• 40% of the features were preserved unchanged
• A particular group caused 40% of the total variation

If PC1 and PC2 explain 40% and 15%, the two-dimensional plot shows 55% of the total variation. The remaining 45% remains in PC3 and later axes, so points close together on the plane may differ along other axes.

Left and right, or up and down, can flip

Multiplying every weight of an axis by -1 still represents the same direction of variation. Consequently, when different tools calculate PCA on the same data, a group that appears on the right side of PC1 in one plot may appear on the left in another.

This sign flip is not an error. Distances between points, separation between groups, and explained variance ratios remain unchanged. When comparing plots, focus on which points cluster and which groups separate, not their absolute left-right position.

Use loadings to find which features created an axis

The scatter plot shows relationships between observations, but it does not directly show which features contributed most to an axis. Features with large absolute loadings on PC1 are columns that contributed strongly to separating observations along the PC1 direction.

Positive and negative loadings indicate the direction of the axis with which each feature is associated. Because the sign of the entire axis can flip, do not fix an interpretation such as “positive always means group A.” Read point positions and loading directions together within the same run.

Check scale before calculation

PCA is affected by numerical units and magnitude. If CPU use is between 0 and 100 while request counts are in the millions, variation in request count may dominate the distance. Inspect the meaning and distribution of each feature before choosing transformations or standardization.

scikit-learn’s PCA centres features by subtracting their means, but it does not automatically scale them to a standard deviation of 1. Converting every feature to a z-score gives features with small and large units equal weight, producing a different analysis. Do this only if it fits the question, not merely by convention.

For data whose variance grows with the mean, a transformation such as VST can help reduce the mean-variance relationship.

Example: applying PCA to a gene expression matrix

In an RNA-seq expression matrix, each row is a sample and each column is a gene. With 20,000 genes, one sample is a vector of 20,000 elements. PCA reduces these vectors to PC1 and PC2 coordinates and shows relationships among samples.

In raw counts, genes with larger means also tend to have larger variances, and sequencing depth differs between samples. Before PCA, analysts often remove low counts and special counters, account for sequencing depth, and apply a transformation such as VST or rlog to reduce the relationship between count mean and variance.

If samples separate by condition in PCA, that is a clue that the condition is associated with a large difference in the overall expression pattern. Batch, tissue characteristics, or quality differences that move with the condition can produce the same separation. PCA is an exploratory tool and does not replace differential-expression analysis, which tests the condition effect gene by gene.

Questions to ask when reading the plot

1. What observational unit does one point represent?
2. Which features and preprocessed values were given to PCA?
3. What percentage of variance do PC1 and PC2 each explain?
4. Have the points also been coloured by metadata such as time, equipment, and batch, not only group?
5. Have the measurement quality and metadata of distant points been checked?
6. Are you inferring the cause of separation from the PCA plot alone?

To compare PCA with UMAP or examine the meaning of high-dimensional distance, continue to Sample Distance and Dimensionality Reduction.

Official resources

• scikit-learn PCA API documentation
• scikit-learn explanation of PCA