PCA on a Small Dataset
Center a small dataset, compute its covariance eigenvectors, and project onto the leading principal component.
Problem
Principal component analysis (PCA) finds orthogonal directions that capture the most variance in data. Projecting onto the top few components reduces dimensionality while preserving structure. The components are eigenvectors of the covariance matrix, ranked by eigenvalue.
Steps
Center the data by subtracting the mean, form the covariance matrix, then find its eigenvectors and eigenvalues. The eigenvector with the largest eigenvalue is the first principal component; the eigenvalue equals the variance captured along it.
import numpy as np
X=np.array([[2.,0.],[0.,2.],[3.,1.],[1.,3.]])
Xc=X-X.mean(0)
C=np.cov(Xc,rowvar=False)
vals,vecs=np.linalg.eigh(C)
order=np.argsort(vals)[::-1]
vals=vals[order]; vecs=vecs[:,order]
pc1=vecs[:,0]
scores=Xc@pc1
print('eigenvalues',np.round(vals,3))
print('PC1',np.round(pc1,3))
print('variance explained',round(vals[0]/vals.sum(),3))Result
The data spread mostly along the diagonal, so the first principal component points near (1,1)/sqrt(2) and captures the majority of the variance. Projecting each centered point onto this direction gives a 1D summary that retains most of the information. The eigenvalues quantify exactly how much variance each direction holds, so you can decide how many components to keep.
- PCA is a linear method; it cannot capture curved structure, for which nonlinear embeddings exist.
- Always center, and often scale, features first, or the largest-magnitude variable dominates.
- PCA and its cousin POD compress high-dimensional simulation output in Kronos surrogate modeling.