Dimensionality Reduction

Dimensionality reduction compresses many features into a few, aiding visualization, speed, and defeating sparsity.

Fewer, richer dimensions

Dimensionality reduction maps high-dimensional data into a lower-dimensional space that keeps the structure that matters. It serves several goals at once: visualizing data in 2-D or 3-D, speeding and stabilizing downstream models, removing noise and redundancy, and countering the curse of dimensionality.

Linear versus nonlinear

• Linear (PCA, factor analysis): finds a linear projection; fast, interpretable, invertible, but blind to curved structure.
• Nonlinear / manifold (t-SNE, UMAP, kernel PCA, autoencoders): capture curved structure, better for visualization, harder to interpret.

See PCA for the linear workhorse and t-SNE and UMAP for nonlinear visualization.

Feature extraction versus selection

Dimensionality reduction creates new features as combinations of the originals (feature extraction), whereas feature selection keeps a subset of the original features unchanged. Extraction usually compresses more but sacrifices the direct meaning of each dimension; selection preserves interpretability.

The manifold hypothesis

Real high-dimensional data, images, spectra, sensor arrays, typically lies near a much lower-dimensional surface embedded in the feature space. Its intrinsic dimension is far below its nominal one. Dimensionality reduction works precisely because it recovers coordinates on that surface. In engineering, reducing a large bank of simulation outputs to a few dominant modes exposes the handful of factors that actually drive a design's behavior.

Choosing a method

Use PCA as a fast first step and for compression that must be interpretable and reversible; use UMAP or t-SNE when you need to see cluster structure; use autoencoders when you have abundant data and want a learned nonlinear code. Always scale features first.