How to Crush Big Data: The Math of PCA (Principal Component Analysis)

How to Crush Big Data: The Math of PCA (Principal Component Analysis)

In the real world, data is massive. A single image has millions of pixels. A financial model has thousands of market indicators.

We call this the Curse of Dimensionality. When you have 10,000 features, your data becomes sparse, distance calculations break down, and models become incredibly slow. Visualization? Impossible.

So, how do we fix this? How do we take a massive, high-dimensional monster and squash it down to just 2 or 3 dimensions without losing the important information?

The answer is Principal Component Analysis (PCA). In this post, we will strip away the complexity and build PCA from scratch in Python using a 5-step linear algebra recipe.

The Intuition: Information = Variance

Before the math, we need to define our goal. What does it mean to "keep information"?

In Data Science, Information is Variance (Spread). If data is squashed together, it looks like noise. If it is spread out, we can distinguish patterns.

Our goal is to find new axes (Principal Components) that maximize this spread.

The Algorithm: A 5-Step Recipe

PCA isn't magic; it is a deterministic algorithm. Here is the blueprint:

  1. Centering: Shift the data so the mean is zero.
  2. Standardization: Scale features so they are comparable.
  3. Covariance Matrix: Calculate how features vary together.
  4. Eigen-Decomposition: Find the axes (Eigenvectors) of that matrix.
  5. Projection: Rotate the data to align with the new axes.

Step 1 & 2: The Data Problem

Let's imagine a dataset of 200 people with two features: Height (cm) and Weight (kg).

# Generate correlated data
height = np.random.normal(170, 10, n_samples)
weight = (0.5 * height) - 20 + noise
X = np.column_stack((height, weight))

Height is around 170. Weight is around 70. The scales are totally different. If we run PCA on raw data, "Height" will dominate the math simply because the numbers are bigger.

We fix this by Standardizing (subtracting the mean and dividing by standard deviation).

X_centered = X - np.mean(X, axis=0)
X_std = X_centered / np.std(X_centered, axis=0)

Step 3 & 4: The Mathematical Engine

Now we need a map of how the data is shaped. We calculate the Covariance Matrix ($\Sigma$).

Σ = (1/n) * XᵀX

To find the "Principal Axes" of this shape, we calculate the Eigenvectors and Eigenvalues.

# Step 3: Covariance
cov_matrix = np.cov(X_std.T)

# Step 4: Eigen Decomposition
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

The Eigenvectors point in the direction of the spread (Height and Weight increasing together). The Eigenvalues tell us the magnitude of that spread.

Step 5: The Projection (Rotation)

Finally, we multiply our data by the sorted Eigenvectors. This effectively rotates the entire dataset.

X_pca = X_std @ eigenvectors

The Result:

  • PC1 (Principal Component 1): Represents "Overall Size." It captures 86% of the variance.
  • PC2 (Principal Component 2): Represents "Build" (Shape). It captures only 14% of the variance.

In a high-dimensional dataset, you might find that the top 3 components hold 95% of the information. You can throw away the other 9,997 dimensions and barely lose a thing.

What's Next?

We have mastered PCA for square Covariance Matrices. But in the real world, raw data matrices are often rectangular (rows ≠ columns).

How do we decompose any matrix of any shape? This leads us to the Fundamental Theorem of Linear Algebra: The Singular Value Decomposition (SVD). That is the topic of our next post.

Get the Code

Want to run this yourself? Check out the Google Colab Notebook to visualize the transformation step-by-step.

This post is part of the "Linear Algebra for Machine Learning" series. For the previous part, check out: The Hidden "DNA" of Matrices: Eigenvalues & Eigenvectors.

Comments

Post a Comment

Popular posts from this blog

Create/Write Data to IFS file from SQL - IBM i

Image
IFS Stream File IFS (Integrated File System) stream files are major part of any (or most of) IBM i applications.  There are various ways to create stream files like CPYFRMIMPF, CPYFRMSTMF, EDTF... Creating the IFS file and writing the data has been made much easier with SQL procedure IFS_WRITE (IFS_WRITE_BINARY for writing the data in Binary format and IFS_WRITE_UTF8 for writing the data in UTF8 format).  In this post, we will see how to write the data into Stream file directly from SQL using IFS_WRITE procedure.  E.g.:  To start with, What we need is 'Path Name (IFS Directory & File Name)' and 'Data to be written'.  First parameter 'PATH_NAME' accepts the IFS path name along with the file name. In this example, '/home/REDDYP' is the directory and 'NewTextFile.txt' is the file name.  Directory should be present by the time the procedure is called. If not, Procedure call would be failed.  File name doesn't necessarily need to be present. I...
Read more

Gram-Schmidt Process Explained (The Math of QR Decomposition)

Image
The "Purification" Algorithm: A Developer's Guide to the Gram-Schmidt Process In linear algebra, we often start with a basis for our vector space. But as we've discussed, not all bases are created equal. A "messy" basis—where vectors are skewed and not at 90-degree angles—can make calculations inefficient and numerically unstable. This is a real problem for engineers and data scientists who need reliable and fast computations. What if there was an algorithm that could systematically take any messy, "contaminated" toolkit of vectors and "purify" it into a perfect, 90-degree, orthonormal one? There is. It's an elegant and powerful procedure called the Gram-Schmidt Process . In this post, we'll decode this algorithm not as a formula to be memorized, but as an intuitive, step-by-step process of cleaning our data's toolkit. Watch the video for the full visual story, then scroll down for the detailed...
Read more

Linear Independence & Dependence Explained (The Key to Feature Engineering)

Image
This is Why Your Machine Learning Model is Unstable (Linear Independence) Imagine you're building a machine learning model to predict a person's weight. You have a dataset with features like height in centimeters and age. To improve the model, you decide to add another feature: height in inches. Your intuition might say "more data is better," but as an engineer, you should spot the flaw immediately. The "height in inches" feature is 100% redundant . It provides zero new information that "height in centimeters" doesn't already. This common-sense idea of redundancy has a formal, powerful name in linear algebra: Linear Dependence . Understanding the difference between linear dependence and independence is not an abstract academic exercise; it's a critical concept with huge real-world consequences for building simpler, faster, and more robust ML models. Watch the video for the full visual explanation, then scrol...
Read more