What is a Vector Space? (The Math That Powers Word2Vec)

Vector Spaces: The "Playground" Where Your Data Lives

How does a computer understand that the word 'King' is related to 'Man' in the same way that 'Queen' is related to 'Woman'? Computers don't understand abstract meaning; they only understand numbers and structure.

To solve this, we represent our data—words, images, users—not as isolated items, but as points on a special kind of mathematical map with a strict set of rules. This 'map' is called a Vector Space. It's the playground where our data lives, and its rules are what allow machine learning models to perform seemingly magical feats of logic.

In this post, we'll build this concept from the ground up, starting with its most fundamental building block.

Watch the video for the full visual explanation, then scroll down for the detailed definitions and axioms.

The Foundation: What is a Mathematical "Group"?

Before we can build a Vector Space, we need to understand a simpler structure: a Group. A Group consists of just two things: a set of elements and a single operation that combines them. For this combination to qualify as a Group, it must satisfy four strict rules, or axioms.

The Developer's Mental Model: Think of these axioms as an Interface or a Contract. Any system that "implements" this interface is guaranteed to be predictable, self-contained, and reversible.

Let's use the set of all integers (ℤ) with the operation of addition (+) as our example.

  • 1. Closure: If you take any two elements from the set and apply the operation, the result is still in the set. (An integer plus an integer is always another integer).
  • 2. Associativity: The grouping of operations doesn't change the outcome. `(a + b) + c = a + (b + c)`.
  • 3. Identity Element: There must be a special "do nothing" element. For integer addition, this is `0`, since `a + 0 = a`.
  • 4. Inverse Element: For every element, there must be an "undo" element that gets you back to the identity. For any integer `a`, its inverse is `-a`, since `a + (-a) = 0`.

Because the integers with addition satisfy all four rules, they form a Group.

Building the Vector Space: The 10 Axioms

A Vector Space is a more complex structure built on top of the Group concept. It is a collection of objects called vectors which can be added together and scaled by numbers, called scalars.

For a collection to be a true Vector Space, it must satisfy 10 axioms. We can group these into two main categories.

Rule Group 1: The Vectors Themselves Form a Group

The first five rules simply state that the set of vectors (V) and the operation of vector addition (+) must form a special kind of Group (an Abelian Group).

  1. Closure under Addition: For any vectors v, w in V, their sum v + w is also in V.
  2. Associativity of Addition: (u + v) + w = u + (v + w).
  3. Identity Element of Addition: There is a zero vector 0 such that v + 0 = v for all v.
  4. Inverse Element of Addition: For every v, there is an inverse -v such that v + (-v) = 0.
  5. Commutativity of Addition: v + w = w + v.

Rule Group 2: The Interaction Between Scalars and Vectors

The next five rules define how scalars (from a field F, like the real numbers) interact with our vectors.

  1. Closure under Scalar Multiplication: For any scalar `a` and vector v, the product `a`v is also in V.
  2. Distributivity of Scalar Multiplication (1): `a`(v + w) = `a`v + `a`w.
  3. Distributivity of Scalar Multiplication (2): (`a` + `b`)v = `a`v + `b`v.
  4. Associativity of Scalar Multiplication: `a`(`b`v) = (`ab`)v.
  5. Identity Element of Scalar Multiplication: `1`v = v, where `1` is the multiplicative identity.

Vector Subspaces: A Space Within a Space

A Vector Subspace is simply a vector space that lives inside a larger parent vector space. To qualify as a subspace, a subset of vectors only needs to satisfy the two closure axioms; it inherits the other eight properties from its parent.

The Subspace Test: A subset is a subspace if and only if:

  1. It is closed under addition.
  2. It is closed under scalar multiplication.

The Developer's Mental Model: Think of a 3D video game world (our vector space). The flat ground is a perfect example of a subspace. If you add two vectors on the ground, the result is still on the ground. If you scale a vector on the ground, it remains on the ground. However, the surface of a sphere in that world is *not* a subspace, because adding two vectors on its surface will likely result in a vector that points outside the sphere.

Conclusion: The Payoff for Machine Learning

These rules might seem abstract, but they are the reason machine learning models like Word2Vec can perform feats of analogy. By representing words as vectors in a high-dimensional vector space, the model can use simple vector arithmetic to understand concepts like gender and royalty.

Vector('King') - Vector('Man') + Vector('Woman') ≈ Vector('Queen')

This is not magic; it's vector arithmetic performed in a well-behaved "playground" where the rules are guaranteed. Understanding the structure of that playground is the key to decoding the complexity of the models that live within it.

Your Turn...

Did the "Playground" or "Interface" analogies help clarify these abstract concepts? What other foundational math topics would you like to see decoded? Let me know in the comments!

This post is part of the "Linear Algebra: The Core of Machine Learning" series. For the previous part, check out: How to Solve Systems of Linear Equations.

Comments

Post a Comment

Popular posts from this blog

Character to Numeric conversion in RPG - Allow Blanks & Thousands Separator - IBM i

Image
Character to Numeric Conversion Converting a numeric value from a character field to decimal field is one thing that we use almost every other day.  There are many BIFs to do this (like %DEC, %INT, %FLOAT...).  What if the character field doesn't contain any data? Or, If the character field contains comma (,) as thousands separator?  Before we see how we deal with these, let's see what happens if a blank character field is used to convert a numeric value or if character field contains thousands separator.  In the above example, We are doing character to numeric conversion twice with a blank value and a number stored in character with thousands separator. We are enclosing both these operations in a Monitor block, so that program doesn't fail. Line - 9: Converting blank character to a decimal value. %DEC doesn't allow blank value by default (with no control options specified) and this statement would fail causing the program to run On-Error block.  Line - 19: Conv...
Read more

Gram-Schmidt Process Explained (The Math of QR Decomposition)

Image
The "Purification" Algorithm: A Developer's Guide to the Gram-Schmidt Process In linear algebra, we often start with a basis for our vector space. But as we've discussed, not all bases are created equal. A "messy" basis—where vectors are skewed and not at 90-degree angles—can make calculations inefficient and numerically unstable. This is a real problem for engineers and data scientists who need reliable and fast computations. What if there was an algorithm that could systematically take any messy, "contaminated" toolkit of vectors and "purify" it into a perfect, 90-degree, orthonormal one? There is. It's an elegant and powerful procedure called the Gram-Schmidt Process . In this post, we'll decode this algorithm not as a formula to be memorized, but as an intuitive, step-by-step process of cleaning our data's toolkit. Watch the video for the full visual story, then scroll down for the detailed...
Read more

What are the Measures of Variability in Statistics?

What are the Measures of Variability in Statistics? Data is everything in AI today. Knowing the middle value—like mean or median—is good, but we also need to know how much the data spreads. Let's take a look about different measures of variability: range, variance, standard deviation, and interquartile range. This is in continuation to understanding Central Tendency of data (If you have not already read that post, Have a quick look at “ What is Central Tendency? ”).  What is Variability? Variability shows how far data points are from each other or the center. For example, If we say a data set has a mean of 11.4 doesn’t tell us if numbers are close to it or all over. In AI, spread matters—big swings might mean bad data for a model. For example, In any application different job run times spread could show system issues. We measure variability with: Range Variance Standard Deviation Interquartile Range Let’s use...
Read more