Why we use Mean Squared Error: Maximum Likelihood Estimation (Python Simulation)

Why We Use Mean Squared Error: Decoding Maximum Likelihood Estimation

In our previous exploration of the Central Limit Theorem, we established that the error in Linear Regression follows a Normal Distribution. This is a huge win—it gives us the shape of the error.

But there is no single distribution that fits every dataset. There are infinite Bell Curves—some wide, some narrow, some shifted to the left or right. This raises a critical question: How do we find the specific distribution parameters that best fit our data?

In simple words, we need to find the parameters (like Mean or Slope) that minimize the error. We do this using a technique called Maximum Likelihood Estimation (MLE).

Likelihood vs. Probability: What's the Difference?

In English, "Likelihood" and "Probability" are synonyms. In Mathematics, they are opposites.

  • Probability is the art of estimating an event (or data) occurring, given that we already have the model parameters. (Forward looking).
  • Likelihood is about estimating the model parameters, given that we already have the data. (Backward looking).

In Machine Learning, we start with the data. We need to find the model. Therefore, we calculate Likelihood.

The "Underflow" Bug & The Log-Likelihood Fix

The Likelihood Function calculates a score for how well a model fits. Mathematically, it is the product of the probability densities of every point in the dataset.

However, probability densities are often small numbers (e.g., 0.01). If you have a dataset with thousands of points, you are multiplying thousands of small numbers together.

0.01 * 0.01 * 0.01 ... quickly approaches Zero.

This causes Numerical Underflow—the computer runs out of precision and rounds the result to 0.0. To avoid this, we use Log-Likelihood.

By taking the Logarithm, we convert multiplication into addition. Instead of a crashing product, we get a stable sum of logs.

The Simulation: Finding the Hidden Mean

Let's simulate this in Python. We generate data from a Normal Distribution with a True Mean of 150 and a Standard Deviation of 10.

# 1. Generate Data (Truth = 150)
data = np.random.normal(150, 10, 1000)

# 2. Naive Approach (Product) -> Results in 0.0
def calculate_likelihood(data, mean, standard_deviation):
    # Get probability density for every point
    probabilities = norm.pdf(data, loc=mean, scale=standard_deviation)
    # Multiply them all probabilities
    return np.prod(probabilities)

# 3. Log Approach (Sum of Logs) -> Results in a stable negative number
def calculate_log_likelihood(data, mean, standard_deviation):
    probabilities = norm.pdf(data, loc=mean, scale=standard_deviation)
    # Take Log first, then Sum (Turns multiplication into addition)
    return np.sum(np.log(probabilities))

In the real world, we don't know the True Mean. So, we search for it. We generate a list of guesses (means ranging from 140 to 160) and calculate the Log-Likelihood for each.

When we plot the results, we see a curve forming a hill. The peak of this hill—the Maximum Likelihood—occurs at approximately 150.10. The math successfully recovered the hidden parameter.

The Grand Unification: MLE = MSE

How does this relate to Linear Regression? We use Mean Squared Error (MSE) as our loss function. Is this just a random choice?

Let's verify this visually. Using the same data, let's calculate the MSE for every guess in our list and plot it on the same graph as the Likelihood.

The Result: The Mean Squared Error is at its Minimum at the exact same point where the Likelihood is at its Maximum.

This proves that Mean Squared Error is statistically the best way to measure error for Linear Regression, assuming Gaussian noise.

Conclusion & What's Next?

Maximum Likelihood Estimation gives us the answer, but it requires a lot of data. What if we have very little data? Or what if we have prior knowledge about the world before we even look at the data?

For that, we need a different approach: Bayesian Inference. That is what we will decode in the next post.

Get the Code

Want to run the simulation yourself? Check out the Google Colab Notebook.

This post is part of the "Probability for Machine Learning" series. For the previous part, check out: Why Linear Regression Actually Works (Central Limit Theorem.

Comments

Post a Comment

Popular posts from this blog

Create/Write Data to IFS file from SQL - IBM i

Image
IFS Stream File IFS (Integrated File System) stream files are major part of any (or most of) IBM i applications.  There are various ways to create stream files like CPYFRMIMPF, CPYFRMSTMF, EDTF... Creating the IFS file and writing the data has been made much easier with SQL procedure IFS_WRITE (IFS_WRITE_BINARY for writing the data in Binary format and IFS_WRITE_UTF8 for writing the data in UTF8 format).  In this post, we will see how to write the data into Stream file directly from SQL using IFS_WRITE procedure.  E.g.:  To start with, What we need is 'Path Name (IFS Directory & File Name)' and 'Data to be written'.  First parameter 'PATH_NAME' accepts the IFS path name along with the file name. In this example, '/home/REDDYP' is the directory and 'NewTextFile.txt' is the file name.  Directory should be present by the time the procedure is called. If not, Procedure call would be failed.  File name doesn't necessarily need to be present. I...
Read more

Linear Independence & Dependence Explained (The Key to Feature Engineering)

Image
This is Why Your Machine Learning Model is Unstable (Linear Independence) Imagine you're building a machine learning model to predict a person's weight. You have a dataset with features like height in centimeters and age. To improve the model, you decide to add another feature: height in inches. Your intuition might say "more data is better," but as an engineer, you should spot the flaw immediately. The "height in inches" feature is 100% redundant . It provides zero new information that "height in centimeters" doesn't already. This common-sense idea of redundancy has a formal, powerful name in linear algebra: Linear Dependence . Understanding the difference between linear dependence and independence is not an abstract academic exercise; it's a critical concept with huge real-world consequences for building simpler, faster, and more robust ML models. Watch the video for the full visual explanation, then scrol...
Read more

All about READ in RPGLE & Why we use it with SETLL/SETGT?

READ READ is one of the most used Opcodes in RPGLE. As the name suggests main purpose of this Opcode is to read a record from Database file. What are the different READ Opcodes? To list, Below are the five Opcodes.  READ - Read a Record READC - Read Next Changed Record READE - Read Equal Key Record READP - Read Prior Record READPE - Read Prior Equal Record We will see more about each of these later in this article. Before that, We will see a bit about SETLL/SETGT .  SETLL (Set Lower Limit) SETLL accepts Key Fields or Relative Record Number (RRN) as Search Arguments and positions the file at the Corresponding Record (or Next Record if exact match isn't found).  SETGT (Set Greater Than) SETGT accepts Key Fields or Relative Record Number (RRN) as Search Arguments and positions the file at the Next Record (Greater Than the Key value). Syntax: SETLL SEARCH-ARGUMENTS/KEYFIELDS FILENAME SETGT  SEARCH-ARGUMENTS/KEYFIELDS FILENAME One of the below can be passed as Search...
Read more