Posts

Why LoRA Works: The SVD Math Behind Efficient Fine-Tuning

Image
We keep talking about Gradient Descent when it comes to training a machine learning model. We calculate the gradient, adjust weights to minimize the loss. That sounds simple enough. But what does it mean to train a large language model with billions of parameters? Gradient descent still works exactly as it should — you compute gradients with respect to every single parameter and store optimizer states for all of them. What's the problem then? It works perfectly, but requires a lot of compute power and tens of gigabytes just to hold the training state. It requires GPUs, possibly a connection to a data center. What if you are a student or a researcher with just a laptop? Do you give up on fine-tuning a model with billions of parameters? What if you could train a model where, instead of modifying billions of weights, you only tune a few parameters — and still get nearly the same result? The answer is Low Rank Approximation . And to understand it, ...
Post a Comment
Read more

The Gradient Vector: The One Mathematical Object That Trains Every Neural Network

Image
If you are standing on a mountain, you check the slope and start walking to reach the bottom of the mountain. How does the machine do it? If the input data contains just one feature, then the machine will adjust its weight to minimize the loss. It means it either has to move right or left depending upon the slope. What if there are more than one feature? ChatGPT has 175 billion parameters. Training it means finding the lowest point in a 175 billion dimensional space. That is the mathematical problem every AI trainer faces. And, there is a solution for this. The Single Parameter Case If we just have a single parameter, the problem gets easy — we calculate the slope and decide which way to move. The moment we have another parameter, just slope doesn't work anymore. Imagine standing on a mountain or somewhere on a bowl — you have infinite directions to move. And which direction to choose is the key. We can't c...
Post a Comment
Read more

Your Code Comments Now Have a Second Audience

I was checking what tools were available to GitHub Copilot from the MCP server I had just built. It showed a list. Each entry had a tool name at the top — the actual function name exported as an MCP tool. Next to it, in slightly smaller text, a description. Wait,  I hadn't written any descriptions. Not in the MCP configuration, not in any separate documentation file. I had written nothing that I thought of as a description. Then I realized. That smaller text was my docstring. The comment I wrote at the top of the function — the one explaining what the procedure does, what arguments it takes, what it returns — was sitting right there, being shown to Copilot as the tool's documentation. The AI was reading my code comments. Not inferring from them. Using them directly, word for word, to understand what the tool does and how to use it. That was the moment I realized something had quietly shifted about what a code comment actually is. The Old ...
Post a Comment
Read more

Connecting an On-Prem MCP Server to Claude Desktop

I built an MCP server that exposes product data and lets users interact with it the same way they would through the UI — just through natural conversation with an AI agent instead. The idea is simple. The setup, less so. The server runs on-premises — not exposed to the internet. And getting Claude Desktop to talk to it for end users who aren't developers turned out to be more interesting than I expected. Here's what I tried, what didn't work, and what finally did. The Goal Once connected, a user can open Claude Desktop and interact with the product naturally — asking questions, retrieving data, performing actions — without navigating the UI. The AI agent handles the interaction through the MCP server. Getting there required working around a constraint most MCP guides don't address — what happens when your server isn't on the internet? What Didn't Work Copilot Studio The first option I explored was cr...
Post a Comment
Read more

The Machine's Compass: Why AI Needs Calculus to Learn

Image
The Machine's Compass: Why AI Needs Calculus to Learn Every Machine Learning tutorial tells you the same thing: To train a Neural Network, you must calculate the Gradient . But... what actually is a Gradient? And what happens to your training loop if you try to compute it without Calculus? Your AI thinks it's standing on flat ground. It stops learning. Permanently. In this Applied Engineering Lab, we decode the math behind Gradient Descent. We prove visually why trying to calculate the slope using basic Python code triggers a silent hardware bug called Catastrophic Cancellation — and why Calculus is the only way to fix it. The Problem: A Blindfolded Machine on a Mountain Imagine you are dropped onto the side of a massive mountain. You are blindfolded. Your only instruction: find the lowest point of the valley. You can't see a map. You can't see the bottom. So you do the only thing you can — you feel...
Post a Comment
Read more

Entropy & Cross-Entropy Explained: Why MSE Fails for Classification

Image
Why Neural Networks use Cross-Entropy: The Math of "Surprise" Every Machine Learning tutorial tells you the same thing: If you're predicting a continuous number (like a house price), use Mean Squared Error (MSE) . But if you are classifying an image (like Cats vs. Dogs), you must use Cross-Entropy Loss . But why is that? What actually happens if you try to use Squared Error to classify a cat? Your Neural Network just gives up. It doesn't get trained properly. In this Applied Engineering Lab, we decode exactly why MSE fails for classification, how Information Theory solves it using the math of "Surprise," and how to fix a critical math bug that will crash your production servers. The Problem: Why MSE Fails for Classification Regression is about how far off you are. Squaring the error is great for punishing a $50,000 mistake on a house price prediction. But Classification is just Tru...
Post a Comment
Read more

How Spam Filters Work: Coding Naive Bayes from Scratch

Image
Building a Spam Detector from Scratch: The Naive Bayes Classifier Three billion emails are sent every hour. And nearly half of them are spam. "Congratulations! You won free money. Click here now." If we look at this email, we immediately realize it is spam. But how does the machine realize it? It's an engineering problem . An email has thousands of words, and each word changes the probability of it being spam or not spam. Checking how these words relate to each other is computationally impossible. We need to make a naive assumption to solve it. This assumption leads us to one of the most effective classifiers: The Naive Bayes Classifier . The "Naive" Assumption The Naive Bayes classifier assumes that all features are conditionally independent given the class . What does this mean in simple terms? It assumes that the presence of one word does not affect the probability of another wo...
3 comments
Read more