Decoding Complexities

A language model (the AI that writes text one word at a time) scores every possible next word, then picks one at random, weighted by those scores. Each score becomes a percentage chance, and the chances add up to 100%. Many systems first cut that list down to a shortlist and pick only from the shortlist. The simplest cut is top-k sampling: keep the k most likely words, throw away everything else, and divide each kept chance by the kept total so the kept chances add up to 100% again. Top-k is popular for a good reason: the shortlist is bounded and predictable. But the same rule that makes it predictable also makes it blind. The k is a number you choose ahead of time. A k of 5 means 5 words whether the model is nearly certain or nearly guessing. This post shows what that blindness does, using a made-up list of 12 possible words and real, rerunnable code. The Rule Top-k takes the model's chances for each word, sorts them from highest to lowest, keeps the fi...

Copy a text-generation tutorial and you'll almost always find this line, unexplained: outputs = model.generate( **inputs, max_new_tokens=200, temperature=1.0, ) That's the standard Hugging Face transformers library convention for generating text — max_new_tokens just caps output length and isn't covered here; temperature is. Left at its default, tweaked without explanation, or blamed vaguely for a model being "too random" or "too repetitive" — most people setting this number couldn't say exactly what it does to the math underneath. Temperature doesn't pick a word. It reshapes the entire probability distribution the model is choosing from, before anything downstream ever runs. This post builds a tiny, fully verifiable version of that reshaping from scratch, so every number below is something you can rerun yourself, not a claim to take on faith. What the Model Actually Outputs At every step, a langua...

Give a small, free language model (an AI that writes text by predicting one small piece of text at a time) the start of a sentence, called the prompt : "In 1969, humans first" . Then continue it two different ways, called decoding strategies (the step where the model's guesses about what comes next get turned into actual words): greedy (always take the single likeliest next word) and beam search (keep several candidate sentences going at once, like exploring a few paths instead of betting on one). Same model, same prompt, two different endings: Greedy: In 1969, humans first used the Internet to communicate. In 2000, the Internet Beam: In 1969, humans first landed a man on the moon. In 2009, the The first one is fluent, and factually wrong. The model's own scoring (explained below) rates the whole second continuation about 950 times more likely than the first, and greedy decoding didn't find it here, because greedy decoding picks the most p...

Ask an LLM (large language model — the kind of AI behind tools like ChatGPT) the exact same question twice. Sometimes you get the exact same answer back, word for word. Other times you get two answers that take different angles on the same question, worded completely differently. Both are normal — and both come down to one setting (a parameter passed into the code — not to be confused with a model's parameters, the separate term for its internal size), hiding in plain sight in the same line of code everyone copies from a tutorial: outputs = model.generate( **inputs, max_new_tokens=40, # caps the reply at 40 tokens (word pieces) do_sample=False, ) That's a simplified version of the standard Hugging Face (a popular platform and code library for running AI models) transformers convention for generating text — the full version used for every result below pins a couple of extra settings for consistency, shown later. do_sample decides which of two en...

A support bot got asked "How many days do I have to recover a deleted file?" and handed back a chunk about refund policy instead. That's a retrieval failure — the step in a RAG (Retrieval-Augmented Generation) pipeline that finds the relevant piece of text before an LLM (Large Language Model) ever sees the question. A previous investigation traced one cause of bad retrieval back to chunking — how a document gets cut into pieces before anything gets compared. This time chunking wasn't the bug. The chunks were fine. The comparison itself still picked the wrong one. The refund-policy chunk scored 0.181 against that query. The chunk that actually explains the file-recovery window scored 0.134 . Not a rounding error. A clear, confident, wrong ranking, reproduced with seven lines of scikit-learn (a free Python library for exactly this kind of text math). Cosine similarity was doing exactly what it was designed to do. The problem was what it was being ask...

A model scores 97% accuracy in testing — the percentage of predictions it gets right on data it never trained on. You deploy it to production, and real-world accuracy drops to under 78%. Nothing crashed. Nothing was miscoded. So what actually broke? Nothing broke. The test score was measuring something that doesn't exist in the real world. This is data leakage — the quietest, most dangerous failure mode in machine learning, because it doesn't look like a failure at all. It looks like success. What Data Leakage Actually Is Data leakage is information that shouldn't be available at prediction time sneaking into training anyway. The model learns to rely on it. The test score looks great, because that same leaked information is sitting in the test set too. The moment the model is in the real world, making a prediction before that information exists, it falls apart. Isn't more information always better for a model, thou...