Decoding Complexities

I was running a language model fully locally: no API, no network call, just the model's weights (the numbers it learned during training) on my own machine. The first token (a word or a piece of a word) took noticeably longer to appear than the ones after it. My first assumption was that the model was "warming up", some one-time setup cost. A large part of the explanation turned out to be structural, and it has a name: the KV cache. A cache is a saved copy kept so you don't redo work, and "KV" stands for Key and Value, two lists of numbers that attention (the step where a model looks back at earlier words) builds for every token. After the first step, every new token is cheap because it computes only its own Key and Value and reuses the saved ones for everything before it. It's one of those pieces of behind-the-scenes machinery that people who build with AI rely on every day without ever being asked to explain it. What Generation...

A language model writes text one token at a time (a token is a word or a piece of a word). At each step it gives every possible next token a probability, then picks one at random, weighted by those probabilities. That weighted random pick is called sampling. Top-k sampling limits the pick to a fixed number of the highest-probability tokens. That's predictable, but it ignores whether the model is sure or spread across many options. Top-p sampling, also called nucleus sampling, keeps the smallest group of top tokens whose probabilities add up to at least p , where p is a number you choose, such as 0.90 for 90%. ("Nucleus" just means the core group of likely tokens.) When one or two tokens dominate, that group is tiny. When the probabilities are spread out, the group widens. This post shows it on a small made-up vocabulary of 12 candidate answers, with code you can rerun. The Rule Sort the probabilities from highest to lowest and add them up in or...

A language model (the AI that writes text one word at a time) scores every possible next word, then picks one at random, weighted by those scores. Each score becomes a percentage chance, and the chances add up to 100%. Many systems first cut that list down to a shortlist and pick only from the shortlist. The simplest cut is top-k sampling: keep the k most likely words, throw away everything else, and divide each kept chance by the kept total so the kept chances add up to 100% again. Top-k is popular for a good reason: the shortlist is bounded and predictable. But the same rule that makes it predictable also makes it blind. The k is a number you choose ahead of time. A k of 5 means 5 words whether the model is nearly certain or nearly guessing. This post shows what that blindness does, using a made-up list of 12 possible words and real, rerunnable code. The Rule Top-k takes the model's chances for each word, sorts them from highest to lowest, keeps the fi...

Copy a text-generation tutorial and you'll almost always find this line, unexplained: outputs = model.generate( **inputs, max_new_tokens=200, temperature=1.0, ) That's the standard Hugging Face transformers library convention for generating text — max_new_tokens just caps output length and isn't covered here; temperature is. Left at its default, tweaked without explanation, or blamed vaguely for a model being "too random" or "too repetitive" — most people setting this number couldn't say exactly what it does to the math underneath. Temperature doesn't pick a word. It reshapes the entire probability distribution the model is choosing from, before anything downstream ever runs. This post builds a tiny, fully verifiable version of that reshaping from scratch, so every number below is something you can rerun yourself, not a claim to take on faith. What the Model Actually Outputs At every step, a langua...

Give a small, free language model (an AI that writes text by predicting one small piece of text at a time) the start of a sentence, called the prompt : "In 1969, humans first" . Then continue it two different ways, called decoding strategies (the step where the model's guesses about what comes next get turned into actual words): greedy (always take the single likeliest next word) and beam search (keep several candidate sentences going at once, like exploring a few paths instead of betting on one). Same model, same prompt, two different endings: Greedy: In 1969, humans first used the Internet to communicate. In 2000, the Internet Beam: In 1969, humans first landed a man on the moon. In 2009, the The first one is fluent, and factually wrong. The model's own scoring (explained below) rates the whole second continuation about 950 times more likely than the first, and greedy decoding didn't find it here, because greedy decoding picks the most p...

Ask an LLM (large language model — the kind of AI behind tools like ChatGPT) the exact same question twice. Sometimes you get the exact same answer back, word for word. Other times you get two answers that take different angles on the same question, worded completely differently. Both are normal — and both come down to one setting (a parameter passed into the code — not to be confused with a model's parameters, the separate term for its internal size), hiding in plain sight in the same line of code everyone copies from a tutorial: outputs = model.generate( **inputs, max_new_tokens=40, # caps the reply at 40 tokens (word pieces) do_sample=False, ) That's a simplified version of the standard Hugging Face (a popular platform and code library for running AI models) transformers convention for generating text — the full version used for every result below pins a couple of extra settings for consistency, shown later. do_sample decides which of two en...