Decoding Complexities

I was running a language model fully locally: no API, no network call, just the model's weights (the numbers it learned during training) on my own machine. The first token (a word or a piece of a word) took noticeably longer to appear than the ones after it. My first assumption was that the model was "warming up", some one-time setup cost. A large part of the explanation turned out to be structural, and it has a name: the KV cache. A cache is a saved copy kept so you don't redo work, and "KV" stands for Key and Value, two lists of numbers that attention (the step where a model looks back at earlier words) builds for every token. After the first step, every new token is cheap because it computes only its own Key and Value and reuses the saved ones for everything before it. It's one of those pieces of behind-the-scenes machinery that people who build with AI rely on every day without ever being asked to explain it. What Generation...

A language model writes text one token at a time (a token is a word or a piece of a word). At each step it gives every possible next token a probability, then picks one at random, weighted by those probabilities. That weighted random pick is called sampling. Top-k sampling limits the pick to a fixed number of the highest-probability tokens. That's predictable, but it ignores whether the model is sure or spread across many options. Top-p sampling, also called nucleus sampling, keeps the smallest group of top tokens whose probabilities add up to at least p , where p is a number you choose, such as 0.90 for 90%. ("Nucleus" just means the core group of likely tokens.) When one or two tokens dominate, that group is tiny. When the probabilities are spread out, the group widens. This post shows it on a small made-up vocabulary of 12 candidate answers, with code you can rerun. The Rule Sort the probabilities from highest to lowest and add them up in or...