Why an AI Picks the Wrong Sentence Even When It Knows the Right One

Give a small, free language model (an AI that writes text by predicting one small piece of text at a time) the start of a sentence, called the prompt: "In 1969, humans first". Then continue it two different ways, called decoding strategies (the step where the model's guesses about what comes next get turned into actual words): greedy (always take the single likeliest next word) and beam search (keep several candidate sentences going at once, like exploring a few paths instead of betting on one). Same model, same prompt, two different endings:

Greedy:  In 1969, humans first used the Internet to communicate. In 2000, the Internet
Beam:    In 1969, humans first landed a man on the moon. In 2009, the

The first one is fluent, and factually wrong. The model's own scoring (explained below) rates the whole second continuation about 950 times more likely than the first, and greedy decoding didn't find it here, because greedy decoding picks the most probable next word, not the most probable sentence. One caution up front: the model's score measures what the model finds likely, not what is true. This post runs both strategies for real, on a real model, so every number below is something you can rerun yourself.

What Greedy Decoding Does, One Word at a Time

At every step, a language model assigns each possible next small piece of text (called a token, often a whole word or part of one) a probability, a number from 0% to 100% for how likely it is to come next. Greedy decoding always takes the single highest-probability token, writes it down, and moves on to the next step. It never goes back. Isn't picking the best word at every step the same as picking the best sentence? Not quite, because the probability of a whole sentence is every step's probability multiplied together (like flipping heads three times in a row: 50% × 50% × 50%), and a word that looks best now can lead into a continuation whose later steps are collectively less likely.

The model here is Qwen2.5-0.5B-Instruct, a small open model from Alibaba (494 million parameters, the adjustable numbers that store what it has learned), used for inference only, meaning it just writes text, and nothing is trained or changed. The prompt is fed in as plain text to be continued, not as a chat message. To compare two sentences fairly, both strategies are limited to the same length of exactly 15 tokens.

Where Greedy Goes Wrong

Here are the model's top candidates for the very first word after "In 1969, humans first" (the leading space is part of each token):

' used'          7.84%
' successfully'  7.65%
' landed'        6.59%
' discovered'    5.92%
' achieved'      4.62%

A nearly flat spread across several options: greedy takes ' used' because it is barely ahead, 0.19 percentage points over ' successfully' and 1.25 over ' landed' (7.84% vs. 6.59% is a difference of 1.25 percentage points: the plain gap between two percentages). Is a lead that small worth worrying about? It would be, if the first word were the whole story, but it isn't. Look at what each choice leads into at the very next step (each percentage is the chance the model gives its best next word, assuming that first word was chosen):

after ' used':    best next word ' the'  17.87%
after ' landed':  best next word ' a'    38.32%

The slightly worse first word opens up a far more confident path: after ' landed' the model is more than twice as sure of its next word. Greedy can't see that, because at step one it only looks at step one. These two steps show where the paths split; most of the difference between the two sentences' scores (explained below) builds up over the remaining tokens, which this post doesn't break down token by token.

Beam Search: Keep Several Candidates Alive

Beam search fixes this by keeping the top few unfinished sentences (a run of tokens is called a sequence) in play at every step instead of committing to one. The number it keeps is the beam width. With a width of 3 (used for this walkthrough; the code below uses 4, which gives the same final sentence), it starts by keeping the three best first words, extends each of them with its three best next words, and then keeps only the three best two-word sequences out of those nine candidates, ranked by their combined probability (the two steps' probabilities multiplied). Each row below is one two-word continuation:

KEPT     ' landed a'                 2.53%
KEPT     ' landed on'                2.18%
KEPT     ' successfully cultivated'  2.00%
dropped  ' used the'                 1.40%
dropped  ' successfully landed'      1.00%
dropped  ' used a'                   0.67%
dropped  ' used computers'           0.59%
dropped  ' landed the'               0.44%
dropped  ' successfully developed'   0.37%

The path greedy committed to is already gone: every ' used ...' candidate was dropped, and the best of them, ' used the', ranks fourth, outside the three kept. Doesn't keeping several candidates make the work explode? No, because the width puts a hard cap on it: at most that many sequences are carried forward at each step. In the Python library transformers from Hugging Face (a company that shares open AI models and tools), it is a single argument:

outputs = model.generate(
    **inputs,              # the tokenized prompt
    max_new_tokens=15,     # stop after 15 tokens
    min_new_tokens=15,     # ...and don't stop earlier
    do_sample=False,       # no randomness: always pick by score
    repetition_penalty=1.0,  # 1.0 = no repeat penalty (this model's default is 1.1)
    num_beams=4,           # 1 = greedy; 4 = keep 4 candidates in play
    early_stopping=False,  # beam-only: don't stop before 15 tokens
    length_penalty=0.0,    # beam-only: no bonus or penalty for sentence length
)

The two settings do_sample=False and repetition_penalty=1.0 matter more than they look: this model's default settings turn random sampling on and add a repetition penalty (a rule that discourages reusing words) of 1.1, so left alone, even num_beams=1 would not be true greedy. The notebook sets a few more arguments in the same spirit (switching off the model's other sampling defaults, plus some housekeeping), and loads the model and its tokenizer (the tool that turns text into tokens and back) first. The fixed 15-token length and the zero length penalty (an optional rule that favors shorter or longer sentences) keep the scores directly comparable.

With a beam width of 4 the result is the moon landing sentence, at a score of -11.754 against greedy's -18.610. Scores are negative because they are logs of probabilities below 100%, so closer to zero means more likely.

The score is the log of every step's probability, added up (the log turns a long product of tiny numbers into a manageable sum), and the notebook computes it by asking the model to rate each finished sentence as a whole. Since exp is the reverse of log, the gap of 6.857 (computed before rounding the two scores) means exp(6.857), roughly 950 times more likely. Running the notebook twice gave identical output for both strategies.

Where Beam Search Still Falls Short

Then why not set the width to something huge and be done? Because beam search checks only a limited number of candidates, not every possible sentence, and the width has to be wide enough to catch the path you want. A width of 1 means keeping one candidate, which is exactly greedy. Sweeping the width on this same prompt:

width 1 (greedy):  -18.610   ...humans first used the Internet to communicate...
width 2:           -14.984   ...humans first successfully cultivated the first genetically modified organism...
width 3:           -11.754   ...humans first landed a man on the moon...
width 4:           -11.754   (same)
width 8:           -11.754   (same)

Width 2 already beats greedy, but still misses the moon landing sentence, because ' landed' only ranked third at step one and a width of 2 drops it immediately. Width 3 is the first to keep it, and going wider added nothing on this prompt. This sweep only shows that width 2 is too narrow. In general a wider beam does more work per step (not timed here), and no width guarantees the best possible sentence: an even better one could exist that none of these widths found.

A better score is not the same as a true answer. The score measures how likely the model finds a sentence. Here the better-scoring sentence also happens to be the correct one, but that is a feature of this example, not a guarantee. These results also come from one prompt, a 15-token limit and no length penalty, so they show how the mechanism works, not that beam search is better in general.

Get the Code

The full setup, every number quoted above, and both charts: BeamSearch.ipynb. Model used: Qwen/Qwen2.5-0.5B-Instruct, via Hugging Face transformers 5.18.0 and PyTorch 2.14.1 (the numerical library underneath).

Summary

Greedy decoding picks the single most likely token at every step and never goes back, so a tiny lead at step one can lock it into a poor sentence: on "In 1969, humans first" it chose ' used' by a margin of 0.19 percentage points and ended up on a sentence scoring -18.610 (closer to zero is better).

Beam search keeps several candidate sentences in play and ranks them by combined probability, so with a beam width of 4 it found the moon landing sentence at -11.754, about 950 times more likely under the model.

It is a limited search, not a guarantee: a width of 2 still missed it, and a better model score is a statement about the model's probabilities, not about truth.