Why Your RAG Pipeline Retrieves the Wrong Chunk?

I asked a support bot: "What happens to my data if I downgrade my plan mid-cycle?" It answered confidently. It also answered a completely different question — about deleted file recovery windows, not downgrades.

The model wasn't broken. The retrieval — the step that finds and hands over the right piece of text — was broken instead. So what caused that? Not the embedding model (the piece that turns text into comparable numbers), not the LLM (the model generating the actual answer), and not a bug in the prompt. It was broken because of a decision made before any of that — how the source document was cut into chunks.

This is the failure mode nobody warns you about when you first build a RAG pipeline, because it doesn't look like a bug. It looks like the system working — just working on the wrong piece of text.

The Default Everyone Reaches For

Retrieval-Augmented Generation (RAG) has one job: find the relevant piece of a document, then hand it to the model as context. Before retrieval can happen, the document has to be split into chunks. The simplest possible way to do that is fixed-size chunking — cut every N characters and move on.

def naive_chunk(text: str, chunk_size: int = 250, overlap: int = 0):
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + chunk_size, len(text))
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Fifteen lines. No dependencies. It runs instantly on any document. So what's not to like? It is also the reason the downgrade question got answered with a deletion policy.

What "Fixed-Size" Actually Ignores

Cutting a document every 250 characters has no concept of a sentence, a section heading, or a table row. It knows exactly one thing: character count.

So when a policy document has a "Downgrades" section that ends 20 characters into a chunk boundary, the rest of that explanation gets severed and glued onto whatever text happens to come next — often an unrelated section, like refund policy or account deletion. The chunk that gets embedded and stored no longer represents one idea. It represents the tail of one idea and the head of another, stitched together by character count.

Isn't a slightly-off chunk still better than nothing, though? The chunk is not wrong text. It is incoherent text — and a retriever can only be as good as the chunks it has to choose from.

Watching the Failure Happen

Take a toy support document, chunk it at 250 characters, and ask a real question:

query = "What happens to my data if I downgrade my plan mid-cycle?"
top_chunk = retrieve(query, chunks, vectorize_fn=tfidf_vectorize, top_k=1)[0][2]

Here's the actual chunk that comes back top-ranked (score 0.2421):

mid-cycle downgrades.

## Data Retention and Deletion

When a file is deleted by a user, it is not immediately erased. Aperture moves
deleted files into a recovery buffer where they remain restorable by the account
owner. This recovery window lasts 30 days from the moment of deletion...

The document says it plainly: partial refunds are not offered for mid-cycle downgrades. So why does the retriever miss something that direct? Because the fixed-size cut slices that exact sentence in half, right between "for" and "mid-cycle downgrades." The chunk that scores highest for this query is the one holding the second half — the trailing words "mid-cycle downgrades," now glued to an unrelated section about deleted-file recovery. It still ranks first under TF-IDF's keyword-overlap scoring (a simple way to measure how many important words two pieces of text share), because it still contains words from the query. It just doesn't contain the sentence that answers it.

Hand that chunk to the LLM as its only context, and — especially without an explicit instruction to admit uncertainty — it's far more likely to answer fluently and confidently wrong than to catch the gap. Not because the model hallucinated — because it was never given the right piece of the document to begin with.

A Second Failure, Even When the Right Chunk Is There

Fix the retrieval problem and the story's over, right? Not quite. Fixed-size chunking has a second cost that shows up even after retrieval gets the right chunk. Bury it in the middle of five plausible-looking chunks — pricing tables, refund intros, account security — instead of handing it over alone, and generation quality can drop even though the right information is technically present in the context window (the batch of text you actually hand the model before it answers).

This is the "lost in the middle" effect: the position of a chunk inside the context, not just its presence, affects whether the model actually uses it. Bad chunking doesn't just risk retrieving the wrong text. It increases how often the right text gets surrounded by noise.

The Fix Is Not a Smarter Model

Wouldn't a bigger LLM or a fancier embedding model just fix this? It's tempting to reach for that first, but neither one touches the actual problem here. If the chunk handed to the model doesn't contain a coherent idea, no amount of model capability recovers the meaning that was severed at the character boundary.

The real fix starts one step earlier than the retriever — chunking by structure (sentences, sections, semantic boundaries — natural shifts from one idea to the next) instead of by character count, so that what gets embedded and compared is one complete thought, not a fragment of two.

Get the Code

The full local demo — naive chunker, retriever, and a locally-run LLM answering both a retrieval-failure query and a lost-in-the-middle query, no API key required: naive_chunker.py and generation_demo.py.

Summary

Fixed-size chunking cuts a document every N characters with no regard for sentences, sections, or tables. That produces chunks that are incoherent rather than wrong — the tail of one idea glued to the head of another — and a retriever can only be as good as the chunks it has to choose from. Even when the right chunk exists, burying it among plausible-looking distractors can still hurt generation quality (the "lost in the middle" effect): position inside the context matters, not just presence. Neither a bigger LLM nor a fancier embedding model fixes this, because the problem happens one step earlier, at the chunk boundary. The real fix is chunking by structure — sentences, sections, semantic boundaries — instead of by character count.

Comments

Post a Comment

Popular posts from this blog

Working with Multi-Member Physical Files - IBM i

Multi-member Physical Files At times, Multi-member physical files are very helpful in IBM i. A physical file is a file that stores data in a specific format. A physical file can have multiple members, each of which is act as a separate data set within the file.  This allows us to store different types of data (format of the file remains same) or data for different purposes within the same file. Creating a Multi-member Physical File Create a physical file (PF) with single member (default) and add an addition member after PF is created. A physical file is created by using CRTPF command. CRTPF FILE(<Library>/<File>) MBR(<Member Name>) By default, Member is created with the same name as the file name. If a specific name is required, member name can be added as well.  CRTPF FILE(LIBRARY/FILE) RCDLEN(10) MBR(MEMBER1) TEXT('Temporary File') For the simplicity, I have created the flat file by directly mentioning the record length (RCDLEN) as 10. We would usually be c...
Read more

Gradient Descent: The Update Rule That Trains Every Model

Image
Every model — a neural network, a regression line, anything that learns from data — is really just a set of internal numbers called parameters, plus a single score called the loss that measures how wrong its current predictions are. Every parameter has a slope: nudge it up or down, and the loss changes. Stack all those slopes into one vector and you get the Gradient . It always points uphill — toward the steepest way to make the loss worse — whether you have 2 parameters or 175 billion. But knowing the uphill direction isn't the same as knowing what to do with it. The gradient only points up. So what good is that to a model trying to reach the bottom? Gradient Descent takes "here is uphill" and turns it into "here is your next step downhill." That's the whole idea. Every neural network, every regression model, every fine-tuning run learns using exactly this. The Update Rule Here is the entire algorithm, in one line: θ(...
Read more

Least Squares Approximation Explained (The Engine of Linear Regression)

Image
The Math Behind "Best Fit": A Guide to Least Squares Approximation In a perfect mathematical world, every system of equations has a clean solution. But the real world is not perfect. Real-world data is noisy, inconsistent, and often creates mathematical contradictions. So, what do we do when our system of equations `Ax=b` is unsolvable? Do we simply give up? No. We find the best possible compromise . The way we define "best" is by finding a solution that makes the sum of the squared errors as small as possible. This powerful technique is the Least Squares Approximation , and it is the mathematical engine behind one of the most fundamental algorithms in machine learning: Linear Regression. In this post, we'll decode the beautiful geometry that allows us to find this "best fit" solution, even when a perfect one doesn't exist. Watch the video for the full visual story, then scroll down for the detailed derivation and c...
Read more