Let’s say you have a pipeline that extracts terms from a stack of asset purchase agreements, and we need to find the applicable interest rate from each contract. In one agreement, section 3.2 defined the term “Applicable Rate,” four paragraphs before the payment clause that used it. The chunker split the document by token count, and the two paragraphs landed in separate chunks. The model reading the payment clause had no idea a rate had been defined anywhere. Here’s where you can see hallucinations creep in as the model fills in a percentage that sounded exactly like the kind of number that clause would have used. It was wrong but there is no error to catch. The problem starts when a piece of code decides where to cut the document.
When you know the structure of your document, a chunk boundary is a fact you can get from the document itself, not something you should ask a model to guess. This isn’t mainly because it produces fewer wrong answers (though it does). It’s because a fact you read off the page can be tested in a way a model’s guess never can. This same certainty that makes it testable also makes it fail worse the moment a document breaks the rule the parser assumed. Let’s go through both halves.
Most chunking bugs are boundary bugs, not model bugs
Fixed-size chunking treats a document as a flat stream of characters or tokens and cuts it every few hundred of them, regardless of what’s actually on the page. It doesn’t know a table has rows, or that a contract defines a term in one section and uses it three sections later. A comparison table is split down the middle of a row it gives the model half a row and a column header that belongs elsewhere. If a defined term is separated from the clause that invokes it, it passes the model a phrase that looks complete but isn’t. It does the thing language models do with an incomplete context: produce the most likely continuation available from what’s there. Often what’s there is enough to sound right without being right.
This is worth thinking through, because it changes what “hallucination” usually means in an extraction pipeline. The common result is a model inventing a fact from nothing. The model is doing its job correctly on a broken input. The model’s tendency to seem confident hides the real defect, which you missed because it lives in the chunking step.
The document already tells you where to cut if you know its schema
If a document lists risk factors under “Item 1A” and the management’s discussion is under “Item 7,” you don’t need a model to find that boundary. You read the document headings. Contracts number their clauses and define terms in a dedicated section before using them. You can parse the numbering and keep every definition bound to its own section, instead of letting proximity in the token stream decide what counts as nearby. The boundary becomes deterministic.
Compare that to the two places people look when a document doesn’t come with an obvious structure to parse. Semantic chunking uses embedding similarity to guess where one topic ends and another begins. That’s a real improvement over blind token counts, but it’s still a guess. It’s scored by how similar two pieces of text are to each other rather than by anything the document really said. LLM-driven chunking goes further and asks a model to decide the cut points directly. That’s the most expensive option and the least predictable one, since a generation call can return a different boundary on a rerun against the identical document.
Looking at this, we can see a pattern. Each step costs more and offers less certainty than the one before it. That’s an argument for using the cheapest, most certain option the document supports, and using the next one only when the document doesn’t declare its own structure. Most teams tend to use the sophisticated option first, because it feels like the technically impressive choice. That instinct leads to worse results when the schema was sitting there the whole time.
Only the deterministic layer can be tested
The important part isn’t about which method finds better boundaries on average. It’s about what kind of claim you can make afterward.
If you write a parser for a known schema, you can write a test for it. Feed it a document, declare that the Item 1A section starts at this heading and ends at the next one. If you run that test a thousand times, it passes or fails the same way every time. That has nothing to do with the chunker being good. It’s a property of the chunker being deterministic, which is a much stronger claim.
Try writing that same test for a semantic chunker or a model-driven one. You can’t, not in the same sense. The boundary decision is itself a probability, so the best you can do is run an eval against a sample of documents and report a percentage that came out right. That percentage is a fact about your system in aggregate. It isn’t a fact about the particular document your pipeline just processed. When someone asks why the model pulled the wrong clause from that particular contract, “our eval showed ninety-four percent accuracy calculated over a thousand samples” isn’t the same kind of answer as “here’s the test that shows exactly why this boundary was drawn where it was.” We tend to use the model or the embedding call because it feels like the more sophisticated engineering decision. In the narrow case where the schema is actually knowable, the more sophisticated decision is to write less clever code and get a deterministic result.
What deterministic chunking fixes
Deterministic chunking fixes one failure mode: fabrication caused by a boundary that separated a fact from the context that would have qualified it. Or one that fused two records into a single chunk so the model attributes a value to the wrong entity. That’s the real failure, and fixing it justifies the engineering effort.
One other failure mode is that a model can answer confidently when it didn’t find the right chunk. A model can invent an answer under genuine uncertainty, because the prompt didn’t give it permission to say “not found.” Neither of those is a chunking problem. This is a grounding issue as I discussed in an earlier post. We can’t call both of these “hallucination,” and claiming one technique fixes hallucination broadly.
Deterministic splitting fails in different ways
If the document you pass to the deterministic parser is in the format it is built for, it will give the right result every time. Give it a document formatted differently, and you don’t see any errors; it just runs and gives you the wrong result. Those boundaries are wrong, and the pipeline doesn’t know it. It did exactly what it was built to do.
Fixed-size chunking with some overlap between chunks is worse on average, but it fails more gracefully. The overlap means a badly placed cut probably still leaves the fact you needed sitting in an adjacent chunk. Redundancy hedges against a bad boundary. A deterministic parser doesn’t give you that hedge unless you deliberately build one, because the certainty it gives on the documents it was designed for hides the cases where the format is different.
That’s the actual tradeoff: you’re trading graceful degradation on the rare malformed input for exact correctness on the well-formed ones. Whether that trade is worth it depends on something I can’t tell you. How often your real documents are going to violate the assumption your parser was built on, and whether anything downstream would ever catch it if they did, is something you need to validate for yourself.
That last question is worth digging into more than I can do in one post, because it points at a problem we haven’t talked about yet: knowing what category of document you’re looking at before you ever choose a parser. Everything above assumes you already know a document is a 10-K, or a specific contract template, so you can hand it to the right code. In an organization handling more than one kind of document, you can’t assume that. What happens when the mistake wasn’t the boundary inside the document, but the decision about which parser to run before anyone opened it? Stay tuned for my next post, which will cover exactly that.