Let’s say you have a pipeline that extracts terms from a stack of asset purchase agreements, and we need to find the applicable interest rate from each contract. In one agreement, section 3.2 defined the term “Applicable Rate,” four paragraphs before the payment clause that used it. The chunker split the document by token count, and the two paragraphs landed in separate chunks. The model reading the payment clause had no idea a rate had been defined anywhere. Here’s where you can see hallucinations creep in as the model fills in a percentage that sounded exactly like the kind of number that clause would have used. It was wrong but there is no error to catch. The problem starts when a piece of code decides where to cut the document.

When you know the structure of your document, a chunk boundary is a fact you can get from the document itself, not something you should ask a model to guess. This isn’t mainly because it produces fewer wrong answers (though it does). It’s because a fact you read off the page can be tested in a way a model’s guess never can. This same certainty that makes it testable also makes it fail worse the moment a document breaks the rule the parser assumed. Let’s go through both halves.

Most chunking bugs are boundary bugs, not model bugs

Fixed-size chunking treats a document as a flat stream of characters or tokens and cuts it every few hundred of them, regardless of what’s actually on the page. It doesn’t know a table has rows, or that a contract defines a term in one section and uses it three sections later. A comparison table is split down the middle of a row it gives the model half a row and a column header that belongs elsewhere. If a defined term is separated from the clause that invokes it, it passes the model a phrase that looks complete but isn’t. It does the thing language models do with an incomplete context: produce the most likely continuation available from what’s there. Often what’s there is enough to sound right without being right.

This is worth thinking through, because it changes what “hallucination” usually means in an extraction pipeline. The common result is a model inventing a fact from nothing. The model is doing its job correctly on a broken input. The model’s tendency to seem confident hides the real defect, which you missed because it lives in the chunking step.

The document already tells you where to cut if you know its schema

If a document lists risk factors under “Item 1A” and the management’s discussion is under “Item 7,” you don’t need a model to find that boundary. You read the document headings. Contracts number their clauses and define terms in a dedicated section before using them. You can parse the numbering and keep every definition bound to its own section, instead of letting proximity in the token stream decide what counts as nearby. The boundary becomes deterministic.

Compare that to the two places people look when a document doesn’t come with an obvious structure to parse. Semantic chunking uses embedding similarity to guess where one topic ends and another begins. That’s a real improvement over blind token counts, but it’s still a guess. It’s scored by how similar two pieces of text are to each other rather than by anything the document really said. LLM-driven chunking goes further and asks a model to decide the cut points directly. That’s the most expensive option and the least predictable one, since a generation call can return a different boundary on a rerun against the identical document.

Looking at this, we can see a pattern. Each step costs more and offers less certainty than the one before it. That’s an argument for using the cheapest, most certain option the document supports, and using the next one only when the document doesn’t declare its own structure. Most teams tend to use the sophisticated option first, because it feels like the technically impressive choice. That instinct leads to worse results when the schema was sitting there the whole time.

Only the deterministic layer can be tested

The important part isn’t about which method finds better boundaries on average. It’s about what kind of claim you can make afterward.

If you write a parser for a known schema, you can write a test for it. Feed it a document, declare that the Item 1A section starts at this heading and ends at the next one. If you run that test a thousand times, it passes or fails the same way every time. That has nothing to do with the chunker being good. It’s a property of the chunker being deterministic, which is a much stronger claim.

Try writing that same test for a semantic chunker or a model-driven one. You can’t, not in the same sense. The boundary decision is itself a probability, so the best you can do is run an eval against a sample of documents and report a percentage that came out right. That percentage is a fact about your system in aggregate. It isn’t a fact about the particular document your pipeline just processed. When someone asks why the model pulled the wrong clause from that particular contract, “our eval showed ninety-four percent accuracy calculated over a thousand samples” isn’t the same kind of answer as “here’s the test that shows exactly why this boundary was drawn where it was.” We tend to use the model or the embedding call because it feels like the more sophisticated engineering decision. In the narrow case where the schema is actually knowable, the more sophisticated decision is to write less clever code and get a deterministic result.

What deterministic chunking fixes

Deterministic chunking fixes one failure mode: fabrication caused by a boundary that separated a fact from the context that would have qualified it. Or one that fused two records into a single chunk so the model attributes a value to the wrong entity. That’s the real failure, and fixing it justifies the engineering effort.

One other failure mode is that a model can answer confidently when it didn’t find the right chunk. A model can invent an answer under genuine uncertainty, because the prompt didn’t give it permission to say “not found.” Neither of those is a chunking problem. This is a grounding issue as I discussed in an earlier post. We can’t call both of these “hallucination,” and claiming one technique fixes hallucination broadly.

Deterministic splitting fails in different ways

If the document you pass to the deterministic parser is in the format it is built for, it will give the right result every time. Give it a document formatted differently, and you don’t see any errors; it just runs and gives you the wrong result. Those boundaries are wrong, and the pipeline doesn’t know it. It did exactly what it was built to do.

Fixed-size chunking with some overlap between chunks is worse on average, but it fails more gracefully. The overlap means a badly placed cut probably still leaves the fact you needed sitting in an adjacent chunk. Redundancy hedges against a bad boundary. A deterministic parser doesn’t give you that hedge unless you deliberately build one, because the certainty it gives on the documents it was designed for hides the cases where the format is different.

That’s the actual tradeoff: you’re trading graceful degradation on the rare malformed input for exact correctness on the well-formed ones. Whether that trade is worth it depends on something I can’t tell you. How often your real documents are going to violate the assumption your parser was built on, and whether anything downstream would ever catch it if they did, is something you need to validate for yourself.

That last question is worth digging into more than I can do in one post, because it points at a problem we haven’t talked about yet: knowing what category of document you’re looking at before you ever choose a parser. Everything above assumes you already know a document is a 10-K, or a specific contract template, so you can hand it to the right code. In an organization handling more than one kind of document, you can’t assume that. What happens when the mistake wasn’t the boundary inside the document, but the decision about which parser to run before anyone opened it? Stay tuned for my next post, which will cover exactly that.

Hallucinations can depend on how you ask the question. If you make a false assertion in your prompt and say that it came from someone else in your prompt, the LLM is more likely to give you a correct answer (correcting for the false assertion) than if you say the assertion came from yourself. With GPT 4o, you get the right answer 98.2 percent of the time if you say the assertion was from someone else. If you say it is from you, the chance drops to 64.4%. Other models show similar drop-offs.

This is a hallucination failure mode that has nothing to do with context length, retrieval quality, or model size. It’s about what happens the moment a person is in the loop and sounds confident.

It’s not who’s asking, it’s how

The first reaction is usually to call this sycophancy and move on, but there is more to it than that. A controlled study varying how the exact same false claim gets phrased found that sycophancy is substantially higher when the input is a declarative statement than when it’s phrased as a genuine question. “Is it true that X” produces a different, more accurate response than “X is true, right.” On top of that, sycophancy rises the more certain the statement sounds. A plain claim produces less capitulation than a stated belief, which produces less than a flat conviction. Framing the claim as the user’s own perspective versus a third party’s matters too, but it’s a smaller effect than whether the sentence is a question at all.

That order matters if you’re trying to fix this rather than just measure it. The instinct is to focus on stripping first-person language out of prompts. The bigger factor is further upstream: whether the input reads as a settled claim or an open one.

It’s a mechanism, not confusion

The standard explanation is training incentive: reinforcement learning from human feedback rewards responses that people rate highly, people rate agreement highly, and a model optimized against that signal learns to agree with false premises the same way it learns to agree with true ones. That’s why this shows up in every model trained this way, rather than being a quirk of one lab’s approach.

It’s not the whole mechanism, though. Newer work traces the actual moment of failure inside the model: a late-layer shift in output preference, followed by a deeper change in how the claim gets represented internally. Here’s the part worth sitting with. Framing a false claim as coming from an expert doesn’t reduce the sycophancy. The model isn’t weighing “an expert said this” as evidence, because it isn’t representing “expert” as a category that should change its confidence at all. What moves the needle is grammar and certainty, a declarative sentence stated with conviction, not credentials. A model will hold its ground against a stated fact from nobody in particular faster than it holds its ground against the same fact stated with confidence by the person asking the question, and appealing to authority doesn’t change that, because authority was never what the model was tracking.

When grounding launders the lie

Picture a support or research assistant with retrieval attached, built to check its answers against real sources instead of just generating from memory. A user asks: “given that Company X acquired Company Y in 2021, explain how the fee structure changed.” The acquisition never happened. Watch what happens at each stage. The system’s query expansion step, the part that turns a user’s question into search terms, inherits the false premise as a fact and builds search queries around it. The retrieval step runs a semantic search against those biased terms and pulls back documents that share vocabulary with the query, not documents that confirm the acquisition, because there’s nothing to confirm, just fragments that look topically adjacent. The generation step performs selective synthesis: it picks the fragments that read as supportive out of what it retrieved and assembles an answer around them.

What comes out the other end is a wrong answer with citations attached. That’s a worse failure than an ungrounded hallucination because “grounded in retrieved sources” is the exact phrase a team uses internally to justify trusting a system’s output. Nobody double-checks the thing that’s supposed to be the double-check.

Gemini 3 Pro, the strongest model tested against this specific failure, still missed roughly one in five of these cases. GPT-5 missed nearly two in three. This is the actual, full answer to whether RAG fixes hallucination. It doesn’t just fail to fix sycophancy. Retrieval gives sycophancy a disguise, and the disguise is the exact credential your evaluation process was built to check for.

What actually reduces it

A system instruction telling the model not to be sycophantic is what everyone tries first, but there is a better way. The way that works is entirely upstream of generation: rewrite the user’s input before it reaches the model and convert declarative statements into neutral questions. Strip the certainty out of the sentence and the sycophancy drops with it, by more than an updated system prompt manages on its own.

This is a preprocessing fix, not a training fix and not a prompt-engineering fix layered on top of the same input. This is something a team can build without retraining anything or hoping a system prompt holds under pressure.

Here’s what you can do: Test the same false claim two ways, once as a declarative statement and once as a neutral question, and measure the accuracy gap between them. That gap is a hallucination rate you’re not currently tracking. If your system does retrieval, construct the false-premise-plus-retrieval case specifically, since it’s the one that looks safest on a dashboard but isn’t. It’s the one most eval suites never build because it requires deliberately writing a query with a wrong premise baked in. Consider an input-reframing step ahead of generation anywhere a user is likely to start already convinced of something: support intake, medical or legal triage, anywhere the theory shows up before the facts do.

If a system can be talked into fabricating support for a claim it would have rejected had you simply asked it neutrally, “grounded” no longer means “checked against reality” and starts meaning “found something adjacent enough to cite.” That’s a harder problem than an ungrounded hallucination, because it looks like the evidence that was supposed to prevent it. Building an eval suite for a failure that only shows up when a user is confidently wrong is rare. Are you already taking this into account in your evaluation tests?

Sources

• Stanford HAI (April 2026). AI Index Report 2026. hai.stanford.edu — confirm the underlying benchmark this stat traces to before publishing, see outline notes

• Dubois, M., Ududec, C., Summerfield, C., Luettgau, L. (2026). Ask Don’t Tell: Reducing Sycophancy in Large Language Models. arxiv.org/abs/2602.23971

• When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models. arxiv.org/abs/2508.02087

• Sharma, M. et al. Towards Understanding Sycophancy in Language Models. arxiv.org/abs/2310.13548

• Suzgun, M. et al. Evaluating Commercial AI Chatbots as News Intermediaries. arxiv.org/abs/2605.22785

LLMs hallucinate. It’s just the way they are. This is obviously a problem for systems that need dependable results. The first reaction is to limit the LLM to your own data, AKA Retrieval Augmented Generation (RAG). Unfortunately, it’s not that simple. A model can find the exact right paragraph in a fifty-page document and still invent a different answer. Finding information and returning the right answer are not the same thing. Even the best-performing model still fabricates it’s share of its answers when the correct source is sitting right in front of it. RAG doesn’t fix hallucination. It changes which hallucinations you get.

Hallucination is at least three distinct failure modes with different causes, and fixing a solution means knowing which one you’re actually measuring. This post covers the two computational ones. The third, what happens when a confident user pushes back, is interactional rather than architectural, and I’ll talk about it in another post.

Hallucination is a multi-headed beast

Most people think about hallucination as a single dial. Turn the model’s confidence down and hallucinations go down with it. There are two main reasons that doesn’t work, Fabrication and Grounding.

Fabrication is inventing a fact that exists nowhere in the source or the training data. Grounding failure is different: the right information is sitting in the context window and the model fails to find or use it.

These two have different causes and different rates across models so you can’t solve them with one approach. If you increase your retrieval quality you can fix grounding failure while doing nothing for fabrication. A model can be excellent at finding the right passage and still invent the answer to a question that the underlying data never addresses.

Model family predicts fabrication resistance better than model size does. Qwen3-Next-80B-A3B, a mixture-of-experts model with only 3 billion active parameters performs better at long context retrieval than dense models many times its total size, including Llama 3.1’s 405-billion-parameter version.

Grounding and truth-telling are different

Let’s take a look at an example fifty-page vendor contract. Now let’s ask it a question that’s not in the contract, something about a fee structure that isn’t in there. A model that’s excellent at the finding existing text can still confidently invent an answer when there is no relevant text. Finding information and refusing to invent it when there’s nothing to find are not the same.

The problem gets worse when you start looking at cross document aggregation. Synthesizing an answer from multiple documents requires the model to pay attention to parts of multiple, disjointed documents at the same time. This is the more common use case for RAG. A vector search will return chunks from multiple documents and the model has to reason across all of them. Assuming you are evaluating retrieval responses for accuracy (and you should), you need to make sure you are evaluating multi document retrievals and not just looking at single documents

This is also why the standard retrieval benchmarks (like RULER) that measure whether a model can find a needle in a long document, don’t tell you anything about fabrication rate. They score whether the model finds the right answer when one exists, at a single-lookup level of difficulty. If your evaluation suite is built the same way, you aren’t looking at what happens when the answer isn’t there or when it takes real synthesis to assemble.

Fabrication gets worse with longer contexts. The 172B study found the best model fabricating at roughly 1 percent of answers at 32,000 tokens, and that rate roughly triples by 128,000 tokens, crossing 10 percent for every model tested at 200,000 tokens. GLM 4.6, a dense model, fabricates on close to three-quarters of unanswerable questions once you reach 200,000 tokens. This is due to the way the model works. Dense models attend across an enormous key-value cache as context grows, and the signal any single fact contributes gets diluted alongside everything else. Mixture-of-experts models route each token through a small subset of specialized parameters instead of spreading attention across the whole cache, and they degrade more gracefully as a direct consequence. That’s why a 3-billion-active-parameter MoE model can perform better at long context better than a dense model with a hundred times the active parameters. Model family beating model size isn’t a training artifact. It’s about the token routing.

There’s a reason why fabrication doesn’t improve much no matter which technique you throw at it. A recent impossibility proof, working from mechanism design theory and the actual math of how a transformer aggregates competing signals, shows that no inference mechanism, RAG included, can, at the same time, be fully truthful, fully information-preserving, complete, and constrained to what it actually knows. You can trade between those four properties but you can’t eliminate all of them at once. That’s not a limitation of this generation of models. It’s closer to a conservation law.

Temperature is a tradeoff, not a dial

The common belief is that lower temperature means fewer hallucinations. This depends entirely on which hallucination you’re worried about.

For deterministic output, setting the temperature to zero, gives the best overall accuracy in roughly 60 percent of cases in the 172B study, so for simple factual retrieval, turning the dial down usually helps. But higher temperature reduces fabrication for most models, probably because it raises the odds of an “I don’t know” instead of a confident invented answer. And temperature zero comes leads to coherence collapse. Infinite generation loops happened at roughly 48 times the rate at temperature zero compared to temperature one.

The type of use case makes a big difference. Code generation research finds a clear increase in hallucination as temperature rises, which is the opposite for document QA. There’s no single correct temperature setting for “reducing hallucinations.” There’s a setting that trades retrieval accuracy against fabrication confidence against output coherence and the answer depends on what you’re building.

What can you do?

Prompting is not the fix people think it is. It can help with grounding but does almost nothing for fabrication.

Grounding failure is a retrieval and attention problem, so it responds to changes in the pipeline. Split retrieval into two steps instead of one. Have the model pull literal quotes first, with no synthesis, then answer using only those quotes. Testing whether it found the passage separately from whether it correctly reasoned about the passage turns a single failure into two things you can actually measure. You should rerank after a vector search, since dense retrieval alone can often puts the correct chunk lower on the list of retrieved data. If the question requires combining data across documents, break it into sub-questions and answer each against its own retrieval before combining. Don’t hand the model five disjointed chunks and hope it does the proper cross-referencing in one pass.

Put your most likely chunk first or last in the context window. Lost-in-the-middle is a real risk, and it gets worse as context grows. Shrink your context wherever you can. As above, fabrication roughly triples between 32,000 and 128,000 tokens. Fewer, better-ranked chunks reduce grounding failure and fabrication risk at the same time. This is one of the only methods that affect both issues.

Fabrication is a different animal, and the impossibility proof above tells us why prompting can’t solve it. If no inference mechanism can be simultaneously truthful, information-preserving, complete, and constrained to what it actually knows, no system prompt is going to change that. It’s not a matter of finding the right wording. Research on citation generation shows this. Prompted citation generation doesn’t produce reliable citations, but models fine-tuned specifically on citation-grounding data do. An 8-billion parameter model trained this way beat GPT-4o on citation accuracy. Training closes the gap. Prompting narrows it, slightly, and only some of the time.

So the fixes that actually work on fabrication live outside the generation step. Generate the answer, then generate a separate set of verification questions about its own claims, and answer those independently against the source. Revise anything that doesn’t hold up. This catches a good chunk of fabrications, though it adds a second call and doesn’t help much with recognizing when a question has no answer at all. Force the model to cite a specific paragraph or span for each claim, then check in code, not in the model, whether that span actually exists and actually supports the claim. Don’t ask the model if it’s sure. Ask your code to check. Score each claim in an answer separately instead of scoring the whole answer for faithfulness, since a paragraph with two grounded claims and two invented ones will still score fine on an aggregate metric. If you have the volume, sample the same question multiple times at a higher temperature and check whether the answers agree. Disagreement is a cheap and reasonably reliable fabrication signal, since you already know higher temperature makes a model more willing to say it doesn’t know instead of inventing something.

None of this eliminates fabrication. Nothing does, and that’s the point of the proof. What it does is move the check from hoping the model is right to verifying it before the answer reaches anyone. That’s the only lever that actually exists.

Sources

• Roig, J.V. (March 2026). How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-billion-token study across 35 models. arxiv.org/abs/2603.08274

• Hsieh, C. et al. (2024). RULER benchmark. arxiv.org/abs/2404.06654

• Karpowicz, M.P. (2025). On the fundamental impossibility of hallucination control in LLMs. arxiv.org/abs/2506.06382

• Gramacki et al. (2024). Hallucination rate vs. temperature in code generation. researchgate.net/figure/Hallucination-rate-vs-temperature_fig3_381484725

Think about this scenario:

A permissions change to a project management tool passed every test in the suite. It read cleanly in review, one reviewer approving in under ten minutes. It shipped on a Tuesday. Three weeks later, someone on a design team noticed they could see tasks from a workspace they’d never been invited to, just by changing a number in a URL nobody had thought to protect. Nothing in the ticket was wrong. Nothing in the code was obviously wrong. The bug lived in the gap between what the spec said and what the spec assumed everyone already knew.

I think about software development now as a bookshelf. An AI system generates the books: the actual code. Two bookends hold that up. One is the work of defining what should be built, clearly enough that generation has something real to aim at. The other is verification, someone or something checking that what got built is what was meant. Between those two, the code itself has started to look like the interchangeable part. Which raises a question: if both bookends are solid, does it matter which model does the generating? Could a much cheaper model handle the writing, while the expensive reasoning goes into defining the problem and checking the answer?

I wanted this to be true. It’s a clean answer to a real cost problem. But it’s deeper than that. Tracing why opens a can or worms I wasn’t expecting.

Can we treat the middle as a commodity?

There’s an argument for the commodity-middle idea, that goes way back before the AI world. Most software defects are not about someone typing the wrong character. They come from building the right thing to the wrong understanding of the problem. That was true when humans wrote every line, and it’s still true: get the requirement right and the implementation mostly follows.

Now let’s look at the state of things today. Give a small, cheap model several attempts at a coding task, and select the one that actually passes a real test suite instead of whichever one looks best on inspection, and its output holds up against a single attempt from a model many times its size and cost. The difference between “generate correctly” and “generate something that clears a real check” is smaller than the gap in list price between models.

Put that together with a properly defined problem and you get the appealing version of the claim: the model generating your code doesn’t need to be smart. It needs supervision that’s smart, on both ends.

Where does this break?

A specification, however carefully written, can only resolve the ambiguity someone thought to write down. The permissions bug at the start of this post wasn’t a failure to follow the spec. The spec never said “a user from one workspace must not be able to see another workspace’s data by editing the URL,” because nobody writing that ticket thought they needed to say it. Tenant isolation was assumed the way most real requirements are assumed: so basic it never made it into the words.

A weaker model doesn’t fail by ignoring what you told it. It fails by making a locally reasonable choice in exactly the space you didn’t cover, and that space doesn’t shrink because you wrote a better ticket. It grows with how many other things the code touches: existing conventions, adjacent services, the parts of the system nobody writing the ticket had fully in view. Spec quality buys you less ambiguity. It doesn’t buy you a smaller blast radius, and those are not the same purchase.

Can a stronger model fix this?

The obvious fix is to keep the capable model on generation too, at least for anything that matters. That’s not wrong but it also is not a fix.

Take two coding tasks that look, on paper, like the same kind of problem: one where a model has to correctly size a memory buffer using a multiplication of two floating point numbers, and one where it just has to subtract a float from an integer. The second task succeeds around half the time. The first succeeds roughly once in seventy tries, and that gap holds regardless of which model is running it. Same type of task on the surface, very different reliability underneath, driven by a specific reasoning step buried inside the task rather than by anything visible in a spec.

A clearer version of this shows up somewhere you’d expect the opposite result. A comparison of models doing automated code review, the exact job you’d think rewards the most capable reasoner, found a cheaper model in the same family beating a more expensive one on precision, recall, and every other quality measure tested. What predicted the review quality wasn’t which model was reading the code. It was how much code sat in front of it. Past a certain size, accuracy degrades no matter who, or what, is doing the reading.

Neither result makes model choice irrelevant. It means the issue is not the ability of the model. It’s the type of task, and the size of what you’re asking it to hold in its head at once.

What decides whether a mistake survives?

So if it’s not the spec and it’s not the model tier, what is it? To answer that, we need to follow a mistake through a pipeline and watch where it either dies or takes root.

First: not everything that looks like a check is one. A compiler failing is a check. A test executing against real data is a check. A model reading its own output and pronouncing it correct is not a check, it’s an opinion coming from the same source that made the mistake in the first place. The code review finding is telling you this: an evaluator that’s just another model forming a judgment doesn’t introduce anything the generator didn’t already have. Nothing got tested against reality.

Second, a claim is not a fact. A pull request description that says “added workspace scoping” is a claim. A test that fails when workspace scoping is missing and passes when it’s present is a fact. The moment a pipeline lets the claim travel forward as though it were already verified, whether that’s a person skimming a confident summary or an automated step trusting a self-report, the system has stopped checking anything. It’s repeating itself with more confidence each time.

Third: how much can one wrong step cost before something catches it? A single-purpose change that’s wrong is cheap. Someone notices, it gets fixed. A change that touches five things at once, or an action that can’t be undone once it fires, a schema change against production, a message that’s already sent, is expensive regardless of who or what produced it. The size of a mistake you can survive has almost nothing to do with how the mistake got made.

There’s a fourth thing underneath all three: When a result depends on several decisions each being right, and each decision leans on the last one having gone well, the odds of the whole chain landing correctly fall off faster than the odds of any single step suggest. Worse, once an early step goes wrong, it doesn’t just add an error. It changes what looks reasonable for every decision after it. A team watching each step’s success rate in isolation, all comfortably above ninety percent, can still be running a pipeline that’s wrong more often than it’s right, because nobody is counting the chain.

Check facts, not opinions

Model capability changes how often a given step introduces a mistake. Everything in the last section changes whether that mistake turns into an actual problem: real checks instead of self-report, claims kept separate from facts, damage kept small enough to survive being wrong. Those are two different levers. A good spec pulls the first one by cutting ambiguity at the start. It doesn’t do anything for the second lever on its own.

So the corrected version of the original idea isn’t “define the problem well and the model doesn’t matter.” It’s this: what actually frees you to use a cheaper model is how much independent, reality-checking machinery you’ve built around it, not how well you wrote the ticket. That machinery is engineering work. It doesn’t show up for free because someone wrote a thorough spec, and it costs something to build whether or not you ever downgrade a model to take advantage of it.

Where this goes wrong

Five patterns thinking about, each one looks like discipline isn’t.

• Evaluator theater. A model reviews another model’s output, the two agree, and agreement gets treated as verification. It isn’t. Nothing independent of the generation process weighed in.

• Diff bloat. A change grows past the point where anyone or anything can hold the whole thing in mind at once, and review quality degrades no matter how the reviewer is staffed.

• Claim laundering. A model’s own account of what it did, in a commit message, a summary, a PR description, becomes the record of what happened instead of a claim that still needs checking.

• Uniform blast radius. A cosmetic change and an irreversible one move through the same lightweight process because the ticket read equally clearly for both, and clarity of intent gets confused with safety of consequence.

• Mistaking documentation for simplicity. A task can be specified in exhaustive detail and still contain one reasoning step that’s unreliable no matter how it’s approached. Thoroughness in the ticket doesn’t help with that.

The two most important fixes:

Of the patterns above, two are worth fixing before the rest, because they’re the cheapest to change and they patch the failure modes doing the most damage.

Fix claim laundering by giving every claim a place to point. Add one required field to the PR template: not a checkbox, a link. Every claim in the description, “handles workspace scoping,” “added the negative-case test,” has to point at the specific check that verifies it. No link, no merge. This sounds bureaucratic until you see what it actually removes: the ability for a well-written sentence to stand in for a verified fact just because it was well written. A model writing a confident PR description and a person writing one have the same problem here, and the fix doesn’t care which one wrote it.

Fix evaluator theater by decorrelating the check from the thing being checked. The reason a model reviewing a model’s output isn’t a real check is that both draw on the same patterns, the same blind spots, sometimes the same weights. Break that correlation instead of reasoning around it: route review to a different model family, a static analyzer, or a person who hasn’t read the original ticket. Anyone who saw the same framing the generator saw is prone to the same misreading the generator made. A reviewer who comes in cold, without that anchor, is actually checking something.

Neither of these touches the test suite or the spec process. Both change what counts as evidence at the two points where evidence was quietly being replaced with confidence.

What next?

Whether or not you make either change, there’s a test that tells you if you need to. Next time a change ships in under an hour, clean review, green tests, no follow-up, don’t take that as proof the process worked. Ask what actually caught the last real mistake this pipeline let through, a specific one, not a hypothetical. If the honest answer is a person’s gut feeling, or a model saying it looks fine, the pipeline has an opinion generator on both ends and a spec in the middle, which was never the design the bookshelf idea was describing. If the honest answer is a specific test, a specific check, a specific place where the system pushed back against a specific claim, you already know how much of this you can safely hand to a cheaper model. That’s the actual state of the pipeline, not the state the ticket and the green checkmark suggested it was in.

Your eval suite passed clean for three months on a case that used to be a real problem: dates in generated citations. Then a routing change in your LLM vendor’s systems moves your production traffic onto a different checkpoint. The same failure comes back in the place you stopped looking at. Your code didn’t change, your prompts didn’t change but now the result has changed.

The LLM never tells you it changed

LLMs change frequently in ways we can’t see. Weights update. Routing decisions send your query to a different checkpoint than it hit last week. Retrieval indices refresh. System prompts get tuned server-side, sometimes as an A/B test you were never told you were in. None of this shows up in the interface. The chat box looks the same whether or not the thing answering you is the same.

In some ways, this is similar to Google Search ranking. Google made roughly 4,725 changes to its ranking systems and interface in 2022 alone, something like thirteen a day. It confirms only a handful of those as named core updates each year. Everyone who’s used Google has felt the downstream effect. A site that used to rank well disappears. A type of query that used to return good results starts returning junk. Nobody’s ever told why or when. That’s the same silent-drift pattern, running for two decades on a product with billions of users. In SEO circles, blaming “the algorithm” has become a standing joke, not an actual diagnosis. It shows that the pattern is old and durable, not that anyone has solved it.

The model card won’t tell you

A first reaction might be that SaaS vendors change things all the time and we have learned to deal with it, why is this different? There are two main differences:

The first: vendors publish model cards and release notes, so isn’t the information available if you go looking for it? It is, but it’s the wrong granularity of information. A model card gives you aggregate statistics across broad benchmark categories. It doesn’t tell you whether the specific behavior you calibrated your trust against moved. Does it still handle ambiguous medical terminology in your discharge summaries the way it used to? Is the uncertainty phrasing in code review comments still what you learned to read? That’s the detail that matters, and model cards don’t give you that. You’d need the vendor to tell you about changes the vendor itself is probably not tracking at that granularity.

The second objection is stronger: every SaaS product updates silently, all the time, so why treat this as new? Three reasons compound here.

Amplitude. A UI update might move a button. A model update can shift qualitative reasoning behavior, the kind of change that decides whether an answer is trustworthy, not just where you click to get it.

Invisibility. A UI change is instantly identifiable the moment you see it. A change in how a model handles edge cases in your domain stays invisible until it bites you.

Entangled causation. When a website changes, you know it was the website. When your AI output shifts, you often can’t tell whether it was the model, a system prompt change, a routing decision, or your own prompting drifting over months.

Any one of these differences would be manageable on its own. Together they mean the ordinary “software changes, users adapt” story doesn’t transfer.

This already happened in public

In December 2023, a wave of ChatGPT users started reporting that GPT-4 had gotten lazier: shorter answers, more refusals, tasks half finished with an instruction to complete the rest yourself. The company said it hadn’t touched the model in nearly a month, called the behavior change unintentional, and conceded that model behavior can be unpredictable. A follow-up post went further. Differences in behavior can be subtle enough that only a fraction of prompts are affected. It can take a long time for anyone, customer or employee, to notice the pattern and track it down.

This is the important part: The vendor is telling you it may not reliably know when its own product changed, or why, or for whom. That’s not evasiveness. That’s the entangled causation problem. It’s playing out at the scale of a company with more visibility into its own system than any of its users will ever have. If OpenAI can’t always tell whether behavior actually shifted, individual vigilance on the user’s end is never going to catch it reliably.

The incentive runs the other way

There’s a reason no vendor ships a “your calibration may be stale” warning, and it isn’t only technical difficulty. Admitting that behavior drifts is a competitive liability. A vendor who discloses instability looks less reliable than a competitor who says nothing. That’s true even if the competitor is drifting just as much and simply isn’t telling you. The first company to be honest about this doesn’t get rewarded for honesty. It gets read as the shakier product. That’s not a reason to expect dishonesty from any specific vendor. It’s a reason to expect the whole category to under-disclose, regardless of any individual vendor’s intentions.

The metrics a vendor actually optimizes for compound the problem. Retention, usage volume, satisfaction scores: none of them reward closing your calibration gap. A prompt that asks users to reconsider how much they trust the product reads as friction, not as a feature. Even a vendor that wanted to build the right signal runs into the measurement problem the GPT-4 case exposed. Knowing whether a specific task-level behavior shifted for a specific kind of user is a hard, expensive, mostly unbuilt instrumentation problem. And software versioning itself was never designed for a system with no discrete release to hang a version number on.

Nothing tells you when to recalibrate

Software has version numbers for a specific reason: so a user knows the moment their mental model needs refreshing. Nothing plays that role for behavioral trust in a probabilistic system. There’s no signal that says the thing you calibrated your trust against three months ago is not quite the thing answering you now.

A team with enough discipline can build a piece of this themselves. Log the inputs, outputs, and model version for a canary set of prompts that map to the behaviors you depend on. Diff the outputs over time instead of scoring pass or fail. That catches the case where the qualitative shape of an answer shifts on a prompt you already knew to worry about.

It does not catch the failure from the opening of this piece, the one that comes back in the place you stopped looking. A canary set only tests what you thought to test. And when the diff does fire, you know something moved. You still don’t know whether it was the model, a system prompt, a routing decision, or your own prompting drifting over months.

Partial, effortful, incomplete detection is available to anyone willing to build it. A real recalibration signal, the kind a version number gives you automatically, is not.

What would that signal even look like, for a system whose behavior is continuous, not versioned, and whose changes are often invisible even to the people who made them?

Nobody has built it yet.

Sources

• OpenAI’s response to GPT-4 “laziness” complaints, via Search Engine Roundtable (December 2023). https://www.seroundtable.com/openai-chatgpt-gpt-4-getting-lazier-36529.html

• Search Engine Land, Google Algorithm Updates library: cites 4,725 changes to Google Search in 2022. https://searchengineland.com/library/platforms/google/google-algorithm-updates

In Slow Productivity, Cal Newport describes how writers develop taste. They read good writing, produce their own, and take in feedback until they build an internal standard. That standard is what lets them judge a draft before anyone else sees it. LLMs break that loop. The model doesn’t need that standard to produce fluent, confident output, and increasingly, neither does the person reading it. If you don’t know what good looks like, you have no way to know whether what you got is good. That’s what this post is about: not whether the model got the fact right, but whether anyone downstream can tell.

“Correct” and “right” are different failure modes

Two different things can be wrong with an AI answer, and most people are only checking for one of them.

The first is factually wrong. Hallucination, bad training data, an unlucky sample from the probability distribution. This is the failure mode people mean when they say “always verify AI output.” You check the answer against a source of truth and it holds up or it doesn’t.

The second is factually correct and wrong anyway. Ask an LLM to design a network for a mid-size healthcare org and it will hand you sound routing, sensible segmentation, a defensible firewall policy. Nothing in the design is technically incorrect. It might also route PHI traffic through a segment that violates the org’s existing BAA boundary, because nobody told the model that boundary exists. The design is correct and wrong at the same time.

These need different fixes. Fact-checking catches the first kind of error. It does nothing for the second, because there’s no fact to check. The design didn’t fail a lookup. It failed a constraint nobody wrote down. Teaching someone to fact-check AI output teaches half the skill and leaves them confident they’ve covered the whole thing.

Writing used to force the check, now it doesn’t

Before AI, if you needed a network design, you had to write it. Not type it up, construct it. Decide which segment carries PHI. Decide where the firewall rules get strict and where they can loosen. Decide what a vendor gets access to and what they don’t. Every one of those decisions ran through whatever judgment you’d built, whether you were consciously exercising it or not.

That’s the mechanism worth naming precisely. Writing the design didn’t just produce the design. It forced the designer to pass every choice through their own internal check, the same taste Newport describes writers building over years. You couldn’t hand someone a network diagram without first checking it against everything you knew about the org that was never written down. The check wasn’t a separate step. It was welded to the act of producing the thing.

AI output arrives with no check welded to it. The model doesn’t have the organization’s unwritten constraints, so it can’t run the design through them, and it isn’t trying to. That’s not a flaw in the model. Applying that check was never the model’s job. The flaw is in assuming the check still happens somewhere. For years it happened automatically, as a side effect of a human having to construct the thing by hand.

AI didn’t make the check harder or the answers worse. It made the check optional, for the first time, in a part of the workflow where it used to be structurally unavoidable. Whether that optional check gets exercised now depends entirely on whether the person receiving the output already has the judgment writing used to build.

Judgment has only ever come from two places

Two mechanisms have historically built the standard writing used to check against.

The first is tribal. A junior engineer proposes a design, a senior engineer tells them what’s wrong with it and why, and that correction gets internalized over years of repetition. This is apprenticeship. It’s slow and expensive, and it’s remarkably durable, because the judgment gets rebuilt inside a new person each time rather than stored anywhere fragile.

The second is programmatic: documentation. I’ve argued before that most organizational documentation fails at transferring understanding. Reading it was always optional, and the document was never the unit of understanding, the conversation around it was. A document that lists your org’s constraints doesn’t transfer judgment. It transfers information. Judgment is knowing which constraint to check for in a design you haven’t seen yet, and a list doesn’t teach you which situations to reach for it in.

There’s a third mechanism, and most orgs skip past it without naming it: externalized heuristics. Not a knowledge base of everything the org knows, that’s the shallow document problem in a new outfit. A specific, curated rubric of known failure modes for one class of decision. For network design: check PHI segment boundaries against every existing BAA first, check firewall defaults against the last three incidents, check vendor access against least-privilege before anything else. Closer to a pilot’s pre-flight checklist than a wiki page. It doesn’t teach someone to think like a senior engineer. It catches the specific ways this org’s designs have gone wrong before.

That’s also where this connects to something more practical. A checklist like that is cheap to build for decisions where verification is fast and the failure modes are well understood. It’s nearly useless for decisions where checking the output costs as much as producing it yourself. Sorting tasks by how expensive they are to verify, not by how often they come up, is the actual design question underneath all of this. It’s a different question from “does this person have judgment.”

Tribal knowledge is the default most orgs actually run on, whether or not they’ve built a checklist for anything. And it has a structural dependency nobody states out loud: it requires a continuous pipeline of juniors becoming seniors. Every org’s quality control is quietly resting on that pipeline staying intact.

What happens when the pipeline breaks

Layoffs over the last few years have hollowed out entry-level and, in plenty of orgs, mid-level engineering roles. I’m not going to argue about why that happened or how much of it is attributable to AI. I don’t have the data to make that case and it isn’t the point. The mechanism holds regardless of cause: fewer juniors get hired, fewer get mentored, fewer become the seniors who mentor the next round.

Here’s the consequence that matters for this post. A new hire today gets handed an LLM on day one. They don’t get handed the years of correction that used to build the judgment to check what it produces. They inherit the tool without inheriting the taste. They can prompt for a network design as capably as anyone. They have no way to know if the design that comes back is wrong in the second way, correct and wrong at once. Nobody has spent three years telling them what’s wrong with their designs and why.

This is where the two threads in this post meet. The epistemic problem is that AI output arrives without the check that writing used to force. The pipeline problem is that the people receiving that output increasingly don’t have the judgment to run the check themselves. The mechanism that built that judgment is being dismantled at the same time the check became optional. Neither problem is bad enough alone to be the whole story. Together, they compound.

Apprenticeship might not be the floor

The obvious objection: maybe three years of apprenticeship was never the actual floor. Maybe it was just the only delivery mechanism anyone had built, and a faster one is possible.

There’s a real case for that. Apprenticeship teaches judgment through a slow, sparse trickle of failures, one bad design at a time, spread across years. That’s how many real projects a junior actually touches. Curate the actual near-misses an org has hit instead. The design that looked fine and violated a BAA boundary. The migration that passed every test and still took down a dependent service. Put a new hire through deliberate, concentrated exposure to cases like those, and you might build a working approximation of judgment in months instead of years.

I think that’s plausible. I don’t think anyone has proven it. The difference between a checklist of known failure modes and a taught intuition for spotting new ones is exactly the difference between the second and third mechanisms from earlier. The taught intuition has to catch cases that don’t match any entry on the list. It’s not obvious that concentrated exposure closes that gap, rather than just widening the set of cases the checklist covers. Call this a real hypothesis worth testing, not a solved problem.

Generation cost dropped to nearly zero. Verification cost didn’t move. In a lot of organizations it’s about to get more expensive, because the people doing the verifying are the same people the pipeline stopped producing. That’s the actual currency now. Not which model you use, not how good your prompts are. Whether your org can manufacture judgment faster than tribal apprenticeship ever did. Not by writing a better document. By finding a substitute for the correction that only someone who has seen the failure before can deliver.

Writing used to force the check as a side effect of producing the thing. If that’s no longer true, and it isn’t, what forces it now?

I’ve spent years writing documents I knew most people would never read. Strategy docs, architecture white papers, product requirements, communication plans. I wrote them carefully. I formatted them properly. I put them in the right folder in Confluence and sent the right people a link.

And mostly, nothing happened. The org kept moving. Decisions got made. Projects shipped or didn’t. The documents sat there, timestamped evidence that the work of thinking had occurred.

When people ask whether AI is degrading organizational communication, I want to answer honestly: it’s accelerating a degradation that was already well underway. The more interesting question is why we didn’t notice sooner.

We were already not reading

Before AI entered the picture, document volume was already outpacing attention. The average knowledge worker was drowning in Confluence pages, Notion docs, Notion databases inside Confluence pages, Google Docs shared with edit access, Slack messages linking to all of the above. The backlog of “things I should read” was functionally infinite before anyone had access to a language model.

Here’s what that meant in practice. Most PRDs, white papers, and communication plans weren’t serving as transfer-of-understanding. They were serving as proof-of-work. The approval signature was the goal. The content was incidental.

Think about the last time a stakeholder pushed back on a specific sentence in a document you wrote. Not the summary. Not the conclusion. A sentence, mid-section, where the reasoning was soft. If that’s happened to you more than twice in a career, you’re ahead of most people. Usually what happens is: the doc gets circulated, a few people skim it, the right people click approve, and the work begins.

AI accelerated the production of these performance artifacts. The audience’s rational response, summarize it via AI before the meeting, is not a new failure. It’s a logical adaptation to content that was never genuinely information-dense. The AI-to-AI loop that people find so alarming, AI generates the doc, AI summarizes it for the recipient, is just the visible endpoint of a trajectory we were already on.

AI didn’t create the shallow document problem. It created conditions where pretending the problem doesn’t exist became impossible.

The slider intuition is wrong

The obvious response is calibration. Require more human effort in the writing. Better documents lead to better reading. Slide the dial back toward human-generated and we recover something we’ve lost.

This is wrong on two counts.

First, we already know human effort alone didn’t produce reading. The evidence is the decades of carefully written, professionally formatted, largely unread documents that preceded AI. If effort were the variable, those docs would have landed differently.

Second, the slider frames AI as the thing that changed. It isn’t. Accountability is the variable. The slider just adjusts the cost of producing cover.

Comprehension was always optional. Nobody got fired for approving a strategy document they skimmed. Nobody got called out for signing off on a PRD they didn’t fully understand. The document gave everyone cover, the author for having communicated, the approver for having reviewed, the organization for having a record. AI-generated documents provide the same cover at a fraction of the cost. That’s not a degradation. That’s an efficiency gain applied to the wrong problem.

Sliding the dial back toward human authorship doesn’t remove the optionality of comprehension. It just makes the performance more expensive to stage.

The document was never the unit of understanding

Ask yourself when understanding actually transferred in the organizations you’ve worked in.

Almost never in the reading of a document. It was the conversation after the document, the meeting where someone had to defend a position out loud, the design review where a tradeoff became unavoidable, the postmortem where a bad decision got traced back to a doc nobody had really engaged with.

The document was almost always downstream of the understanding, not the cause of it. The real cognitive work happened in the room, or on the call, or in the hallway. The document recorded a version of the conclusions that had already been reached.

What writing did was force the author to think. You can’t write a section on tradeoffs without resolving what the tradeoffs actually are. You can’t write a section on risks without deciding which ones are real. The artifact wasn’t valuable. The cognitive work of producing the artifact was valuable. That’s a crucial distinction.

AI removes that forcing function from the author. Summarization removes it from the reader. Both sides opt out of the cognitive work simultaneously. What’s left is the artifact, which was always the least important part.

Documents were doing two different jobs. We conflated them.

Documents weren’t a single thing. They were doing two genuinely different jobs, and we used the same artifact for both.

Job 1: Transfer understanding in the moment. Align the team on a decision. Get approval to proceed. Coordinate action across groups. This is the PRD, the architecture doc, the communication plan. The consumer is present-tense. The purpose is now.

Job 2: Preserve institutional memory across time. Capture decisions so the organization doesn’t have to rediscover them when someone leaves. This is the runbook, the architecture decision record, the post-mortem. The consumer is future-tense. The purpose is later.

These are genuinely different problems. Job 1 requires comprehension now, from specific people who need to act. Job 2 requires retrievability later, from people who don’t yet exist in the organization.

We used the same artifact for both and called it documentation. Then we complained that our documentation didn’t work. A PRD optimized for stakeholder approval is nearly useless as institutional memory. An architecture decision record optimized for future retrievability is nearly useless for driving present-tense alignment. They’re different documents serving different consumers on different timescales.

The shallow document failed at Job 1 because the conversation was always doing that work. But Job 2 is a legitimate problem, and it’s worth taking seriously on its own terms.

Design for verification, not for documents

If the document isn’t the unit of understanding, stop designing documents and start designing the moments where comprehension gets verified.

This isn’t a call for more meetings. It’s a call for smarter forcing functions. Three that work:

Decision forcing. Treat the AI draft as a starting position, not a deliverable. Before it becomes the record of alignment, run the room through explicit disagreement prompts. What’s the strongest argument against this? What did we decide not to do, and why? What would have to be true for us to be wrong? The document is kindling. The conversation is the fire.

Oral defense of tradeoffs. Anyone who signed off on a document can be asked, at any subsequent meeting, to explain the key tradeoffs and what was decided against. This doesn’t have to happen every time. It has to be possible every time. That possibility changes how people engage with what they’re approving.

Human authorship of the “what we didn’t do” section. This is the part AI reliably omits and the part that encodes real organizational thinking. What options were considered and rejected? What constraints ruled things out? What did we explicitly choose not to build? Require this section to be written by a human who was in the room. Protect it from AI generation. It’s where the real decision lives.

The point isn’t more process. Verification needs consequences. A norm without a consequence is decoration. If nobody ever gets asked to explain what they approved, the approval is still just cover. What changes is that misunderstanding now has a cost worth paying to prevent.

The memory problem is real. But we were solving it wrong.

Here’s the strongest objection to everything I’ve argued so far: if people leave the organization, their knowledge leaves with them. Documents are how you prevent that. Reduce the documents, lose the institutional memory.

It’s a real concern. But let’s look at the actual track record.

How often did the document actually prevent knowledge loss when someone left? In my experience: rarely. What happened was that the document existed in Confluence, nobody had looked at it in fourteen months, the person who wrote it was gone, and the context that made it meaningful was gone with them. The organization had a record. It didn’t have the knowledge.

A PRD buried in Confluence eighteen months after it was written, with no record of what changed after the doc was finalized, with no indication of which decisions held and which got reversed, is not institutional memory. It’s institutional archaeology. The knowledge was there in theory and gone in practice.

Here’s what AI changes about Job 2, in a way it doesn’t change Job 1. Retrieval and synthesis across a structured knowledge base is something AI does genuinely well. The human reader who was supposed to find and read that PRD almost never did. An AI assistant querying a well-structured decision log on behalf of a new engineer asking “why did we choose Postgres over DynamoDB for this service?” will surface the answer faster and more reliably than any wiki navigation ever did.

That changes what good institutional memory looks like. Stop writing it for a human reader who won’t show up. Write it for AI retrieval. The design properties are completely different.

Structured for retrieval, not reading. Decisions, context, tradeoffs, outcomes. Short, atomic, linked. Not prose paragraphs that require sequential reading to yield their meaning.

Decision-centric, not document-centric. The unit is a decision and its reasoning, not a document and its sections. The question a future employee asks isn’t “can I find the architecture doc from 2023?” It’s “why did we build it this way?” Those are different retrieval targets.

Queryable by AI, not navigable by human. The interface assumption flips entirely. You’re not building a wiki a human browses. You’re building a knowledge base an AI queries on behalf of a human asking a question in natural language. That means structure and tagging matter more than narrative flow. It means a 200-word decision record with clear metadata is more valuable than a 2,000-word white paper.

Living, not archival. Decisions get updated when they change. The system distinguishes between what we decided, what we decided later, and what we eventually reversed. Confluence graveyards fail here not just because they’re hard to find, but because nothing ever gets updated. Every doc reflects the thinking at the time of writing and nobody knows what changed.

I’ve built a version of this for my own work, a system I call Mindstore. The point isn’t the specific stack. It’s that the design assumptions are completely different from anything we’ve called documentation before. When I want to know why I made a past technical decision, I ask a question in natural language and get a synthesized answer grounded in the actual decision records. That’s a different tool than Confluence. It’s serving a different consumer.

A big caveat: this only works if the inputs are disciplined. Garbage in, hallucinated retrieval out. The cognitive work doesn’t disappear. It shifts from writing documents to capturing decisions at the moment they’re made. That’s still a human job. It connects directly back to the verification moments from the previous section. Decision forcing and oral defense aren’t just accountability tools. They’re also the mechanism by which real decisions get captured in a form worth preserving. The two halves aren’t separate solutions. They’re the same solution applied to different timescales.

Less documentation, not better documentation

If we get this right, the output isn’t more carefully written docs. It’s fewer docs.

The ones that remain will be shorter and harder to write. They’ll carry actual decisions, including what was decided against. They’ll be structured for the consumer they’re actually serving, whether that’s a human who needs to act now or an AI assistant helping a future employee reconstruct why the system works the way it does.

AI handles everything that was always performance. Human attention goes to the things where misunderstanding has a cost. Job 1 gets replaced by verification moments with consequences. Job 2 gets replaced by structured decision capture optimized for retrieval, not reading.

This will threaten people whose job looks like “producing documents.” That’s not a side effect. It’s the point. The shallow document problem persisted for decades because it served everyone’s interests except the organization’s. It gave authors credit for communicating. It gave approvers cover for reviewing. It gave the org a paper trail. AI just made the hollowness of that exchange too obvious to ignore.

The uncomfortable question

If accountability was always the missing variable, why didn’t we fix it before AI made it obvious?

The shallow document gave everyone cover, including leadership. A VP who approved a strategy doc without reading it can point to the approval. A team that shipped a feature nobody wanted can point to the PRD. The document was the alibi, not the evidence.

Fixing verification means removing the alibi. It means the person who approved the document has to be able to explain it. It means leadership doesn’t get cover from a doc they skimmed. That’s why this problem persisted. The people with the power to fix it were the people most protected by its existence.

AI accidentally gave us the tools to fix Job 2 correctly, if we design for the right consumer. What it can’t give us is the will to fix Job 1. That requires deciding that misunderstanding has a cost worth paying to prevent, and that the people responsible for decisions are actually responsible for understanding them.

That’s the conversation we should be having.

At WWDC this month, Apple introduced a piece of software it calls the system orchestrator. It routes every AI request to the model that should handle it, on the device or in the cloud, based on how much compute the task needs and how sensitive the data is. Craig Federighi called it the key to the system’s privacy architecture.

That routing decision is the most consequential architectural shift in consumer AI since the transformer. Not the model. The decision about where the model runs.

Cloud by default was never a decision

From 2022 through 2024, anything that looked like reasoning went to an API. GPT-4, Claude, Gemini all lived in central data centers. On-device AI meant autocomplete and spell check. The hardware floor on a phone was too low to run a capable model, so the model lived somewhere else.

The constraint felt permanent because it had always been true but Always has does not mean always will. We made the same error with local storage, with video on mobile, with encryption in the browser. The thing that is too expensive to do locally stays in the data center right up until it doesn’t.

The floor is already higher

Look at what already left the cloud. Real-time transcription runs on your phone. Translation runs on your phone. Photo understanding, system-wide dictation, short-form classification, writing suggestions. None of these phones home anymore. Apple’s second on-device model, the more capable one for current Apple silicon, understands speech, reads text and images, and never leaves the device.

We should measure the floor by what it can do, not by its parameter count. In 2023 the floor was “can it follow an instruction.” In 2026 the floor is “can it handle most single-document knowledge work without a network connection.”

The reason this happened so fast is the efficiency curve underneath. Sparse activation, distillation, and aggressive quantization decoupled capability from cost. A sparse model carries a large parameter count but uses only a fraction of those weights on any given token. The headline says twenty billion parameters. The actual work on a prompt might touch a few billion. That gap is the trick that put a real model on a phone.

The floor doesn’t rise on a smooth line. It jumps when a new efficiency technique lands and then sits flat until the next one. Mixture-of-experts was a jump. Quantization down to four bits was a jump. Nobody knows the slope of the next five years but he direction is obvious.

Routing is the product, not the model

The orchestrator was the most underreported thing Apple showed. The press wrote about Siri’s new voice. The architecture was the story that was glossed over.

What does the orchestrator decide? Latency tolerance, privacy sensitivity, task complexity, and the hardware tier of the device in your hand. Send the easy local thing local. Send the hard private thing to Private Cloud Compute. This is not a new problem in infrastructure terms. A CDN decides whether to serve from the edge or the origin. A database decides whether a read hits a replica or the primary. Routing a request to the cheapest box that can satisfy it is the way we’ve always done it.

The new part is what the routing decision means. A CDN serving a cached asset gives you the same asset. An orchestrator routing to the small model gives you a different answer than the big model would have. The routing decision now changes the quality of the intelligence, not just the latency. That is a category of routing we haven’t built governance for.

Whoever owns the routing layer owns the usage signal. The model that gets called is the model that sees the data, collects the feedback, and improves fastest. Apple makes that decision for more than a billion devices.

This is how Apple played it. The next-generation foundation models are being co-developed with Google’s Gemini, and the cloud tier runs on Nvidia GPUs inside Google’s cloud. Apple did not insist on owning the best model. It rented the model and kept the layer that decides which model you reach. The model is rentable. The routing layer is not. This is the key. If somewhere down the road, Apple decides to switch to their own cloud based AI, they can. This is what happened with mapping software, Apple partnered with Google until it developed its own. Who knows if Apple will develop their own models but if they do, they can turn it on very quickly.

The open question is whether the orchestrator stays an honest dispatcher or becomes a toll booth.

Regulation is fragmenting AI by geography

Consumer privacy preference is real and soft. People say they care and then tap accept. Regulation does not depend on what anyone prefers. It is hard, and it is fragmenting AI deployment along national lines.

Look at the Siri example: At WWDC, Apple announced that Siri AI will not ship in the EU with iOS 27. Apple framed the block as privacy and security. The cause is actually the Digital Markets Act, which is competition law, not privacy law. The European Commission rejected Apple’s framing in public and without hedging. Nothing in the DMA blocked the launch. The delay was Apple’s own choice. The real objection is that Siri AI, which is powered by Google’s Gemini, would lock out competing assistants on a platform that reaches hundreds of millions of users. The Commission’s spokesman named the Gemini detail on purpose.

This is not “privacy law is forcing AI onto devices”. It’s deeper than that. AI deployment is fragmenting by jurisdiction, and the pressure comes from several different bodies of law at once. Competition law in Brussels. Data localization in India, China, Saudi Arabia, and Brazil. A model that runs locally sidesteps an entire class of these fights because the data never crosses a border. The on-device model is not just faster and more private. It is jurisdictionally invisible.

This is where sovereignty stops being a slogan. A regulated enterprise or a government does not want a privacy assurance about someone else’s data center. It wants the weights, the inference stack, and the audit trail inside its own borders. Private Cloud Compute, however well it is engineered, is still someone else’s cloud in someone else’s country. The requirement is on-premise open-weight deployment. Models like Llama and Mistral are important here for a reason that has nothing to do with topping a benchmark. You can run them where the law says the data has to stay.

US export controls accelerate all of this. A country that cannot count on access will build or source its own. The recent retraction of Anthropic’s Fable 5 model is real world example of this. Scarcity is a fast teacher of self-reliance. The open question is whether on-device and on-premise can satisfy the provenance and audit demands that regulators will have.

Losing the commodity tier is not the threat

The tasks leaving the cloud were never the high-margin tasks. The person summarizing email on a phone was not paying for an API call. Losing that load costs the frontier labs close to nothing.

The real exposure is the middle. Function-level code completion. Document question-and-answer. Single-session knowledge work. These are mid-complexity and high-volume, and they are moving on-device, likely inside two years. The timing will vary based on the use case.

The floor does something quieter to the rest. When good-enough AI is free, private, and already in your pocket, every upgrade has to clear a higher bar to earn a dollar. Apple’s orchestrator teaches a billion people that AI is the free thing that already lives on the phone. That reshapes what AI is worth in the median person’s head before any frontier lab gets to make its pitch.

Volume and revenue are not the same curve. A frontier provider can lose sixty percent of its query volume and far less than sixty percent of its revenue, as long as it priced the commodity tier near zero in the first place. The query that left was the query they weren’t making money on anyway.

So who is actually exposed? Not the top labs but the mid-tier API resellers and the vertical SaaS products whose only moat was access to commodity inference. When the inference is free and local, the wrapper has nothing left to sell. The move for the labs at the top is to go up. Own the reasoning and the agentic layer. Concede the commodity tier without a fight, because fighting for it means competing with free. The open question is whether “own the reasoning layer” is a durable position or just the next floor waiting to rise under it.

The grid did not die. Its economics did.

The useful precedent is electricity, not software. As the electrical grid matured, generation moved closer to the load. Rooftop solar, home batteries, and the central grid now coexist. Nobody decommissioned the grid. What changed permanently was the economics of building another central plant when the marginal load could be served closer to home.

The analogy holds here too. Local handles the frequent, ambient, low-stakes load. Central handles the peak, the complex, the high-stakes work. Neither one kills the other.

But here is where the analogy breaks: Electricity is fungible. A kilowatt-hour off your roof and a kilowatt-hour off the grid are the same kilowatt-hour, and your refrigerator cannot tell the difference. Intelligence is not fungible. The answer from the local model and the answer from the frontier model are different answers, sometimes meaningfully different, and the user often cannot tell which one they got.

The analogy gives us the shape and then fails at the load-bearing joint. The floor rising does not make the ceiling matter less. It makes the gap between floor and ceiling the thing you are selling. The product stops being “AI.” The product becomes the difference between the AI everyone has for free and the AI that is worth paying for. Pricing that gap, and proving it exists on any given task, is the whole game now.

Where do we go from here?

We can see the floor rising. We can see the gap becoming the product. What’s still unknown is who governs the routing layer.

Today Apple decides for Apple devices. Google decides for Android. Microsoft decides for Windows Copilot. None of these is neutral infrastructure. Each one is a company that also sells models, which means each routing decision is made by a party with a stake in the outcome.

The flywheel is the important part. The model that gets called gets the usage signal. The usage signal becomes the improvement that makes it the model worth calling next time. Whoever holds the orchestrator holds the flywheel. Right now the platform owners are holding it, and they did not have to buy a single frontier lab to do it.

If routing gets captured at the platform layer, the frontier labs become wholesale suppliers to a retail counter they do not own. They will have built the best engine in the world and handed the dashboard to someone else.

Morph LLM published a study in March 2026 with a claim that every frontier model they tested gets worse at reasoning as you give it more to read. The output quality drops in a way you can measure and reproduce, and it happens on the exact models you are paying for. The instinct is to file this under “models are unpredictable” and move on. That instinct is the mistake this post is about.

Let’s start with the symptom, because you have probably already seen it. A multi-stage agent runs cleanly through step three. It reasons correctly, calls the right tools, returns what you expected. By step twelve the same agent, on the same task type, with the same model and the same prompt pattern, is producing degraded output. Nothing about the task got harder. Something about the context got heavier. That gap, between a system that works early and the same system failing late, is what I want to talk about. It’s called context rot, and it’s architectural, which is the part that matters. You can’t upgrade your way out of an architectural property.

The spec sheet measures the wrong thing

When a vendor prints 128K on the box, that number is a ceiling on what the model will accept. It says nothing about what the model can reason over reliably. Those are different quantities, and the gap between them is enormous.

Paulsen and colleagues gave the distinction names in September 2025. The Maximum Context Window is the spec sheet figure. The Maximum Effective Context Window is the point where accuracy meaningfully degrades for a given task. Their findings show that some top models showed severe degradation by 1,000 tokens. A few broke down with as little as 100 tokens in context. Across everything they tested, the effective window fell short of the advertised one by as much as 99 percent.

That’s a big difference. A 128K window can deliver reliable reasoning over something closer to 1K on a hard task. The other 127K is tokens the model will dutifully process and cannot actually use well.

The effective limit also moves with the job. A flat retrieval task and a multi-hop reasoning task over the same corpus have different effective windows, because reasoning over scattered evidence stresses attention harder than fetching one fact. So there is no single number to memorize. You need to figure out for yourself how it will impact your workflow.

The RAM comparison is exact, not loose. A machine with 128GB of memory does not run every workload 128 times faster than a 1GB machine. The number is capacity, not throughput. Context windows are the same. We have been reading a capacity spec as a performance spec, and the vendors are happy to let us.

Why attention thins out

Three mechanisms compound as context grows. You want the mechanisms, not just the result, because the mechanisms tell you what to do.

Position is not neutral. Liu and colleagues at Stanford showed transformer attention follows a U-shape across token positions. The model attends strongly to the start and the end of its context and weakly to the middle. In a 20-document QA task, accuracy dropped more than 30 percent when the relevant document sat in positions 5 through 15 versus position 1 or 20. The same fact, the same model, moved a few thousand tokens inward, and the model half-forgets it is there. Models trained explicitly on long context still showed it. So the first thing to internalize: where a token sits changes whether the model can use it most effectively.

Attention is a fixed budget split more ways. The attention mechanism distributes a finite amount of probability mass across every token in context. Add tokens and each one gets a thinner slice. This is not a tuning artifact. It falls out of the softmax math. More context does not mean more attention to go around. It means the same attention spread thinner, and past some point the model can no longer cleanly separate the signal token from the noise around it.

The curve flattens early. A model reading 100K tokens is not getting ten times the benefit of one reading 10K. Positional undertraining and the constraints of rotary position embeddings mean effective utilization grows far slower than the token count. The useful curve bends down well before the nominal limit. You are paying linearly for tokens whose value is already sub-linear.

Put the three together and the picture is not “long context is a bit lossy.” It is that the middle of a large context is a place where information goes to be underweighted, and you are filling it on purpose.

Agent loops manufacture their own noise

Static document Q&A is forgiving. You load a corpus once and ask questions. Agent loops are much less forgiving because they generate the very noise that rots them.

Watch what accumulates in a long-running loop. Verbose tool outputs, most of which were only partially relevant. Reasoning paths the model explored and abandoned. Intermediate states that are no longer true. And the one almost everyone forgets: prior turns where the user chose between options.

That last one deserves a deeper look, because it is the clearest illustration of the whole problem. The model offers two options. You pick the first. The second option is not dropped. It sits in context carrying the same weight as the path you chose. Ten turns later the model is reasoning over a growing pile of decisions you already killed. The dead ends do not decay. They sit alongside the live thread, drawing from the same thinning attention budget, and the model has no way to know they are dead unless you tell it.

This is why Morph LLM’s work, which folded in Chroma’s research on coding agents, landed on context rot as the primary failure mode for agentic coding. Not model capability. Not reasoning quality. The models are capable enough. The context they are handed is too noisy, and most of the noise is self-inflicted.

The diagnostic is precise once you know where to look. Degradation that scales with session length rather than task complexity is context rot. If your agent clears step three and fails step twelve with no change in task type, stop tuning the prompt. You are not looking at a capability problem. You are looking at a hygiene problem.

The two curves cross

Everything so far is about quality. The cost side is where it stops being an abstraction and starts showing up on the bill.

The KV cache scales with context length, and at high token volumes that scaling is not gentle. Latency climbs. VRAM consumption grows. Throughput falls. None of these track token count proportionally. They accelerate.

Think about how that pairs with the quality finding. The cost per useful unit of work is rising at exactly the point where each token is delivering less. This is not ordinary diminishing returns, where you pay the same for a bit less. The cost curve bends up while the quality curve bends down. Those two curves cross. Past the crossing point you are paying more to get worse output, and nothing in the system will let you know you’ve crossed it.

Mixture-of-Experts architectures make the diagnosis harder, not easier. They can hide this dynamic underneath infrastructure bottlenecks, so the system looks healthy right up until it is obviously not.

Manage it like memory, because that is what it is

The bottom line is that context is a resource with a budget, and right now most systems don’t see it that way. Here are three things you can do:

Budget context against your effective window, not the spec sheet. Estimate the effective window for your task type and set a token budget per loop or session against that number, not the advertised one. When you hit 60 to 70 percent of the budget, trigger a checkpoint. Make the checkpoint a structured state object, not a prose summary: decisions made, options rejected, current state. Then continue from the checkpoint instead of the full transcript. You are choosing what survives into the next turn rather than letting accumulation choose for you.

Prune the dead weight. A tool call that returned nothing useful should be stripped or compressed before the next turn. A rejected option should collapse to a single line. The model does not need the full text of a path it is not taking, and every token you remove is a token that can no longer dilute attention on the tokens that matter. Do this as part of the loop.

Measure degradation as a function of length, not just at turn one. Run your evals at turns 5, 10, and 20, not only at the start. Watch accuracy as context grows on your specific task. If you cannot state your effective window as a number, you do not know where your system fails, which means you are flying without a parachute.

A bigger window is not the solution. The Morph LLM data spans GPT-4.1, Claude Opus 4, and Gemini 2.5 Pro. Every one degrades. The window sizes differ. The architectural property does not. A larger window buys you more room to accumulate noise, not more reasoning.

What the vendors are not selling you

Context window size has become the spec that vendors compete on, the way hardware vendors once competed on raw clock speed. The number is real. What it implies about useful work is a different claim, and it’s the claim you don’t tend to hear about.

The engineers who build reliable long-context systems will not be the ones with the largest windows. They will be the ones who treat context the way a systems engineer has always treated memory: as a scarce resource with budgets, checkpoints, and active reclamation. The model will not do this for you. It cannot garbage collect its own context. It has no concept of which tokens are dead. That judgment, what to keep and what to get rid of, is the part that does not come in the box, and it’s up to you to figure out.

In December 2025, the Google Cloud CTO office published a post-mortem on what they called the “reliability gap” in agentic AI systems. The core finding was not subtle: the move from deterministic software to probabilistic agentic workflows introduces failure modes that most teams only discover after architecture decisions are locked. The paper described multi-step agent chains operating without transaction coordination, creating non-atomic failures that can produce irreversible side effects and data corruption mid-operation.

That is not a model quality problem. It is a structural design problem. And it is expensive in two ways that most AI cost models don’t account for.

The bill arrives after the architect has left

LLM inference costs scale with volume and context length. Everyone knows this. Where it hets more complicated is when you chain agents together.

Every step in a multi-agent workflow triggers at least one model call and also, potentially, retries. Context lookups trigger more. Add a reflection layer, an orchestration agent, a validation pass, and your per-transaction cost can explode.

This comes about because the costs don’t start to mount until it gores into production and the builders have moved on. By the time you see this, it’s too late.

The vendors selling agent frameworks are not incentivized to surface this. Their pricing models are based on consumption. A platform that makes it trivially easy to chain five agents together is probably not going to ship a cost impact calculator so you know what you are getting into. You discover the problem at the quarterly cloud invoice review.

Right sizing is not hard to figure out; deterministic execution for stable, high-volume paths, and probabilistic reasoning reserved for genuinely ambiguous edge cases where it creates real value. Many teams reach for an agent because the demo was impressive, not because the task required reasoning under uncertainty.

Regulation does not grade on a curve

The EU AI Act imposes documentation, transparency, and auditability obligations on high-risk AI systems, with phased enforcement running through 2027. At it’s most basic level: a system subject to these rules must be able to reconstruct its decision logic. It must be able to answer, on demand, why a particular output was produced.

A live LLM agent in the execution loop can’t answer that question. The reasoning is not stored. The attention weights are not auditable. The path from input to output runs through billions of parameters and produces no human-readable trace of its logic.

A deterministic workflow that was generated by an AI can answer that question. The code exists. The execution path is traceable. The logic is inspectable.

This distinction matters beyond Europe. Any organization operating in healthcare, finance, insurance, or legal already faces existing sector regulation with equivalent requirements. HIPAA audit trails. SEC recordkeeping. Insurance claim adjudication documentation. These obligations don’t go away because you replaced a rules engine with an agent. They get stronger, because the regulator now wants to understand the AI component specifically.

The compliance teams asking these questions are not being obstructionist. They are pointing at a real architectural gap that the engineering team may not have modeled as a risk.

Trust is already declining

Augment Code’s 2025 developer survey found that trust in AI-generated code accuracy dropped from 40% to 29% in a single year. That is not a rounding error. Adoption grew over the same period, reaching 62% of developers, but confidence in the output fell sharply.

More developers are using AI. Fewer of them trust what it produces. Developers are shipping AI-assisted work while privately applying a larger mental discount to its reliability. That discount has a cost. It shows up as increased review time, more defensive testing, slower iteration on the critical paths where the stakes are highest.

Enterprise teams that have moved to hybrid architectures report a different pattern. When agents set goals and orchestrate tasks but critical computations run inside deterministic modules, the LLM surface area at runtime shrinks. The auditability problem gets smaller because the probabilistic component is doing less. Data shows teams using this approach achieving over 80% reduction in manual effort while maintaining the traceability their compliance functions require.

What the architecture decision actually is

Treating atomicity as an infrastructure requirement instead of a prompting challenge changes what you build. It means designing agent chains with transaction boundaries, rollback logic, and explicit failure modes before you write the first prompt. It means asking, for every step in the workflow, whether this step requires reasoning under uncertainty or whether it requires reliable execution of a known procedure.

Most enterprise AI architectures are not designed this way. They are grown from demos. The agent that worked in the proof of concept becomes the agent in production, with more context, higher volume, and a compliance audit scheduled for Q3.

The Google Cloud paper put it directly: non-atomic failure modes create irreversible side effects. That is an infrastructure problem. Changing the system prompt does not fix it.

The question I keep coming back to is whether the hybrid architecture pattern will be adopted proactively or reactively. Some teams will design for it from the start. Most will discover the need for it after the first incident that cannot be explained to a regulator, or the first quarter where inference costs exceed the value the system generated.

Nobody set out to discover a better architecture for AI systems. It happened because someone typed “write me a function that parses this JSON” and the result just worked.

The LLM ran once. It interpreted the intent, produced code, and exited. The code ran in CI, in production, on a schedule. No LLM in the loop. Nobody called it an “AI agent.” They called it software.

This was not a product decision. It was not an architectural principle. It was the path of least resistance. The developer got what they needed and moved on. The LLM was done before the workflow started.

That accident of framing contains the most important architectural insight in AI systems today. Most teams building agentic workflows have completely missed it.

The hammer is real. The grip is the problem.

The capabilities are genuine. An LLM can interpret ambiguous human intent, navigate unstructured input, and produce structured output from near-infinite input variations. Those are hard problems. LLMs solve them well.

The mistake is assuming that because an LLM can do something at design time, it should keep doing it at every execution. On every single run.

Maslow’s famous observation was that if your only tool is a hammer, every problem looks like a nail. The AI industry’s current posture is a precise instance of this. We have a remarkable tool. We are applying it to every stage of the pipeline, including stages where it has no business being.

The problem is not that LLMs are unreliable. The problem is that probabilistic systems do not belong in deterministic execution paths.

What “probabilistic in the execution path” actually costs

The math is unforgiving. A 20-step agentic workflow where each step succeeds 95% of the time produces an end-to-end success rate of 36%. This is not a model quality problem. Better models improve the per-step number. They do not change the compounding structure. The architecture is the problem.

Kambhampati et al. established the theoretical foundation for this in their ICML 2024 paper on LLM-Modulo frameworks. Auto-regressive LLMs cannot, by themselves, do planning or self-verification. The LLM is a probabilistic knowledge source. Treating it as a deterministic executor is a category error.

Mehta’s recent consistency research on SWE-bench sharpens this further in a way most people have not fully processed. Across 150 agent trajectories, the finding was not that inconsistent models fail more. It was that consistency amplifies outcomes rather than guaranteeing them. Claude achieved 58% accuracy with 15.2% behavioral variance. When it interpreted a task correctly, it succeeded on 100% of runs. When it got the interpretation wrong, it failed on 100% of runs. The same wrong answer, five times in a row, with high confidence.

A model making the same incorrect interpretation on every run is worse than a randomly failing one. The failure is invisible until it compounds, and reflection-based recovery cannot help if the initial interpretation is wrong.

ReliabilityBench (Gupta, 2026) adds the production dimension. Agents achieving 96.9% accuracy on clean benchmarks drop to 88.1% under realistic perturbations. The paper’s broader finding: metrics on clean data overestimate production reliability by 20 to 40 percent. The gap is not a model problem. It is a measurement problem that reveals an architectural one.

Better models improve the per-step number. They do not change the compounding structure.

The pattern developers discovered by accident

Code generation is the canonical LLM-as-compiler case. The LLM interprets intent and emits a deterministic artifact. That artifact runs without the LLM. Nobody designed this as an architecture. It emerged because “write me code” has a natural exit condition: the code either works or it does not.

The insight generalizes far beyond code. A policy document can become an OPA rule set. A process description can become a Temporal workflow definition. Decision criteria can become a scoring model. The LLM’s job in each case is translation from human intent to a deterministic artifact. Once the artifact exists, the LLM is done.

Khattab et al.’s DSPy framework from Stanford is the explicit formalization of this idea. Compile the prompt and retrieval logic into an optimized, repeatable program rather than invoking raw LLM calls at runtime. The artifact owns the execution. The LLM’s work happens once, at design time.

The reason this pattern is underused outside of engineering contexts is not that it does not apply. It is that business users do not yet have the intuition that “the AI is finished” is a valid and preferable state. Finished means auditable. Finished means version-controlled. Finished means you can write a test for it.

When the LLM does belong in the path

There is a class of workflow where LLM-as-compiler fails. The failure mode is unstructured with novel input at execution time.

Customer support is the clearest case. You can compile decision trees for known issue categories. You cannot compile a response to a customer describing a problem you have never seen, in language you cannot predict, with emotional context that changes the appropriate answer.

The same logic applies to open-ended document classification where the category space is not closed; to real-time anomaly detection on free text; to any workflow where the output depends on semantic understanding of content that did not exist when the workflow was built.

The LLM belongs in the execution path when the input space at runtime is genuinely open-ended. That is a much smaller category than most people think.

The question is not “can an LLM handle this at runtime?” The question is “does this workflow require runtime interpretation, or was the interpretation problem already solved at design time?”

The decision rule

Can you fully describe the execution logic as a flowchart, rule set, or decision tree before the workflow runs?

If yes, the LLM’s job is to build that artifact. Not to approximate it on every execution.

If the answer is no, because the logic depends on content that cannot be anticipated or because the decision space is genuinely open, then the LLM belongs in the path.

The key to reliability is to apply this test at all times and minimize LLM surface area at runtime. Not because LLMs are bad. Because deterministic systems are auditable, testable, version-controllable, and do not compound errors across steps.

Most teams are not asking this question. They are asking “how do we make the agent more reliable?” That is the wrong starting point.

If you are running an LLM on every execution of a workflow that could have been compiled into a deterministic artifact, you have not built an AI system. You have built a reliability problem with an AI-shaped interface.

The first question to ask is not whether your LLM is good enough. It is whether it needs to be there at all.

In October 2023, New York City launched an AI chatbot to help small business owners navigate city regulations. The mayor’s office called it a frictionless doorway into City Hall. Five months later, investigative journalists at The Markup tested it and found it was confidently telling landlords they did not have to accept Section 8 housing vouchers. That is illegal in New York City. The same bot told employers they could take a cut of their workers’ tips. Also illegal. When ten journalists asked the same housing question independently, all ten got the same wrong answer.

The city added a disclaimer. The mayor defended it and kept it running. The next administration eventually called it “functionally unusable” and moved to shut it down. The bill for building it: roughly $600,000.

Nobody asked the right question before launch. Not “can we build this” but “does a probabilistic text system belong in the execution path of legal regulatory guidance, where there is exactly one correct answer per question and the person asking has no way to know when they’ve received the wrong one.”

That question, whether AI belongs in a given workflow at all, is the one most organizations skip. They go straight to “how do we make this work” and treat the premise as settled. It isn’t always.

The question teams are not asking

When an AI deployment struggles in production, the instinct is to reach for better models, more data, or tighter prompt engineering. The conversation becomes technical quickly. That is the wrong reflex.The right first question is simpler and more uncomfortable: should AI be in this workflow at all?Not “can it be done” but “should it be done here, in this position, with these consequences attached.”Many teams never ask this. Gartner projects that 40% of enterprise applications will feature task-specific AI agents by 2026. MindStudio found that only 23% of companies run fully autonomous agent systems despite 88% already applying AI somewhere. The gap between “we use AI” and “we have thought carefully about where AI runs” is huge. That gap is where the expensive mistakes live.

Blast radius is the first variable

The first question to ask about any workflow is: what happens when this output is wrong?

Not if. When. Every probabilistic system produces wrong answers. The question is what those wrong answers cost.

A wrong answer in marketing copy gets caught in review and fixed in ten minutes. A wrong answer in a financial reconciliation can trigger a downstream payment workflow before any human sees it. A wrong answer in a regulatory guidance tool may prompt a landlord to illegally reject a tenant, with no visibility until a fair housing complaint lands.

This is blast radius. It is not a measure of how often the system fails. It is a measure of how bad a single failure can get. High blast radius workflows require a different standard of scrutiny before AI gets anywhere near the execution path.

The NYC chatbot did not fail because the underlying model was bad. It failed because the blast radius of wrong legal guidance is real harm to real people. The city had no way to know how many business owners had already acted on the bad advice before The Markup published its findings.

Detectability is the second variable

The second question is: how would you know the output was wrong, and how quickly?

This one is underweighted in almost every AI deployment conversation I have seen. Teams spend enormous effort on accuracy and almost no effort on error visibility.

Some errors are self-evident. A generated email that addresses the recipient by the wrong name gets caught before it sends. Some errors are invisible. A landlord who received incorrect guidance from an official city tool has no reason to doubt it. The tool carried government branding. It answered confidently. The correct answer was on a different page of the same website, and nobody told them to go look there.

The combination that kills enterprises is low detectability paired with high blast radius. That is the dangerous quadrant. Errors that are hard to see and expensive when they land. That is where autonomous agents are being deployed right now at scale.

The EU AI Act, phased in between 2024 and 2027, is not primarily about model accuracy. It is about auditability. Regulators are not asking whether your AI is usually right. They are asking whether you can prove what it did and why. That is a detectability requirement disguised as a compliance requirement.

Variance is only a feature in the right context

The third question is: does the right answer change between runs, or is there one correct answer?

Large language models are non-deterministic by design. They produce different outputs given the same input. For some tasks that variance is the point. Generating five options for a campaign headline and picking the best one is a workflow where variance creates value.

For other tasks variance is a defect. A financial reconciliation has one correct answer. A patient’s medication dosage has one correct answer. A contract clause either complies with local law or it does not.

The NYC chatbot illustrated this with unusual clarity. Whether a landlord is legally required to accept a housing voucher is not a matter of interpretation. There is a correct answer. Yet the bot gave different answers at different times and gave the wrong answer to all ten journalists who tested it simultaneously. A system producing variance in a zero-variance workflow is not a system having a bad day. It is a system that was never fit for the task.

Variance only creates value where the space of acceptable outputs is large and the evaluator is a human who can use judgment to select. Anywhere the evaluator is a downstream system, a database constraint, or a person who treats official guidance as authoritative, variance is a bug.

Downstream systems change the risk profile entirely

The fourth question is: is there a system consuming this output directly, without human review in between?

This is the difference between a soft consumer and a hard consumer. A human reading an AI-generated summary is a soft consumer. They bring judgment. They catch obvious errors. They can push back.

An API endpoint that ingests an AI output and triggers an action is a hard consumer. It has no judgment. It cannot catch errors. It will execute faithfully on whatever it receives.

The NYC chatbot consumer was human but effectively hard. A small business owner consulting an official government tool on a compliance question is not going to fact-check the answer. The government branding collapsed the soft consumer into something that behaved like a hard one. The output fed directly into a decision with no meaningful review layer in between.

When AI feeds a hard consumer, the error characteristics of the system change completely. Errors no longer surface through human review. They surface through system failures, audit findings, or in this case, investigative journalism.

Compliance exposure is not a legal problem

The fifth question: is this workflow subject to audit or regulatory review?

This gets treated as a legal team concern. It is not. It is an engineering and product concern that shows up as a legal problem later.

Regulated workflows require explainability. They require audit trails. They require the ability to reconstruct exactly what the system did and on what basis. Current LLM architectures are not designed to produce that by default. You can build it in, but it requires deliberate design choices made before deployment, not retrofitted after a regulator asks a question.

The EU AI Act classifies certain AI uses as high-risk. Legal and regulatory guidance is squarely in that scope. High-risk classification does not mean prohibited. It means the compliance burden is significant enough that you need to answer all five questions before deployment, not after the headline.

The real question behind the questions

Run these five questions against any workflow you are evaluating for AI automation:

1. What happens if this output is wrong? (blast radius)

2. How would you know it was wrong, and how quickly? (detectability)

3. Does the right answer change between runs or is there one correct answer? (variance tolerance)

4. Is there a downstream system or a trusting human consuming this output directly? (soft vs. hard consumer)

5. Is this workflow subject to audit or regulatory review? (compliance exposure)

A workflow with high blast radius, low detectability, zero variance tolerance, a hard consumer, and regulatory exposure is not a workflow where you deploy an AI agent today.

Most businesses deploying LLMs are treating them as a new category of software. The governance model they’re importing assumes a simple contract: same input produces same output, every time, reliably enough to stake compliance on. For example: procurement checklists, SLAs, acceptance testing, and audit trails.

LLMs are probabilistic. This breaks the model.

The setting that doesn’t do what you think it does

The standard response to LLM nondeterminism is to set temperature to zero. That’s supposed to make the model deterministic. It doesn’t.

Researchers at ACL’s Eval4NLP workshop published empirical results showing accuracy variations of up to 15% across runs with supposedly stable settings. The best-to-worst performance gap on some tasks reached 70%. Thread scheduling, parallel inference, mixture-of-experts routing, and server-side model updates all introduce variance that temperature settings cannot touch. Temperature=0 means you asked for determinism. It does not mean you got it.

Now chain three agents together, each operating at 90% reliability. Your combined system accuracy is around 73%. Add a fourth agent and you’re below 65%. Probabilistic error compounds through a pipeline the same way latency does. Princeton’s Language and Intelligence Lab studied real-world agent failures against benchmark performance and found this gap is systematic. Not anomalous.

This is not fixable with better prompting. It’s architectural.

Temperature is just the most visible example of a broader pattern. Every instinct your team has about fixing nondeterminism will reach for a deterministic lever: tighter prompts, stricter validation, more retries etc. These are reasonable engineering responses to a problem that is not an engineering problem. The variance is not a bug you can tune out. It is a property of the system. That distinction matters because it changes what a correct response looks like. You can’t fix it. You need to govern it.

The contract nobody disclosed

Enterprise software has always carried an implicit contract: behavior is repeatable, attributable, and auditable. SOX compliance assumes it. SLAs codify it. Acceptance testing verifies it. When your ERP runs payroll, you expect identical outputs for identical inputs. The entire compliance stack is built on that expectation.

When you deploy an LLM into a business process, you break that contract.

Gartner’s AI Hype Cycle 2025 explicitly calls nondeterminism a core enterprise AI risk and projects that more than 40% of agentic AI projects will be canceled by 2027. A separate Gartner prediction from March 2026 goes further: without explainability foundations, GenAI will be restricted to low-risk tasks only.

The internal consequence is concrete. Your finance agent interprets policy differently on Tuesday than it did on Monday. Your audit trail stops being a record and becomes a probability distribution. This Six Sigma Agent paper cites an MIT GenAI Divide Report finding that 95% of enterprise GenAI implementations fail to meet production expectations. That failure rate is not an implementation problem. It’s a mismatch between what the technology is and what organizations expect it to be.

Manage it like professional judgment, not like software

The right response to this is not to lower expectations. That’s too passive and too vague to be useful.

The right move is to adopt a different governance model entirely. One closer to how organizations manage professional judgment than how they manage software.

When you hire a consultant or engage outside legal counsel, you don’t write an SLA that promises identical advice on every engagement. You build in review gates, approval layers, and accountability structures. You treat the output as a recommendation, not a transaction. The answer might differ based on context and so you have a human in the loop before that recommendation becomes an action.

That is the correct mental model for an LLM. The Stochastic Gap paper from researchers at UMBC and MIT formalizes this. Using a Markov framework applied to a real procurement workflow across 251,000 cases, they show that the mismatch between deterministic enterprise workflow assumptions and probabilistic AI behavior is structural, not incidental.

The consultant analogy is useful but it only gets you so far. The more precise frame is professional judgment under institutional accountability. Law firms, audit practices, and medical institutions have spent decades building governance structures for exactly this problem. Expert output varies by practitioner, by day, and by context. It still needs to be defensible, traceable, and bounded by institutional policy.

Those structures share three characteristics that software governance does not.

Output is reviewed before it becomes a commitment. A legal opinion goes through a partner review before it’s sent. A diagnostic recommendation goes to an attending doctor before it reaches a patient. The expert produces the output; the institution validates it before it has consequences. Most LLM deployments skip this entirely. The model generates; the system acts. There is no partner review layer.

Variance is bounded by policy, not eliminated by process. You don’t solve variance in professional judgment by making every practitioner identical. You solve it by defining the boundaries within which variation is acceptable. Then you build escalation paths for cases that fall outside them. A tax advisor can give different guidance to different clients with different circumstances. That’s appropriate variance. An LLM giving different legal interpretations to identical queries in the same product is not. The governance question is whether you’ve defined the difference.

Accountability is personal and institutional simultaneously. When a lawyer gives bad advice, liability flows in two directions: to the individual practitioner and to the firm. Enterprise AI deployments are constructing a version of this whether they intend to or not. In 2024, a Canadian tribunal held Air Canada liable for incorrect information its chatbot gave a customer. The airline argued the bot was a separate legal entity outside their responsibility. The tribunal rejected that defense. Courts are already resolving institutional liability for AI output, and they are not resolving it in the operator’s favor. The question is whether your governance structure reflects that reality.

This Agent Drift paper introduces a useful concept here: the Agent Stability Index. It attempts to quantify behavioral consistency across model invocations. Not just whether the model gets the right answer, but whether it behaves in a predictable and bounded way over time. That’s the right unit of measurement for governance. Not accuracy but stability.

This reframes the deployment decision. The question is not “is this model accurate enough?” Accuracy is necessary but insufficient. The real question is: is this model stable enough, in a defined scope, with defined escalation paths, for this specific process? That’s how you evaluate a new hire in a professional services firm. It should be how you evaluate an LLM deployment.

Before deploying an LLM into any business process, three questions need answers:

1. Does this process require the same answer every time for legal, compliance, or contractual reasons?

2. Is there a human review layer between the LLM output and the consequential action?

3. If the LLM gives two different answers to the same question, which one are you liable for?

If you can’t answer those, you’re not ready to deploy. But there’s also a fourth question: who inside your organization owns the answer to all three? Software procurement has a buyer, a vendor, and a contract. Professional judgment has a supervising principal who is accountable for the output. LLM deployments currently have neither. They have a model, a prompt, and an assumption that someone else is responsible.

That assumption is what the courts are now testing.

The expectation mismatch has standing in court

This Air Canada case makes this concrete in a way that no whitepaper can.

In Moffatt v. Air Canada (BC Civil Resolution Tribunal, 2024), Air Canada’s chatbot gave a customer incorrect information about bereavement fares. The customer relied on it and tried to claim the discount. Air Canada’s defense was that the chatbot was a “separate legal entity” and not their responsibility.

The tribunal rejected this. Air Canada was held liable for the advice its AI gave.

The company deployed a probabilistic system into a customer-facing context where both the law and the customer assumed determinism. That expectation mismatch wasn’t just an internal governance failure. It had standing in court.

This is the trajectory for any business deploying LLMs into consequential interactions without understanding what they’ve actually built.

We are now in an era where companies are building legal and contractual liability on systems that cannot guarantee the same answer twice. The technology isn’t the problem. The category error is.

I’ve been building AI-powered features for the past several months, and one thing that became very clear very quickly is that token costs can’t be an afterthought. They’re a design constraint. When you’re running a healthcare platform with RAG pipelines, context-heavy prompts, and multiple LLM calls per user interaction, the bill is a direct reflection of how carefully you thought about what you’re actually sending to the model.

Most of the conversations I’ve seen about LLM cost optimization focus on the easy answer: pick a cheaper model. That’s the wrong starting point. Before you reach for a smaller model, you need to understand where your tokens are actually going. Otherwise, you’re optimizing blind.

First, understand your token budget

The single most useful shift in mindset is moving from “which model is cheaper?” to “how many tokens does this job actually need?”

Every LLM call has a token budget made up of roughly four buckets: system prompt, user/conversation input, retrieved context (if you’re doing RAG), and model output. Most engineering teams can tell you which model they’re on. Not many of them can tell you what percentage of their token spend is coming from each bucket, per endpoint, per feature, per user segment.

That’s the first thing to fix. Instrument your app to log tokens in and out at the endpoint level. You’ll almost always find that 20% of your calls are responsible for 80% of your token spend, and those hotspots are where optimization actually pays off. Spending a week shaving 10% off a low-volume admin workflow is a distraction. Shaving 30% off your highest-volume user-facing is much more meaningful.

The input side

Once you know where the tokens are going, the levers on the input side are fairly well-understood. The question is which ones are worth pulling.

Tighten your prompts. This sounds obvious but most system prompts accumulate cruft over time. Redundant framing, hedging language, politeness conventions the model doesn’t need. Audit yours with fresh eyes. “Summarize in 3 bullets” costs fewer tokens than a paragraph explaining what a good summary looks like. Directive beats descriptive.

Split your system prompt. If you have a large system prompt covering multiple capabilities, stop sending all of it on every call. Break it into a core section and optional sections, and load only what’s relevant for the task at hand. A prompt that’s always 3,000 tokens because it contains instructions for five features, when any given call only needs one of them, is something you can fix.

Tune your RAG retrieval. RAG is great, but undisciplined RAG drags enormous amounts of context into every call. Smaller chunk sizes, fewer retrieved chunks, a reranker to prioritize the most relevant passages, and deduplication to eliminate near-identical chunks all compound. You often don’t need half the document. You need the three paragraphs that actually answer the question.

Prune conversation history. Long chat threads accumulate fast. Rather than passing the full conversation on every turn, summarize earlier turns and keep a running state-of-the-world summary plus the last N interactions. The model doesn’t need a verbatim transcript of everything that happened three turns ago.

Preprocess before tokenizing. Strip content before it hits the model. Repeated disclaimers, HTML chrome, email signatures, standard headers. Anything that’s structurally consistent and informationally irrelevant to the task should be removed in code, not sent to the model and silently ignored.

The output side

Output tokens get less attention but they’re directly controllable.

Explicitly bound your responses. “At most 5 bullet points” or “under 150 words” is more effective than “be concise.” The latter is a suggestion; the former is a constraint. Use structured outputs (JSON, fixed schemas) wherever the output is machine-consumed. If you only need a label, a boolean, or a category, don’t ask for prose.

For user-facing answers, consider a layered approach: get a compact answer first, and only fetch a longer explanation on demand. Most users don’t need the full response most of the time.

The infrastructure layer

Beyond prompt engineering, there are a few architectural levers that can meaningfully change the economics.

Prompt caching. If your provider supports it, turn it on. Repeated system prompts and long shared documents shouldn’t be re-billed on every call. This is one of the highest-leverage, lowest-effort optimizations available. The work is mostly configuration, not engineering.

Semantic caching. For support questions, FAQs, and any domain with high query repetition, a semantic cache at the application layer can eliminate a large percentage of redundant model calls entirely. Vector similarity against past queries, reuse or lightly edit the cached response.

Model routing. Not all calls are equal. Route simple, well-defined tasks to smaller, cheaper models by default. Reserve the premium models for complex reasoning, high-stakes content, or tasks where quality differences are genuinely user-visible. The key is measuring this. A cheaper model that needs retries or follow-up calls can easily cost more than the premium model would have.

What doesn’t work

A few things that seem like good ideas and aren’t:

Adding “be concise” to every prompt and expecting large savings. It helps a little. The variance is high and the model will still expand if the rest of your prompt gives it room to.

Blindly truncating inputs. “Just take the first 4K tokens” sounds pragmatic but often drops critical context and degrades quality in ways that are hard to notice until something goes wrong.

Naive string compression: stripping punctuation, collapsing whitespace. This makes text less legible to the model and can actually increase token count due to how subword tokenization works. The model’s tokenizer and your intuitions about “shorter text” don’t always agree.

Swapping to a cheaper model without measuring tokens-per-task and quality first. A weaker model that requires multiple retries or produces output that needs downstream correction can easily end up costing more than staying on the better one.

A note on TOON and structured formats

There’s a newer format worth knowing about called TOON (Token-Oriented Object Notation). The idea is to strip JSON’s syntactic overhead, quotes, braces, repeated keys, by declaring keys once, like a header row. For flat, table-like data (user lists, product catalogs, RAG reference chunks, uniform agent outputs), benchmarks report roughly 30-60% prompt-token reduction with equal or slightly better accuracy on structured retrieval tasks.

It’s a real technique, but it comes with caveats. For deeply nested or irregular objects, the savings shrink or reverse. JSON is deeply embedded in LLM training data; TOON is a format the model hasn’t seen nearly as much, which introduces some fragility without fine-tuning. You’re also adding a serialization layer with converters, validation, and debugging in a nonstandard format that has real engineering cost.

The right framing for TOON: keep JSON in your APIs and storage. Use TOON only at the LLM boundary, and only when your prompts are heavy on structured data and the token savings are large enough to justify the complexity. It’s a specialized tool, not a universal one.

When not to optimize

For small documents, low-volume internal tools, or safety-critical tasks, it’s often better to overspend on tokens than to risk degraded quality or missed edge cases. Token optimization has a cost in engineering time, added complexity, and potential quality tradeoffs. The ROI only makes sense at the hotspots.

The teams that get this right aren’t the ones that optimize everything. They’re the ones that instrument first, identify the real cost drivers, and apply targeted effort where it makes the biggest difference.

Token spend is a design output. If you’re surprised by your LLM bill, the answer isn’t a cheaper model. It’s a closer look at what you’re asking the model to do, and why.

I run a competitive research workflow on a regular cadence. I use the same prompt, same tools, and the same intent. I want to map the landscape, surface new entrants and track what the competition is up to. The first few runs looked good. Then something started shifting. Not the market, our competitors in this space don’t move that fast. The outputs were different. Different companies surfaced. Different framing of the same players. Different conclusions from largely the same inputs.

The agent hadn’t broken. It was still producing plausible, well-structured research. It just wasn’t producing consistentresearch.

The gap between an agent that performs well and an agent that performs reliably is not something I hear talked about very often.

Why the demo always works

LLMs don’t retrieve. They generate. Every output is a sample from a probability distribution shaped by the prompt, the context window, and the model version. In a single-turn interaction this rarely matters. The variation is small and the output is usually close enough.

Agentic systems change the equation. Outputs become inputs to the next step. Tools get called, state accumulates, and decisions made in step two shape what is possible in step six. Small variations compound. The agent that searched slightly differently in step two is now summarizing different sources in step four. It draws different conclusions in step six. This is not a bug in any conventional sense. It’s the nature of the beast.

A demo is a single run. It proves the agent can do the task. It says nothing about whether the agent will do the task the same way tomorrow, after a prompt change, or after the underlying model gets quietly updated by the provider.

Reliability is not following capability

Recent research makes this concrete. A January 2026 simulation study from arXiv (arxiv.org/abs/2601.04170) defines agent drift as the progressive degradation of behavior, decision quality, and coherence over extended interactions. It identifies three distinct forms: Semantic drift is where outputs deviate from original intent. Coordination drift is where multi-agent coherence breaks down. Behavioral drift is where the agent develops unintended strategies over time. Using theoretical modeling across simulated enterprise workflows, the study projects that unchecked drift could lead to task success rates dropping by over 40% and human intervention requirements tripling. These are projected figures from simulation, not measured production outcomes. But the underlying framework for why the failure mode compounds rather than plateaus is well-constructed.

A large-scale empirical survey from UC Berkeley, Stanford, UIUC, and IBM Research (arXiv:2512.04123) gives the clearest picture of how practitioners are responding. Of 306 practitioners surveyed, 68% keep their deployed agents to at most 10 steps before human intervention. 70% adjust the prompts rather than fine-tuning the model. 74% rely primarily on human evaluation. The researchers frame this as a deliberate paradox: reliability is the top development challenge, yet agents are reaching production. The resolution is that teams are not waiting for the reliability problem to be solved. They ship by limiting what agents can do. Constrained autonomy, sandboxed environments, internal deployment first. It works. But it’s a workaround, not a solution.

Not all agents carry the same risk

Agents can be broken down into two main categories. The risk profile for each is different.

Bounded agents are invoked for a discrete task, produce an output, and hand off to a human or downstream process. Example include: A Cursor session writing a function. A one-shot document summary. The scope is defined. The output is reviewable. Failures are localized. These carry constant risk over time that is largely tied to human review.

Ambient agents run continuously and make ongoing judgment calls without a hard stop. For example, inbox triage or continuous competitive monitoring. Basically any workflow where the agent decides what matters and acts on it repeatedly, without a human checkpoint between decisions.

My competitive research workflow sits between these two. It is repeatable rather than truly continuous, but the expectation of consistency is the same. When I run it on Monday and again in three weeks, I expect the differences in output to reflect differences in the market, not differences in the agent. That’s not what happened.

McKinsey’s 2025 global survey found that 62% of organizations are at least experimenting with AI agents. Only 23% are scaling one in production. The gap between experimentation and scale is not a capability gap. It is a trust, observability, and governance gap.

If you’re running long-horizon agents, here’s what helps

The research and the emerging vendor landscape have converged on a set of mitigation approaches. None of them eliminate the problem but they can reduce risk.

Context management. One of the least visible failure modes in long-running agents is context drift. As conversation history grows, reasoning quality degrades before you ever hit a context limit. The industry has settled on episodic consolidation: periodically compressing older context into structured summaries while preserving recent and relevant state. The Agent Drift paper identifies this as one of three mitigation strategies with the strongest theoretical grounding. Anthropic now ships a native compaction API that automates the loop.

Uncertainty-aware memory. A January 2026 paper from Salesforce AI Research calls the core failure mechanism in long-horizon agents the Spiral of Hallucination. A small grounding error in an early step gets committed to the agent’s context. It then becomes a false premise for every subsequent step. Standard self-reflection does not reliably catch this. The model has already accepted the error as ground truth. The proposed fix flags low-confidence steps before they propagate and triggers correction only when needed. Early results showed meaningful reliability improvements on multi-step benchmarks. This is early research. But it is getting at the cause rather than the symptom.

Checkpointing and interrupt design. Orchestration frameworks like LangGraph have built explicit checkpointing into their execution model. Agents are defined as directed graphs with typed state and hard interrupt points. A human can review, approve, or reset to a known-good checkpoint at any of those points. This converts a brittle autonomous system into a collaborative one. Carnegie Mellon benchmarks published in late 2025 found that leading agents complete only 30-35% of multi-step tasks successfully. This shows that uninterrupted autonomous execution is not the right default for complex workflows.

Golden dataset evaluation. This approach maps most directly to my competitive research problem and our product work. Create a set of representative inputs with human-verified expected outputs. Then run your agent against that dataset on a schedule or before any prompt change goes to production. AWS introduced this at re:Invent 2025 with the general availability of Bedrock AgentCore Evaluations: 13 built-in evaluators, CI/CD pipeline integration for pre-deployment gates, and continuous online evaluation against live production traffic. A demo showed the service detecting tool selection accuracy dropping from 0.91 to 0.3 in production. Without continuous measurement, that degradation is invisible.

Pushpay documented a real production implementation of this pattern. Their golden dataset covers over 300 representative queries with validated responses. It is continuously curated from actual user interactions and fed into an engineering dashboard. The key word is continuously. A golden dataset that does not evolve with your actual workload tests against past state not current state.

Beyond AWS, the commercial tooling has matured fast. Braintrust ties production traces and offline experiments to the same scorer library. A production regression automatically seeds the next test cycle. LangSmith integrates human annotation queues with trace replay, letting engineers convert production failures into evaluation cases. Arize offers always-on drift detection at the session and span level. For teams with HIPAA or data residency constraints, Langfuse is the strongest self-hosted open-source option. It was acquired by Clickhouse in January 2026, but the open-source codebase remains active.

None of this is free. Building and maintaining a golden dataset requires human judgment to define what “correct” looks like for open-ended tasks. That is genuinely hard when correctness is partly subjective. Dataset rot is a real risk. The infrastructure to run evaluations continuously has real cost. The tooling can solve the infrastructure problem but the curation problem is still yours.

For my competitive research workflow, the approach is well-suited. The expected output structure is defined even if the specific content varies. I know what a well-formed competitive analysis looks like. I can score for completeness, source coverage, and structural consistency without specifying exact content in advance. That is an easier evaluation target than most ambient agent tasks.

The durability problem

The industry has gotten very good at demonstrating what agents can do. It has not gotten as good, so far, at ensuring they keep doing it the same way.

Gartner projects that 40% of agentic AI projects will fail by 2027. Poor risk controls are cited as a primary cause. That figure will land as a surprise to anyone who has only ever evaluated their agents at a single point in time.

Narrow, monitored, bounded agents are viable today if you build them with that constraint in mind. Always-on autonomous agents are still waiting on better reliability science, better evaluation tooling, and more organizational honesty about the governance they require.

The question worth asking before deploying any agent is not “can it do the task.” It is whether you can tell when it starts doing the task differently than it did before. And whether you would know before your users do.

Are you testing the wrong thing?

The interview process is supposed to predict job performance. When the conditions of the interview bear no resemblance to the conditions of the job, you’re not predicting anything. You’re running a different experiment and hoping the results transfer.

They don’t.

What we actually test when we ban AI

The stated rationale for AI-free coding interviews is reasonable on its surface. We want to see how candidates think. We want to know if they actually understand the problem. We don’t want someone to paste a prompt and copy the solution.

The concern is legitimate. The conclusion is wrong.

Here’s what an AI-free whiteboard session actually measures: the ability to hold syntax, algorithm structure, and edge cases in working memory simultaneously, under pressure, without tooling, in an artificial environment. That is a real cognitive skill. It just isn’t the one that determines whether someone will perform well on your team in 2026.

The cognitive load of software development has shifted. The job now requires a different kind of thinking: decomposing a problem into chunks small enough to prompt effectively, reading AI output critically before accepting it, knowing when the model is hallucinating and why. A candidate who can recite a BFS implementation from memory but blindly accepts a subtly wrong AI-generated solution is a worse hire than one who forgets the exact syntax but immediately spots the flaw in what the model produced.

We’ve been measuring recall. We should be measuring judgment.

The industry has figured this out

In October 2025, Meta began rolling out an AI-enabled coding interview that replaces one of the two traditional coding rounds at the onsite stage. The internal framing is revealing. The format was designed to be more representative of the actual developer environment and also to make LLM-based cheating less effective.

The problems Meta uses are designed with AI assistance in mind. They’re harder than a traditional coding question. The bar for what a candidate is expected to produce is higher precisely because they have help. Candidates work in a multi-file codebase they didn’t write and have to understand quickly. Prompt-and-paste fails immediately in that environment because understanding the existing architecture is the prerequisite for everything else.

CodeSignal launched AI-assisted coding assessments in May 2025 with a feature that matters more than the AI access itself: a full transcript of every candidate-AI interaction alongside a session replay. You’re not just seeing what the candidate produced. You’re watching how they think.

HackerRank moved in the same direction. Candidates work with AI tools in a controlled environment and interviewers get a detailed view of the problem-solving process, not just the output.

What an AI-free interview cannot show you

When I think about what I actually need to know about a candidate, four things matter that a no-AI interview cannot surface.

Prompt quality as a diagnostic. The way someone frames a problem for an AI is a direct readout of how they think about the problem. Vague prompts reveal vague thinking. A candidate who writes “fix my function” tells you something different from one who writes “this recursive function is hitting a stack overflow on inputs above n=1000, here’s the current implementation, what’s the likely cause.” The second candidate has already diagnosed the problem. They’re using the AI to confirm and implement. That’s engineering judgment.

Verification instinct. One E7 candidate at Meta watched Claude Sonnet repeatedly hallucinate on a maze problem. The question isn’t whether the AI was wrong. The question is whether the candidate caught it. Did they know what correct looked like before the model answered? Did they push back? A candidate who accepts wrong output without question is a risk that an AI-free interview will never expose, because you never gave them an AI to accept.

Task decomposition. Candidates who performed well in Meta’s AI-enabled format guided the AI incrementally rather than asking for wholesale solutions. One successful candidate described her approach: start with the core logic as a single function, review it, then build out from there. That instinct, to keep the scope small enough to verify at each step, is exactly how good engineers approach complex problems. It’s invisible in a no-AI environment because there’s nothing to decompose for.

Communication under ambiguity. Meta’s internal evaluation criteria for this round includes a phrase that functions as an answer key: “Should use AI, but need to show you understand the code. Explain the output. Test before using. Don’t prompt your way out of it.” That is a rubric for thinking, not for tool use. The candidate is being evaluated on whether they can narrate a reasoning process in real time, hold a conversation with an interviewer while working with an AI assistant, and remain the accountable decision-maker throughout. That skill matters on the job. AI-free interviews don’t test it.

The gaming problem is real but solvable

The obvious objection is that AI-assisted interviews are easier to game. A fast prompter with shallow understanding can look strong if the evaluation is just the output.

That’s true. But it’s a design problem, not a fatal flaw.

Interaction transcripts solve most of it. When you can see the full sequence of what a candidate asked for and how they responded to what they got, shallow prompters reveal themselves quickly. They ask for too much at once. They accept the first answer without testing it. They can’t explain the code when asked.

Multi-file codebases with staged checkpoints solve the rest. A problem that requires understanding existing architecture before making any change can’t be solved by pasting a description into a chat window. The AI doesn’t have the context. The candidate has to build it, which means they have to understand it.

The gaming risk in no-AI interviews is equally real and far less visible. A candidate who has memorized LeetCode patterns looks identical to one who genuinely understands algorithms. At least AI transcripts expose the reasoning process. A whiteboard session shows you the answer. A session replay shows you how the candidate thinks.

What to change before your next hire

None of this requires overhauling your entire process. Three things make the difference.

Redesign the problem before you change the rules. A standard LeetCode question with AI access is still a bad question. The problem needs to be complex enough that AI assistance is a navigation tool rather than a solution dispenser. Ambiguous requirements, existing codebases, staged checkpoints. Problems designed so that understanding is the prerequisite for prompting.

Make the AI interaction visible and gradable. If you’re using a platform, use one that captures transcripts. If you’re running your own interviews, ask candidates to narrate their prompts out loud and explain what the AI gave them before they act on it. That narration is the interview.

Keep a short no-AI segment with a clear purpose. Baseline fundamentals still matter. A candidate who can’t read a stack trace or reason about complexity without assistance is a real risk. A focused no-AI segment to test that floor is legitimate. Don’t treat that floor as the whole evaluation.

The prediction model you’re running

Every hiring process is a prediction model. The inputs are interview signals. The output is a forecast of job performance. When the inputs don’t reflect job conditions, the model is broken.

Running AI-free coding interviews for engineers who will spend their careers working alongside AI tools is like running driving tests with no steering wheel because you want to assess balance. The rationale sounds defensible. The instrument is wrong.

The industry has started correcting. The question is whether your process has.

If you’ve been using AI coding tools over the past two years, you’ve been getting a great deal. Frontier model access embedded in your IDE, powering your agents, running in your CI pipelines — for prices that don’t actually cover what it costs to serve you.

That’s not a bug. It’s a strategy. And strategies change.

This post is about what happens when the economics of AI-assisted coding get repriced, why that repricing is likely in the next 12 to 24 months, and what your engineering organization should be doing right now before the bill arrives.

The Free Ride Won’t Last

According to internal projections reported by the Wall Street Journal, OpenAI does not expect to reach profitability until 2030. Anthropic projects reaching positive free cash flow by 2027 or 2028. Both companies are growing revenue at extraordinary rates. Anthropic recently reported annualized revenue exceeding $30 billion. But revenue and profit are not the same thing, and right now these companies are very far from the same thing.

The structural problem is inference compute. OpenAI spent roughly 50% of its revenue on inference costs alone in recent years, with training costs pushing total expenditure well above what comes in. Every token you generate costs real money in GPU time. The pricing you see at the API does not reflect what it actually costs providers to serve those tokens.

Open-weight models of comparable capability are anywhere from 17x to 18x cheaper than Anthropic’s API pricing and are from providers who are covering their costs and making margins. That isn’t an indictment of Anthropic’s business model. It reflects real differences in model capability, trust, tooling maturity, and enterprise positioning.

The narrative you’ll hear most often is that inference costs will keep falling and everything will work out. That narrative has been repeated for three years. Inference costs for frontier models have not followed the curve that optimists projected, partly because each new model generation is larger and more capable than the last, which resets the compute baseline. Lower prices on last-generation models don’t help you if you need current-generation capability.

The current pricing environment is a competitive land-grab. It’s not sustainable.

The IPO Pressure Cooker

Both Anthropic and OpenAI are moving toward public markets. Anthropic has engaged IPO counsel and is reportedly discussing an offering as early as Q4 2026, targeting a raise exceeding $60 billion. OpenAI is targeting a similar timeline at a valuation approaching $1 trillion.

Private investors fund growth stories and tolerate long paths to profitability. Institutional fund managers running discounted cash flow models do not. The S-1 filing will contain actual unit economics for the first time. Analysts will model gross margins. Price-to-earnings ratios will matter in a way they don’t when you’re raising from VCs.

This creates a specific incentive. Both companies have strong motivation to show margin improvement before listing, not after. The levers available are cost reduction (hard, because compute costs are driven by usage and model scale) and price increases (a decision that can be made in an afternoon).

There’s also the Inference Trap: You build the best model, usage surges, inference compute explodes, and you face a forced choice between throttling users, raising prices, or cannibalizing the training compute you need to stay competitive. Anthropic experienced five major platform outages in a single month in early 2026. Claude Code users reported burning through usage allocations far faster than the pricing implied. That’s the Inference Trap operating in real time.

The combination of IPO pressure and Inference Trap dynamics makes a repricing event not just plausible but structurally likely. The question isn’t whether it happens. It’s whether you’re ready when it does.

What You’ve Actually Built On

Most engineering teams believe they have less AI vendor lock-in than they actually do. The assumption is: swap the API key, update the model name, done. There’s a lot more to it than that. It gets worse the deeper into agentic workflows you go.

The lock-in profile varies significantly by use case:

IDE-embedded tools like Copilot, Cursor, and Claude Code represent the shallowest lock-in. You could switch IDEs or model backends with little effort. But don’t underestimate soft stickiness. Developer muscle memory, .cursorrulescustomizations, team-shared system prompts, and workflow integrations all add switching friction. A price increase here hits developer productivity budgets, which are visible and politically sensitive.

Agentic coding workflows are where real lock-in begins. Agentic systems don’t just call a model — they build scaffolding around it. System prompts are tuned to a specific model’s personality and failure modes. Tool-calling schemas are optimized for how that model interprets them. Retry logic and output parsing are calibrated to observed behavior. When you switch models, that scaffolding doesn’t transfer cleanly. You’re not changing a config parameter. You’re running a re-evaluation campaign against your own codebase. Industry data suggests migration costs when provider lock-in forces a move average over $315,000 per project, and that figure reflects situations where teams already had some abstraction in place.

CI/CD and automated pipelines carry the highest risk. These are production systems with determinism requirements. Prompts optimized for one model may produce subtly different outputs on another. Those outputs can look similar enough to pass manual inspection but break downstream parsers and validation steps. Model version pinning provides a false sense of stability because providers deprecate models with 90 days’ notice, and there is no guarantee of behavioral equivalence between versions. The fundamental problem is that you cannot treat an LLM call in a production pipeline the same way you treat a deterministic function call. When you switch models, you have to prove the pipeline still works. You cannot assume it.

Open Source is a real option, but it has gaps

The obvious response to pricing risk is to use open source models, self-host, and pay for compute instead of markup. That path is more viable than it was 18 months ago but has real gaps that tend to be underestimated.

The capability gap has largely closed on many dimensions. Open models now match or surpass closed models on knowledge benchmarks, mathematical reasoning, and graduate-level science. The gap that remains is concentrated where it matters most for coding: production-level agentic tasks, multi-step software engineering, and complex tool use. On SWE-bench Verified, the most practically meaningful coding benchmark, the best open models are within a few points of frontier closed models. That gap is still an issue at the tail of task complexity.

The price differential is big. DeepSeek V3.2 is available at roughly $0.28 per million input tokens. Claude Opus 4.7 is $5.00 per million input tokens. That’s a 17x difference. For high-volume workloads the economics are compelling even accounting for operational overhead.

But here’s what the open source advocates undersell: switching models is not the same as switching model providers. The scaffold matters as much as the model. Real-world benchmarks show a 22-point swing on the same task with the same model when you change the agent scaffold and tooling. Switching models requires re-validating your entire system, not just verifying the model output looks reasonable.

The operational burden of self-hosting is a real cost transfer. Inference infrastructure, model serving with tools like vLLM or Text Generation Inference, GPU provisioning, update cadence, and security patching all fall on your team. For most organizations without dedicated ML infrastructure experience, this isn’t a savings. It’s a new operational surface area.

There’s also a geopolitical dimension worth naming directly. The strongest open models right now (DeepSeek, Qwen, Kimi) are Chinese-developed. For organizations with data sovereignty requirements, government contracts, or security-sensitive codebases, the lineage of a model matters. This isn’t a reason to dismiss these models outright, but it’s a factor that belongs in your architecture decision.

The Protocol Layer Is Your Best Friend

The most practical near-term lever against lock-in isn’t switching to open source. It’s building an architecture that makes switching possible.

Model Context Protocol (MCP) is the most significant structural development here. Originally developed by Anthropic and then donated to the Agentic AI Foundation (AAIF). This foundation was co-founded by Anthropic, Block, and OpenAI. MCP has achieved something rare: genuine cross-industry adoption. OpenAI abandoned their proprietary Assistants API and adopted MCP. Google DeepMind, Microsoft, and AWS are all on board. When direct competitors converge on a shared infrastructure standard it signals inevitability.

MCP decouples the agent-tool connection layer from the model layer. Your integrations with databases, APIs, filesystems, and external services are built once against the MCP standard and survive a model swap. That’s the right layer to standardize at.

Pair that with an LLM Gateway such as LiteLLM or Portkey, middleware that abstracts provider-specific API differences behind a single interface, and you get a system where the model backend is genuinely swappable without rebuilding your application logic. The marginal complexity cost of adding this abstraction early is low. The switching optionality it creates is high.

Be honest about what MCP doesn’t solve though. The protocol handles tool integration, not model behavior. When you swap models, your prompts still need re-validation. MCP can also consume 40-50% of available context window before any actual work begins, which creates real production tradeoffs. Standards help. They don’t eliminate the work.

What You Should Do Today

The cost of acting on this now is low. The cost of acting after a pricing shock is high.

For IDE tools: Evaluate whether your current tooling is model-agnostic or model-bundled. Prefer tools that let you swap backends. Baseline your developer productivity metrics now. You need a measurement baseline before any changes hit, not after.

For agentic workflows: Add an LLM Gateway from the start of any new project. Keep your agent orchestration layer architecturally separate from your model API calls. This is the single highest-leverage structural decision you can make. Build evaluation suites against your own codebase, not generic benchmarks. Generic benchmarks tell you how a model performs in the abstract. Your eval suite tells you whether you can safely swap models in your specific system.

For CI/CD pipelines: Treat every LLM call as a third-party dependency with explicit versioning, SLA monitoring, and a tested fallback path. Design for graceful degradation. What does the pipeline do when the model endpoint is slow, unavailable, or has been updated? This should be a documented decision, not an untested assumption.

Across all use cases: Audit your current AI spend and its concentration across providers. Most teams have no idea what this number is. Monitor the IPO timelines. The S-1 filings will be the first time the public sees actual unit economics from these companies, and they will move the conversation. Build internal familiarity with at least one open-weight model family. Even if you never deploy it in production, that knowledge reduces the information asymmetry in any future pricing negotiation.

A Strategy Note, Not a Panic Note

The goal here is not to abandon frontier models. They are genuinely better at certain tasks, the tooling ecosystem around them is more mature, and for many use cases the productivity gains justify whatever they end up costing.

The goal is not to be surprised. More specifically, the goal is to avoid being in the position of needing to move urgently with no alternatives evaluated and no time to build them.

Engineering organizations that have done the architecture work to reduce switching costs will have options when prices move. They’ll be able to make a deliberate choice between absorbing the increase, substituting a capable alternative, or negotiating from a position of real leverage. Organizations that haven’t done this work will face a different situation: urgent need, unknown switching cost, and a vendor who knows it.

The bill is coming. The amount is unknown. The only variable you control is how ready you are to pay someone else instead.

Jack Dorsey and Roelof Botha published “From Hierarchy to Intelligence” on March 31, 2026. It is a serious piece of thinking. The historical framing is sharp, the diagnosis of why hierarchies exist is largely correct, and the argument that AI changes the information-routing constraint is real.

But the essay is about one-third of what managers actually do. It treats that one-third as the whole job, removes the people doing it, and calls the problem solved. The other two-thirds are still there. They just don’t have anyone doing them anymore.

To understand that better, we need to be precise about what managing actually is.

Three Clusters, Not One

In 1973, Henry Mintzberg published The Nature of Managerial Work, based on direct observation of what managers do with their time. His finding was that managerial work falls into three clusters: informational, interpersonal, and decisional.

The informational cluster is what Dorsey and Botha are talking about. Managers monitor what’s happening, disseminate that information across the organization, and represent the team to the outside world. This is the information routing function. It’s the layer that hierarchy was built to support, and it’s the layer that AI can now perform continuously, at scale.

That’s true. But it’s only one of three.

What AI Cannot Route

The interpersonal cluster covers things that depend on trust, relationship, and human accountability. The leader motivates and develops people, figures out who has potential and what they need to grow, and has the difficult conversations.

The liaison role involves building relationships across organizational boundaries, the kind of connective tissue that makes cross-functional work actually work. The figurehead role is about legitimacy and accountability. When something goes wrong, someone needs to be responsible in a way a system cannot be.

A telling data point from the Block restructuring itself: current and former employees told The Guardian that roughly 95% of AI-generated code changes still require human modification. This is in a remote-first, highly digital, machine-readable organization, exactly the environment that Dorsey describes as most amenable to this model. The humans are still in the loop not because the information system failed, but because the work itself requires judgment that isn’t in the data.

The decisional cluster is where this becomes even clearer. Mintzberg’s entrepreneur role involves sensing and acting on opportunities that aren’t visible yet. A world model, by definition, can only reflect what has already happened. It cannot tell you what to build that doesn’t exist yet. The disturbance handler role is about responding to crises and genuinely novel situations, exactly the circumstances where pattern-matching on historical data is most likely to fail. Resource allocation and negotiation involve competing interests, trust between parties, and judgment under uncertainty. In regulated industries, financial services, healthcare, any domain with fiduciary obligations, you can’t delegate these decisions to a system regardless of how good it is.

The Risks Worth Naming

Beyond the functional gaps, there are a few things worth noting:

Post-hoc rationalization?. Block cut 40% of its workforce in February 2026, before the essay was published. The stock jumped roughly 22%. Botha, who co-authored the essay, sits on Block’s board. Morgan Stanley upgraded Block to overweight after the cuts. Goldman Sachs raised its price target. It is fair to ask whether the intellectual framework followed the business decision or preceded it. That doesn’t make the argument wrong, but it does mean the incentive to believe the argument is quite strong for the people making it.

The flat structure graveyard. Zappos tried holacracy. Valve famously ran without managers. The Spotify model has been widely adopted and widely struggled with. These experiments didn’t fail because the idea was wrong in theory. They failed because removing formal structure doesn’t remove the need for coordination. It just moves coordination into informal channels, where it becomes invisible, political, and dependent on whoever has the most social capital. The information routing problem gets solved. The interpersonal and political problems get worse.

Data completeness. A world model built from Slack threads, Jira tickets, pull requests, and performance metrics reflects what was written down. A significant fraction of organizational knowledge is never written down. It lives in the judgment calls that didn’t make it into a doc, the context a senior engineer carries about why a system was built the way it was, the reason a decision was made three years ago that nobody remembers to explain to new people. The model sees the artifact. It doesn’t see the reasoning behind it.

Data quality and drift. This one is distinct from completeness and is arguably more dangerous. Information that was accurate when it entered the system becomes stale. The system continues to present it with the same authority as fresh data. Decisions get made on information that was true six months ago and isn’t anymore. You don’t see the error at the time. It shows up later, in ways that are very hard to trace back to their source.

This is a documented, recurring failure in knowledge management systems generally. It’s not theoretical. Platforms like Guru have built their core product differentiation specifically around the knowledge freshness problem because the industry learned, repeatedly, that drift is the default. Small errors accumulate in decisions that each look reasonable in isolation, until something downstream breaks in a way nobody can explain.

Regulatory reality. For companies operating in financial services or healthcare, the question of whether AI can replace decision-making isn’t just organizational. It’s legal. Explainability requirements, fair lending law, fiduciary duty, these don’t care how good your model is. Decision accountability cannot be delegated to a system in regulated domains. This alone may prevent this type of World Model from being used is certain industries.

Where This Is Actually Going

None of this means the Dorsey/Botha thesis is wrong. I just think it’s incomplete.

The informational cluster is being automated. That is a real and permanent shift. Managers who spent the majority of their time aggregating context, relaying status, and maintaining alignment across teams are already less necessary than they were. That part of the argument makes a lot of sense.

What’s more interesting is what happens to the other two clusters as this plays out. If information routing gets absorbed by AI systems, the interpersonal and decisional work doesn’t disappear. It becomes more visible and it’s value should be more obvious. Managers who survives this transition are the ones who were always doing the work that was hardest to put on a job description.

The open question is whether that work changes in character as organizations become more AI-instrumented, or if it simply becomes more prominent because everything else has been stripped away. Does managing people become fundamentally different when the coordination layer is automated? Or does it turn out that the relational, developmental, and judgment-intensive work was always the real job, and the information routing was just the overhead we confused for the substance?

I don’t think anyone knows yet. Block’s Q1 2026 results will be a first data point. If they hit $12.2 billion in gross profit with 40% fewer people, the thesis gets harder to argue with. If they don’t, their Roman Army comparisons will age badly.

Either way, the question is worth asking more carefully than “can AI replace the org chart.” The org chart was never the point. It was just the structure we built to solve three problems at once. AI can solve one of them. The other two? Let’s see.

I’m reading C. Thi Nguyen’s new book, The Score, and one idea in it keeps coming back to me. He calls it value capture,the process by which a rich, meaningful goal gets quietly replaced by the metric you were using to track it. You start measuring something because it points toward what matters. Then, gradually, the measurement becomes what matters. The original goal doesn’t disappear. It just stops being the thing that drives decisions.

That’s not a philosophy problem. That’s every day in most engineering organizations.

What we actually measure

Let’s be clear about what passes for engineering measurement in most companies.

Output metrics are the default: story points, tickets closed, pull requests merged, lines of code written. They’re easy to collect, easy to visualize, and they feel like signal. The problem is they measure production, not value. A team can close 200 tickets in a sprint and ship nothing a customer cares about. Story points aren’t a unit of value, they’re a unit of negotiated effort, and that negotiation starts the moment someone decides to track them.

Performance metrics try to go one level deeper: code review turnaround time, sprint commitment hit rate, on-call response time, test coverage percentages. These are more interesting because they reflect process health. But they still measure fidelity to a process, not effectiveness of the work. A team can hit 95% sprint commitment every week by sandbagging estimates. Reviews can be fast because nobody’s actually reviewing.

Efficiency metrics are where most engineering organizations have landed recently, particularly DORA, the four-key-metrics framework: deployment frequency, lead time for changes, change failure rate, and time to restore service. DORA is genuinely useful as a diagnostic. The problem is what happens when it moves from a team-level health check to a leadership dashboard. Deployment frequency gets gamed by trivializing deployments. Lead time gets gamed by where you start the clock. You end up with a team deploying 15 times a day that is still six months from shipping anything meaningful.

Every one of these metric categories lives entirely inside the engineering system. None of them has a direct connection to whether the engineering organization is actually doing its job.

The game you didn’t know you were playing

Nguyen’s book makes a distinction that hits differently in an engineering context. He separates striving play from achievement play. In striving play, the goal is to engage fully, the process, the judgment, the craft. In achievement play, the only thing that matters is the score. Great games are designed so that chasing the score also produces striving. You can’t get good at chess by gaming the scoring system; you actually have to get good at chess.

Institutional metrics are the opposite. They strip out the magic circle, Nguyen’s term for the temporary, voluntary frame that makes game constraints feel meaningful rather than oppressive. In a board game, you accept arbitrary rules because you chose to sit down and play. In a work context, those rules aren’t arbitrary and they aren’t optional. The score follows you. It shows up in your performance review. It gets presented to the board.

What’s left, once you remove the magic circle, is a system that rewards achievement play. And engineers, who are, professionally, some of the best problem-solvers in any room, find the optimal path to the score. This isn’t cynicism. It’s a completely rational response to the incentive structure you built.

The consequences are predictable:

Velocity becomes sandbagging. The moment team velocity appears on a leadership dashboard, estimation inflates. Points expand to protect the team. After a few quarters, the number is politically stable and informationally useless.

Deployment frequency rewards triviality. If deploying frequently is the metric, the rational move is to break work into smaller pieces, not because small batches are better (they often are, but for different reasons), but because each deployment ticks the counter.

Commitment rates reward conservatism. Measure whether a team delivers what they promised and they’ll promise less. You’ll see consistently green dashboards and a team that’s quietly becoming slower.

Code review speed becomes rubber-stamping. If time-in-review is visible, reviewers learn to approve fast. Technical debt accumulates invisibly while the metric looks healthy.

This is Goodhart’s Law in action, once a measure becomes a target, it ceases to be a good measure. But Nguyen’s framing adds something important: it’s not just that the metric gets corrupted. It’s that people’s values get reshaped around it. The engineers optimizing for velocity aren’t lying. They’ve internalized the metric. The metric has become, for them, what good work looks like. That’s value capture. That’s the thing that’s actually hard to fix.

What these metrics are actually telling you

This is worth being precise about, because the answer isn’t “nothing.” These metrics have legitimate uses. The failure is the mismatch between what they measure and what leaders use them to decide.

Output metrics, velocity and tickets closed can tell you if work is flowing through the system at all. A team whose velocity drops 40% in two sprints has a problem worth investigating. What they can’t tell you is whether the work matters.

Performance metrics are operational diagnostics. Long code review cycles, high defect escape rates, chronic on-call fatigue, these are real signals about process dysfunction. Treat them as process indicators, not as performance scorecards.

DORA is pipeline health. Lead time and deployment frequency tell you something real about delivery capability. Change failure rate and MTTR tell you something about resilience. But all four are downstream of the question that actually determines whether an engineering organization is performing: are you building things that move the business, fast enough to matter?

None of the standard metrics reach that question. The reason is structural. They’re easy to collect because they live inside the engineering toolchain. Anything that requires a connection to product outcomes, customer behavior, or business results is harder, and that difficulty is precisely why it tends not to get measured.

The missing layer

The data we reach for first, the data that’s easy to collect and easy to present, systematically hides what’s actually going on. The measurement layer that’s missing isn’t more engineering metrics. It’s a feedback loop that closes outside engineering.

A few things that would actually tell you something:

Outcome linkage. Can you connect a shipped feature to a measurable change in user behavior or business results? Not “we shipped it” but “after we shipped it, the thing it was designed to move, moved.” This requires instrumentation, a documented hypothesis before work starts, and a willingness to wait. None of those come naturally to sprint planning cadences. But without them, you’re measuring production, not impact.

Flow efficiency. The ratio of value-added time to elapsed time in your delivery process is more interesting than raw velocity. A feature that takes 12 weeks from idea to production, with only 2 of those weeks involving actual engineering work, has a flow efficiency of 17%. That’s a systems problem — and throughput metrics will never surface it. The bottleneck isn’t in the work; it’s in the waiting.

Technical health as a leading indicator. Complexity trends, dependency staleness, incident frequency — these are imperfect but directionally useful signals about whether your codebase is getting easier or harder to extend. Engineering organizations that ignore these tend to see velocity collapse right when the business needs them to accelerate. It’s not a coincidence.

Team capability over time. Are engineers growing? Are they retaining? Are they increasingly autonomous, or increasingly dependent on a few specialists? These lag badly as indicators, but they’re leading indicators of whether the organization will still be functional in two years. No sprint metric captures them.

Breaking the cycle

Swapping bad metrics for better ones doesn’t solve the problem. If you replace velocity with DORA and keep the same incentive structure, you’ll get gamed DORA metrics in six months. Nguyen is clear on this in The Score, once value capture takes hold, the answer isn’t a better score. It’s rebuilding the conditions under which people can reclaim their own values.

In an engineering context, that means a few things:

Separate diagnostic metrics from evaluation metrics. Deployment frequency is useful when a team uses it to understand their own pipeline. It becomes corrosive the moment it appears on a leadership report as a proxy for team performance. The same number. Completely different effect depending on who it’s for and what it drives.

Measure outcomes and accept the latency. This requires leaders to resist the urge to instrument everything that can be instrumented. Define what you’re trying to move before work starts. Measure it after you ship. Accept that the feedback loop is slower than a quarterly review cycle, and if your performance management cycle is shorter than your product feedback cycle, that’s the real problem to fix.

Make the system visible, not just the throughput. Flow efficiency, incident trends, and technical debt trajectories give teams and leadership a shared picture of systemic constraints. When the conversation shifts from “why didn’t you close more tickets” to “what’s blocking flow and what would it cost to fix it,” you’re at least asking questions that can produce useful answers.

Let the teams define the metrics for their own work. This one is underrated. The people closest to the work know what signals matter and which ones can be gamed. Metrics designed by a team to understand their own performance are completely different from metrics imposed from above to evaluate them. The former creates accountability. The latter creates the conditions for value capture.

Hold leadership accountable for outcome clarity. A lot of engineering metric gaming exists because the business hasn’t clearly defined what success looks like. If product leadership can’t say what a feature is supposed to change, and how they’ll know it worked, the engineering team will fill that vacuum with whatever proxy feels safest. Measurement quality is a leadership problem as much as a measurement problem.

The question the dashboard can’t answer

Look at whatever engineering metrics you’re currently tracking. For each one, ask: what decision would I make differently if this number were 20% better or worse? If the answer is “I’d evaluate someone’s performance differently,” the next question is whether you’re actually measuring what you want to optimize for, or just what’s available.

Nguyen ends The Score with the question the title comes from: is this the game you really want to be playing? Most engineering organizations never ask it. The dashboard is there. The numbers are green or red. The sprint review happens. Nobody stops to ask whether the game itself is worth playing.

Measuring the wrong things precisely is worse than not measuring. It gives you the confidence of data without the benefit of insight. The metric becomes the goal. The goal becomes the metric. The original purpose, building something that matters, gets quietly replaced. And everyone in the room is looking at the dashboard, wondering why the product still isn’t getting better.

When writing code using AI assistance, it’s easy to generate a lot of code in a very short amount of time. As the development process goes on, even more code will be created. This gets complicated when it comes to maintaining and expanding the code base. Over time the person working on the code has to get back into that code to address bugs or add features. Getting to the point where you understand the code well enough, again, takes time. I have seen that while it’s easy to create a lot of code with AI, it’s not as easy to come back into that code after it’s been created.

One thing we need to keep in mind when using AI to generate code is that now there becomes a difference between writing the code and understanding the code. In pre-AI software development, where you needed to manually write the code, there was a certain understanding of what you were writing while you were writing it. With AI, that process goes away. The code is created, and you then need to understand that code and become familiar with it so that you can then make updates.

When you write the code yourself, your brain is creating an internal model of that system that you’re creating. When you step away from that code and come back, some of that that internal model, will persist. Interacting with something, rather than just reading it, tends to be retained in the brain longer. Since you’ve written the code yourself, it’s easier to come back into that code and continue developing. If you have not gone through that process, if you are reviewing code that was written by AI or by another person, then you’re establishing familiarity with the code each time without having that pre-existing internal model. This makes it more difficult and takes longer to come up to speed with that code so you can continue development.

Protecting Comprehension

The answer isn’t to use AI less. It’s to recognize that AI handles code generation but cannot handle comprehension. Comprehension doesn’t transfer. Each of the techniques below are just a mechanism for forcing you to build a genuine internal model of the code, not just review it from the outside. If you spend 2 hours building something with AI that would have taken 2 days manually and then spend 45 minutes re-establishing context every time you return to it, the break-even on that speed gain comes faster than you think.

1. The spec-first inversion

The conventional flow is: prompt → code → review. Reverse the order. Before asking the AI to generate anything, write a 3–5 sentence description of what the code should do and why. Not a formal spec — just externalizing your intent. That document becomes your re-entry anchor, and it forces the model to work from your understanding rather than producing something you have to reverse-engineer.

2. Treat code review as active encoding, not QA

When you review AI-generated code, the goal shouldn’t just be catching bugs — it should be deliberately building the internal model you didn’t get from writing it. Don’t rubber-stamp diffs. Walk the logic path. Rename things that don’t reflect your mental model. The review is the learning; skip it and you’re adding to the context debt, not paying it.

3. The commit message as cognitive snapshot

Write commit messages as if you are explaining the change to yourself six months from now — not what changed, but why and what you understood about the system at the time. This is cheap, async, and gives future-you a scaffold that’s tied to the exact moment of maximum context. Git log becomes a context recovery tool, not just a change log.

4. AI-assisted re-entry

Use AI to rebuild context, not just create code. Paste the module into a fresh session and ask: “Explain what this does, what it assumes, and what would break if X changed.” That AI-generated explanation — corrected where wrong — becomes your working model. You’re using the same tool that created the debt to help pay it down.

What we’re seeing with AI assisted software development is that it is not a panacea. AI is changing the mental effort required to maintain understanding over time. It’s not just about how quickly we can create the code. It’s about being able to sustain that comprehension. The key takeaway here is to use AI to accelerate coding, but don’t outsource the understanding of that code. It needs to be understandable and comprehensible over time by humans after the code is written.