At WWDC this month, Apple introduced a piece of software it calls the system orchestrator. It routes every AI request to the model that should handle it, on the device or in the cloud, based on how much compute the task needs and how sensitive the data is. Craig Federighi called it the key to the system’s privacy architecture.

That routing decision is the most consequential architectural shift in consumer AI since the transformer. Not the model. The decision about where the model runs.

Cloud by default was never a decision

From 2022 through 2024, anything that looked like reasoning went to an API. GPT-4, Claude, Gemini all lived in central data centers. On-device AI meant autocomplete and spell check. The hardware floor on a phone was too low to run a capable model, so the model lived somewhere else.

The constraint felt permanent because it had always been true but Always has does not mean always will. We made the same error with local storage, with video on mobile, with encryption in the browser. The thing that is too expensive to do locally stays in the data center right up until it doesn’t.

The floor is already higher

Look at what already left the cloud. Real-time transcription runs on your phone. Translation runs on your phone. Photo understanding, system-wide dictation, short-form classification, writing suggestions. None of these phones home anymore. Apple’s second on-device model, the more capable one for current Apple silicon, understands speech, reads text and images, and never leaves the device.

We should measure the floor by what it can do, not by its parameter count. In 2023 the floor was “can it follow an instruction.” In 2026 the floor is “can it handle most single-document knowledge work without a network connection.”

The reason this happened so fast is the efficiency curve underneath. Sparse activation, distillation, and aggressive quantization decoupled capability from cost. A sparse model carries a large parameter count but uses only a fraction of those weights on any given token. The headline says twenty billion parameters. The actual work on a prompt might touch a few billion. That gap is the trick that put a real model on a phone.

The floor doesn’t rise on a smooth line. It jumps when a new efficiency technique lands and then sits flat until the next one. Mixture-of-experts was a jump. Quantization down to four bits was a jump. Nobody knows the slope of the next five years but he direction is obvious.

Routing is the product, not the model

The orchestrator was the most underreported thing Apple showed. The press wrote about Siri’s new voice. The architecture was the story that was glossed over.

What does the orchestrator decide? Latency tolerance, privacy sensitivity, task complexity, and the hardware tier of the device in your hand. Send the easy local thing local. Send the hard private thing to Private Cloud Compute. This is not a new problem in infrastructure terms. A CDN decides whether to serve from the edge or the origin. A database decides whether a read hits a replica or the primary. Routing a request to the cheapest box that can satisfy it is the way we’ve always done it.

The new part is what the routing decision means. A CDN serving a cached asset gives you the same asset. An orchestrator routing to the small model gives you a different answer than the big model would have. The routing decision now changes the quality of the intelligence, not just the latency. That is a category of routing we haven’t built governance for.

Whoever owns the routing layer owns the usage signal. The model that gets called is the model that sees the data, collects the feedback, and improves fastest. Apple makes that decision for more than a billion devices.

This is how Apple played it. The next-generation foundation models are being co-developed with Google’s Gemini, and the cloud tier runs on Nvidia GPUs inside Google’s cloud. Apple did not insist on owning the best model. It rented the model and kept the layer that decides which model you reach. The model is rentable. The routing layer is not. This is the key. If somewhere down the road, Apple decides to switch to their own cloud based AI, they can. This is what happened with mapping software, Apple partnered with Google until it developed its own. Who knows if Apple will develop their own models but if they do, they can turn it on very quickly.

The open question is whether the orchestrator stays an honest dispatcher or becomes a toll booth.

Regulation is fragmenting AI by geography

Consumer privacy preference is real and soft. People say they care and then tap accept. Regulation does not depend on what anyone prefers. It is hard, and it is fragmenting AI deployment along national lines.

Look at the Siri example: At WWDC, Apple announced that Siri AI will not ship in the EU with iOS 27. Apple framed the block as privacy and security. The cause is actually the Digital Markets Act, which is competition law, not privacy law. The European Commission rejected Apple’s framing in public and without hedging. Nothing in the DMA blocked the launch. The delay was Apple’s own choice. The real objection is that Siri AI, which is powered by Google’s Gemini, would lock out competing assistants on a platform that reaches hundreds of millions of users. The Commission’s spokesman named the Gemini detail on purpose.

This is not “privacy law is forcing AI onto devices”. It’s deeper than that. AI deployment is fragmenting by jurisdiction, and the pressure comes from several different bodies of law at once. Competition law in Brussels. Data localization in India, China, Saudi Arabia, and Brazil. A model that runs locally sidesteps an entire class of these fights because the data never crosses a border. The on-device model is not just faster and more private. It is jurisdictionally invisible.

This is where sovereignty stops being a slogan. A regulated enterprise or a government does not want a privacy assurance about someone else’s data center. It wants the weights, the inference stack, and the audit trail inside its own borders. Private Cloud Compute, however well it is engineered, is still someone else’s cloud in someone else’s country. The requirement is on-premise open-weight deployment. Models like Llama and Mistral are important here for a reason that has nothing to do with topping a benchmark. You can run them where the law says the data has to stay.

US export controls accelerate all of this. A country that cannot count on access will build or source its own. The recent retraction of Anthropic’s Fable 5 model is real world example of this. Scarcity is a fast teacher of self-reliance. The open question is whether on-device and on-premise can satisfy the provenance and audit demands that regulators will have.

Losing the commodity tier is not the threat

The tasks leaving the cloud were never the high-margin tasks. The person summarizing email on a phone was not paying for an API call. Losing that load costs the frontier labs close to nothing.

The real exposure is the middle. Function-level code completion. Document question-and-answer. Single-session knowledge work. These are mid-complexity and high-volume, and they are moving on-device, likely inside two years. The timing will vary based on the use case.

The floor does something quieter to the rest. When good-enough AI is free, private, and already in your pocket, every upgrade has to clear a higher bar to earn a dollar. Apple’s orchestrator teaches a billion people that AI is the free thing that already lives on the phone. That reshapes what AI is worth in the median person’s head before any frontier lab gets to make its pitch.

Volume and revenue are not the same curve. A frontier provider can lose sixty percent of its query volume and far less than sixty percent of its revenue, as long as it priced the commodity tier near zero in the first place. The query that left was the query they weren’t making money on anyway.

So who is actually exposed? Not the top labs but the mid-tier API resellers and the vertical SaaS products whose only moat was access to commodity inference. When the inference is free and local, the wrapper has nothing left to sell. The move for the labs at the top is to go up. Own the reasoning and the agentic layer. Concede the commodity tier without a fight, because fighting for it means competing with free. The open question is whether “own the reasoning layer” is a durable position or just the next floor waiting to rise under it.

The grid did not die. Its economics did.

The useful precedent is electricity, not software. As the electrical grid matured, generation moved closer to the load. Rooftop solar, home batteries, and the central grid now coexist. Nobody decommissioned the grid. What changed permanently was the economics of building another central plant when the marginal load could be served closer to home.

The analogy holds here too. Local handles the frequent, ambient, low-stakes load. Central handles the peak, the complex, the high-stakes work. Neither one kills the other.

But here is where the analogy breaks: Electricity is fungible. A kilowatt-hour off your roof and a kilowatt-hour off the grid are the same kilowatt-hour, and your refrigerator cannot tell the difference. Intelligence is not fungible. The answer from the local model and the answer from the frontier model are different answers, sometimes meaningfully different, and the user often cannot tell which one they got.

The analogy gives us the shape and then fails at the load-bearing joint. The floor rising does not make the ceiling matter less. It makes the gap between floor and ceiling the thing you are selling. The product stops being “AI.” The product becomes the difference between the AI everyone has for free and the AI that is worth paying for. Pricing that gap, and proving it exists on any given task, is the whole game now.

Where do we go from here?

We can see the floor rising. We can see the gap becoming the product. What’s still unknown is who governs the routing layer.

Today Apple decides for Apple devices. Google decides for Android. Microsoft decides for Windows Copilot. None of these is neutral infrastructure. Each one is a company that also sells models, which means each routing decision is made by a party with a stake in the outcome.

The flywheel is the important part. The model that gets called gets the usage signal. The usage signal becomes the improvement that makes it the model worth calling next time. Whoever holds the orchestrator holds the flywheel. Right now the platform owners are holding it, and they did not have to buy a single frontier lab to do it.

If routing gets captured at the platform layer, the frontier labs become wholesale suppliers to a retail counter they do not own. They will have built the best engine in the world and handed the dashboard to someone else.

Morph LLM published a study in March 2026 with a claim that every frontier model they tested gets worse at reasoning as you give it more to read. The output quality drops in a way you can measure and reproduce, and it happens on the exact models you are paying for. The instinct is to file this under “models are unpredictable” and move on. That instinct is the mistake this post is about.

Let’s start with the symptom, because you have probably already seen it. A multi-stage agent runs cleanly through step three. It reasons correctly, calls the right tools, returns what you expected. By step twelve the same agent, on the same task type, with the same model and the same prompt pattern, is producing degraded output. Nothing about the task got harder. Something about the context got heavier. That gap, between a system that works early and the same system failing late, is what I want to talk about. It’s called context rot, and it’s architectural, which is the part that matters. You can’t upgrade your way out of an architectural property.

The spec sheet measures the wrong thing

When a vendor prints 128K on the box, that number is a ceiling on what the model will accept. It says nothing about what the model can reason over reliably. Those are different quantities, and the gap between them is enormous.

Paulsen and colleagues gave the distinction names in September 2025. The Maximum Context Window is the spec sheet figure. The Maximum Effective Context Window is the point where accuracy meaningfully degrades for a given task. Their findings show that some top models showed severe degradation by 1,000 tokens. A few broke down with as little as 100 tokens in context. Across everything they tested, the effective window fell short of the advertised one by as much as 99 percent.

That’s a big difference. A 128K window can deliver reliable reasoning over something closer to 1K on a hard task. The other 127K is tokens the model will dutifully process and cannot actually use well.

The effective limit also moves with the job. A flat retrieval task and a multi-hop reasoning task over the same corpus have different effective windows, because reasoning over scattered evidence stresses attention harder than fetching one fact. So there is no single number to memorize. You need to figure out for yourself how it will impact your workflow.

The RAM comparison is exact, not loose. A machine with 128GB of memory does not run every workload 128 times faster than a 1GB machine. The number is capacity, not throughput. Context windows are the same. We have been reading a capacity spec as a performance spec, and the vendors are happy to let us.

Why attention thins out

Three mechanisms compound as context grows. You want the mechanisms, not just the result, because the mechanisms tell you what to do.

Position is not neutral. Liu and colleagues at Stanford showed transformer attention follows a U-shape across token positions. The model attends strongly to the start and the end of its context and weakly to the middle. In a 20-document QA task, accuracy dropped more than 30 percent when the relevant document sat in positions 5 through 15 versus position 1 or 20. The same fact, the same model, moved a few thousand tokens inward, and the model half-forgets it is there. Models trained explicitly on long context still showed it. So the first thing to internalize: where a token sits changes whether the model can use it most effectively.

Attention is a fixed budget split more ways. The attention mechanism distributes a finite amount of probability mass across every token in context. Add tokens and each one gets a thinner slice. This is not a tuning artifact. It falls out of the softmax math. More context does not mean more attention to go around. It means the same attention spread thinner, and past some point the model can no longer cleanly separate the signal token from the noise around it.

The curve flattens early. A model reading 100K tokens is not getting ten times the benefit of one reading 10K. Positional undertraining and the constraints of rotary position embeddings mean effective utilization grows far slower than the token count. The useful curve bends down well before the nominal limit. You are paying linearly for tokens whose value is already sub-linear.

Put the three together and the picture is not “long context is a bit lossy.” It is that the middle of a large context is a place where information goes to be underweighted, and you are filling it on purpose.

Agent loops manufacture their own noise

Static document Q&A is forgiving. You load a corpus once and ask questions. Agent loops are much less forgiving because they generate the very noise that rots them.

Watch what accumulates in a long-running loop. Verbose tool outputs, most of which were only partially relevant. Reasoning paths the model explored and abandoned. Intermediate states that are no longer true. And the one almost everyone forgets: prior turns where the user chose between options.

That last one deserves a deeper look, because it is the clearest illustration of the whole problem. The model offers two options. You pick the first. The second option is not dropped. It sits in context carrying the same weight as the path you chose. Ten turns later the model is reasoning over a growing pile of decisions you already killed. The dead ends do not decay. They sit alongside the live thread, drawing from the same thinning attention budget, and the model has no way to know they are dead unless you tell it.

This is why Morph LLM’s work, which folded in Chroma’s research on coding agents, landed on context rot as the primary failure mode for agentic coding. Not model capability. Not reasoning quality. The models are capable enough. The context they are handed is too noisy, and most of the noise is self-inflicted.

The diagnostic is precise once you know where to look. Degradation that scales with session length rather than task complexity is context rot. If your agent clears step three and fails step twelve with no change in task type, stop tuning the prompt. You are not looking at a capability problem. You are looking at a hygiene problem.

The two curves cross

Everything so far is about quality. The cost side is where it stops being an abstraction and starts showing up on the bill.

The KV cache scales with context length, and at high token volumes that scaling is not gentle. Latency climbs. VRAM consumption grows. Throughput falls. None of these track token count proportionally. They accelerate.

Think about how that pairs with the quality finding. The cost per useful unit of work is rising at exactly the point where each token is delivering less. This is not ordinary diminishing returns, where you pay the same for a bit less. The cost curve bends up while the quality curve bends down. Those two curves cross. Past the crossing point you are paying more to get worse output, and nothing in the system will let you know you’ve crossed it.

Mixture-of-Experts architectures make the diagnosis harder, not easier. They can hide this dynamic underneath infrastructure bottlenecks, so the system looks healthy right up until it is obviously not.

Manage it like memory, because that is what it is

The bottom line is that context is a resource with a budget, and right now most systems don’t see it that way. Here are three things you can do:

Budget context against your effective window, not the spec sheet. Estimate the effective window for your task type and set a token budget per loop or session against that number, not the advertised one. When you hit 60 to 70 percent of the budget, trigger a checkpoint. Make the checkpoint a structured state object, not a prose summary: decisions made, options rejected, current state. Then continue from the checkpoint instead of the full transcript. You are choosing what survives into the next turn rather than letting accumulation choose for you.

Prune the dead weight. A tool call that returned nothing useful should be stripped or compressed before the next turn. A rejected option should collapse to a single line. The model does not need the full text of a path it is not taking, and every token you remove is a token that can no longer dilute attention on the tokens that matter. Do this as part of the loop.

Measure degradation as a function of length, not just at turn one. Run your evals at turns 5, 10, and 20, not only at the start. Watch accuracy as context grows on your specific task. If you cannot state your effective window as a number, you do not know where your system fails, which means you are flying without a parachute.

A bigger window is not the solution. The Morph LLM data spans GPT-4.1, Claude Opus 4, and Gemini 2.5 Pro. Every one degrades. The window sizes differ. The architectural property does not. A larger window buys you more room to accumulate noise, not more reasoning.

What the vendors are not selling you

Context window size has become the spec that vendors compete on, the way hardware vendors once competed on raw clock speed. The number is real. What it implies about useful work is a different claim, and it’s the claim you don’t tend to hear about.

The engineers who build reliable long-context systems will not be the ones with the largest windows. They will be the ones who treat context the way a systems engineer has always treated memory: as a scarce resource with budgets, checkpoints, and active reclamation. The model will not do this for you. It cannot garbage collect its own context. It has no concept of which tokens are dead. That judgment, what to keep and what to get rid of, is the part that does not come in the box, and it’s up to you to figure out.

In December 2025, the Google Cloud CTO office published a post-mortem on what they called the “reliability gap” in agentic AI systems. The core finding was not subtle: the move from deterministic software to probabilistic agentic workflows introduces failure modes that most teams only discover after architecture decisions are locked. The paper described multi-step agent chains operating without transaction coordination, creating non-atomic failures that can produce irreversible side effects and data corruption mid-operation.

That is not a model quality problem. It is a structural design problem. And it is expensive in two ways that most AI cost models don’t account for.

The bill arrives after the architect has left

LLM inference costs scale with volume and context length. Everyone knows this. Where it hets more complicated is when you chain agents together.

Every step in a multi-agent workflow triggers at least one model call and also, potentially, retries. Context lookups trigger more. Add a reflection layer, an orchestration agent, a validation pass, and your per-transaction cost can explode.

This comes about because the costs don’t start to mount until it gores into production and the builders have moved on. By the time you see this, it’s too late.

The vendors selling agent frameworks are not incentivized to surface this. Their pricing models are based on consumption. A platform that makes it trivially easy to chain five agents together is probably not going to ship a cost impact calculator so you know what you are getting into. You discover the problem at the quarterly cloud invoice review.

Right sizing is not hard to figure out; deterministic execution for stable, high-volume paths, and probabilistic reasoning reserved for genuinely ambiguous edge cases where it creates real value. Many teams reach for an agent because the demo was impressive, not because the task required reasoning under uncertainty.

Regulation does not grade on a curve

The EU AI Act imposes documentation, transparency, and auditability obligations on high-risk AI systems, with phased enforcement running through 2027. At it’s most basic level: a system subject to these rules must be able to reconstruct its decision logic. It must be able to answer, on demand, why a particular output was produced.

A live LLM agent in the execution loop can’t answer that question. The reasoning is not stored. The attention weights are not auditable. The path from input to output runs through billions of parameters and produces no human-readable trace of its logic.

A deterministic workflow that was generated by an AI can answer that question. The code exists. The execution path is traceable. The logic is inspectable.

This distinction matters beyond Europe. Any organization operating in healthcare, finance, insurance, or legal already faces existing sector regulation with equivalent requirements. HIPAA audit trails. SEC recordkeeping. Insurance claim adjudication documentation. These obligations don’t go away because you replaced a rules engine with an agent. They get stronger, because the regulator now wants to understand the AI component specifically.

The compliance teams asking these questions are not being obstructionist. They are pointing at a real architectural gap that the engineering team may not have modeled as a risk.

Trust is already declining

Augment Code’s 2025 developer survey found that trust in AI-generated code accuracy dropped from 40% to 29% in a single year. That is not a rounding error. Adoption grew over the same period, reaching 62% of developers, but confidence in the output fell sharply.

More developers are using AI. Fewer of them trust what it produces. Developers are shipping AI-assisted work while privately applying a larger mental discount to its reliability. That discount has a cost. It shows up as increased review time, more defensive testing, slower iteration on the critical paths where the stakes are highest.

Enterprise teams that have moved to hybrid architectures report a different pattern. When agents set goals and orchestrate tasks but critical computations run inside deterministic modules, the LLM surface area at runtime shrinks. The auditability problem gets smaller because the probabilistic component is doing less. Data shows teams using this approach achieving over 80% reduction in manual effort while maintaining the traceability their compliance functions require.

What the architecture decision actually is

Treating atomicity as an infrastructure requirement instead of a prompting challenge changes what you build. It means designing agent chains with transaction boundaries, rollback logic, and explicit failure modes before you write the first prompt. It means asking, for every step in the workflow, whether this step requires reasoning under uncertainty or whether it requires reliable execution of a known procedure.

Most enterprise AI architectures are not designed this way. They are grown from demos. The agent that worked in the proof of concept becomes the agent in production, with more context, higher volume, and a compliance audit scheduled for Q3.

The Google Cloud paper put it directly: non-atomic failure modes create irreversible side effects. That is an infrastructure problem. Changing the system prompt does not fix it.

The question I keep coming back to is whether the hybrid architecture pattern will be adopted proactively or reactively. Some teams will design for it from the start. Most will discover the need for it after the first incident that cannot be explained to a regulator, or the first quarter where inference costs exceed the value the system generated.

Nobody set out to discover a better architecture for AI systems. It happened because someone typed “write me a function that parses this JSON” and the result just worked.

The LLM ran once. It interpreted the intent, produced code, and exited. The code ran in CI, in production, on a schedule. No LLM in the loop. Nobody called it an “AI agent.” They called it software.

This was not a product decision. It was not an architectural principle. It was the path of least resistance. The developer got what they needed and moved on. The LLM was done before the workflow started.

That accident of framing contains the most important architectural insight in AI systems today. Most teams building agentic workflows have completely missed it.

The hammer is real. The grip is the problem.

The capabilities are genuine. An LLM can interpret ambiguous human intent, navigate unstructured input, and produce structured output from near-infinite input variations. Those are hard problems. LLMs solve them well.

The mistake is assuming that because an LLM can do something at design time, it should keep doing it at every execution. On every single run.

Maslow’s famous observation was that if your only tool is a hammer, every problem looks like a nail. The AI industry’s current posture is a precise instance of this. We have a remarkable tool. We are applying it to every stage of the pipeline, including stages where it has no business being.

The problem is not that LLMs are unreliable. The problem is that probabilistic systems do not belong in deterministic execution paths.

What “probabilistic in the execution path” actually costs

The math is unforgiving. A 20-step agentic workflow where each step succeeds 95% of the time produces an end-to-end success rate of 36%. This is not a model quality problem. Better models improve the per-step number. They do not change the compounding structure. The architecture is the problem.

Kambhampati et al. established the theoretical foundation for this in their ICML 2024 paper on LLM-Modulo frameworks. Auto-regressive LLMs cannot, by themselves, do planning or self-verification. The LLM is a probabilistic knowledge source. Treating it as a deterministic executor is a category error.

Mehta’s recent consistency research on SWE-bench sharpens this further in a way most people have not fully processed. Across 150 agent trajectories, the finding was not that inconsistent models fail more. It was that consistency amplifies outcomes rather than guaranteeing them. Claude achieved 58% accuracy with 15.2% behavioral variance. When it interpreted a task correctly, it succeeded on 100% of runs. When it got the interpretation wrong, it failed on 100% of runs. The same wrong answer, five times in a row, with high confidence.

A model making the same incorrect interpretation on every run is worse than a randomly failing one. The failure is invisible until it compounds, and reflection-based recovery cannot help if the initial interpretation is wrong.

ReliabilityBench (Gupta, 2026) adds the production dimension. Agents achieving 96.9% accuracy on clean benchmarks drop to 88.1% under realistic perturbations. The paper’s broader finding: metrics on clean data overestimate production reliability by 20 to 40 percent. The gap is not a model problem. It is a measurement problem that reveals an architectural one.

Better models improve the per-step number. They do not change the compounding structure.

The pattern developers discovered by accident

Code generation is the canonical LLM-as-compiler case. The LLM interprets intent and emits a deterministic artifact. That artifact runs without the LLM. Nobody designed this as an architecture. It emerged because “write me code” has a natural exit condition: the code either works or it does not.

The insight generalizes far beyond code. A policy document can become an OPA rule set. A process description can become a Temporal workflow definition. Decision criteria can become a scoring model. The LLM’s job in each case is translation from human intent to a deterministic artifact. Once the artifact exists, the LLM is done.

Khattab et al.’s DSPy framework from Stanford is the explicit formalization of this idea. Compile the prompt and retrieval logic into an optimized, repeatable program rather than invoking raw LLM calls at runtime. The artifact owns the execution. The LLM’s work happens once, at design time.

The reason this pattern is underused outside of engineering contexts is not that it does not apply. It is that business users do not yet have the intuition that “the AI is finished” is a valid and preferable state. Finished means auditable. Finished means version-controlled. Finished means you can write a test for it.

When the LLM does belong in the path

There is a class of workflow where LLM-as-compiler fails. The failure mode is unstructured with novel input at execution time.

Customer support is the clearest case. You can compile decision trees for known issue categories. You cannot compile a response to a customer describing a problem you have never seen, in language you cannot predict, with emotional context that changes the appropriate answer.

The same logic applies to open-ended document classification where the category space is not closed; to real-time anomaly detection on free text; to any workflow where the output depends on semantic understanding of content that did not exist when the workflow was built.

The LLM belongs in the execution path when the input space at runtime is genuinely open-ended. That is a much smaller category than most people think.

The question is not “can an LLM handle this at runtime?” The question is “does this workflow require runtime interpretation, or was the interpretation problem already solved at design time?”

The decision rule

Can you fully describe the execution logic as a flowchart, rule set, or decision tree before the workflow runs?

If yes, the LLM’s job is to build that artifact. Not to approximate it on every execution.

If the answer is no, because the logic depends on content that cannot be anticipated or because the decision space is genuinely open, then the LLM belongs in the path.

The key to reliability is to apply this test at all times and minimize LLM surface area at runtime. Not because LLMs are bad. Because deterministic systems are auditable, testable, version-controllable, and do not compound errors across steps.

Most teams are not asking this question. They are asking “how do we make the agent more reliable?” That is the wrong starting point.

If you are running an LLM on every execution of a workflow that could have been compiled into a deterministic artifact, you have not built an AI system. You have built a reliability problem with an AI-shaped interface.

The first question to ask is not whether your LLM is good enough. It is whether it needs to be there at all.

In October 2023, New York City launched an AI chatbot to help small business owners navigate city regulations. The mayor’s office called it a frictionless doorway into City Hall. Five months later, investigative journalists at The Markup tested it and found it was confidently telling landlords they did not have to accept Section 8 housing vouchers. That is illegal in New York City. The same bot told employers they could take a cut of their workers’ tips. Also illegal. When ten journalists asked the same housing question independently, all ten got the same wrong answer.

The city added a disclaimer. The mayor defended it and kept it running. The next administration eventually called it “functionally unusable” and moved to shut it down. The bill for building it: roughly $600,000.

Nobody asked the right question before launch. Not “can we build this” but “does a probabilistic text system belong in the execution path of legal regulatory guidance, where there is exactly one correct answer per question and the person asking has no way to know when they’ve received the wrong one.”

That question, whether AI belongs in a given workflow at all, is the one most organizations skip. They go straight to “how do we make this work” and treat the premise as settled. It isn’t always.

The question teams are not asking

When an AI deployment struggles in production, the instinct is to reach for better models, more data, or tighter prompt engineering. The conversation becomes technical quickly. That is the wrong reflex.The right first question is simpler and more uncomfortable: should AI be in this workflow at all?Not “can it be done” but “should it be done here, in this position, with these consequences attached.”Many teams never ask this. Gartner projects that 40% of enterprise applications will feature task-specific AI agents by 2026. MindStudio found that only 23% of companies run fully autonomous agent systems despite 88% already applying AI somewhere. The gap between “we use AI” and “we have thought carefully about where AI runs” is huge. That gap is where the expensive mistakes live.

Blast radius is the first variable

The first question to ask about any workflow is: what happens when this output is wrong?

Not if. When. Every probabilistic system produces wrong answers. The question is what those wrong answers cost.

A wrong answer in marketing copy gets caught in review and fixed in ten minutes. A wrong answer in a financial reconciliation can trigger a downstream payment workflow before any human sees it. A wrong answer in a regulatory guidance tool may prompt a landlord to illegally reject a tenant, with no visibility until a fair housing complaint lands.

This is blast radius. It is not a measure of how often the system fails. It is a measure of how bad a single failure can get. High blast radius workflows require a different standard of scrutiny before AI gets anywhere near the execution path.

The NYC chatbot did not fail because the underlying model was bad. It failed because the blast radius of wrong legal guidance is real harm to real people. The city had no way to know how many business owners had already acted on the bad advice before The Markup published its findings.

Detectability is the second variable

The second question is: how would you know the output was wrong, and how quickly?

This one is underweighted in almost every AI deployment conversation I have seen. Teams spend enormous effort on accuracy and almost no effort on error visibility.

Some errors are self-evident. A generated email that addresses the recipient by the wrong name gets caught before it sends. Some errors are invisible. A landlord who received incorrect guidance from an official city tool has no reason to doubt it. The tool carried government branding. It answered confidently. The correct answer was on a different page of the same website, and nobody told them to go look there.

The combination that kills enterprises is low detectability paired with high blast radius. That is the dangerous quadrant. Errors that are hard to see and expensive when they land. That is where autonomous agents are being deployed right now at scale.

The EU AI Act, phased in between 2024 and 2027, is not primarily about model accuracy. It is about auditability. Regulators are not asking whether your AI is usually right. They are asking whether you can prove what it did and why. That is a detectability requirement disguised as a compliance requirement.

Variance is only a feature in the right context

The third question is: does the right answer change between runs, or is there one correct answer?

Large language models are non-deterministic by design. They produce different outputs given the same input. For some tasks that variance is the point. Generating five options for a campaign headline and picking the best one is a workflow where variance creates value.

For other tasks variance is a defect. A financial reconciliation has one correct answer. A patient’s medication dosage has one correct answer. A contract clause either complies with local law or it does not.

The NYC chatbot illustrated this with unusual clarity. Whether a landlord is legally required to accept a housing voucher is not a matter of interpretation. There is a correct answer. Yet the bot gave different answers at different times and gave the wrong answer to all ten journalists who tested it simultaneously. A system producing variance in a zero-variance workflow is not a system having a bad day. It is a system that was never fit for the task.

Variance only creates value where the space of acceptable outputs is large and the evaluator is a human who can use judgment to select. Anywhere the evaluator is a downstream system, a database constraint, or a person who treats official guidance as authoritative, variance is a bug.

Downstream systems change the risk profile entirely

The fourth question is: is there a system consuming this output directly, without human review in between?

This is the difference between a soft consumer and a hard consumer. A human reading an AI-generated summary is a soft consumer. They bring judgment. They catch obvious errors. They can push back.

An API endpoint that ingests an AI output and triggers an action is a hard consumer. It has no judgment. It cannot catch errors. It will execute faithfully on whatever it receives.

The NYC chatbot consumer was human but effectively hard. A small business owner consulting an official government tool on a compliance question is not going to fact-check the answer. The government branding collapsed the soft consumer into something that behaved like a hard one. The output fed directly into a decision with no meaningful review layer in between.

When AI feeds a hard consumer, the error characteristics of the system change completely. Errors no longer surface through human review. They surface through system failures, audit findings, or in this case, investigative journalism.

Compliance exposure is not a legal problem

The fifth question: is this workflow subject to audit or regulatory review?

This gets treated as a legal team concern. It is not. It is an engineering and product concern that shows up as a legal problem later.

Regulated workflows require explainability. They require audit trails. They require the ability to reconstruct exactly what the system did and on what basis. Current LLM architectures are not designed to produce that by default. You can build it in, but it requires deliberate design choices made before deployment, not retrofitted after a regulator asks a question.

The EU AI Act classifies certain AI uses as high-risk. Legal and regulatory guidance is squarely in that scope. High-risk classification does not mean prohibited. It means the compliance burden is significant enough that you need to answer all five questions before deployment, not after the headline.

The real question behind the questions

Run these five questions against any workflow you are evaluating for AI automation:

1. What happens if this output is wrong? (blast radius)

2. How would you know it was wrong, and how quickly? (detectability)

3. Does the right answer change between runs or is there one correct answer? (variance tolerance)

4. Is there a downstream system or a trusting human consuming this output directly? (soft vs. hard consumer)

5. Is this workflow subject to audit or regulatory review? (compliance exposure)

A workflow with high blast radius, low detectability, zero variance tolerance, a hard consumer, and regulatory exposure is not a workflow where you deploy an AI agent today.

Most businesses deploying LLMs are treating them as a new category of software. The governance model they’re importing assumes a simple contract: same input produces same output, every time, reliably enough to stake compliance on. For example: procurement checklists, SLAs, acceptance testing, and audit trails.

LLMs are probabilistic. This breaks the model.

The setting that doesn’t do what you think it does

The standard response to LLM nondeterminism is to set temperature to zero. That’s supposed to make the model deterministic. It doesn’t.

Researchers at ACL’s Eval4NLP workshop published empirical results showing accuracy variations of up to 15% across runs with supposedly stable settings. The best-to-worst performance gap on some tasks reached 70%. Thread scheduling, parallel inference, mixture-of-experts routing, and server-side model updates all introduce variance that temperature settings cannot touch. Temperature=0 means you asked for determinism. It does not mean you got it.

Now chain three agents together, each operating at 90% reliability. Your combined system accuracy is around 73%. Add a fourth agent and you’re below 65%. Probabilistic error compounds through a pipeline the same way latency does. Princeton’s Language and Intelligence Lab studied real-world agent failures against benchmark performance and found this gap is systematic. Not anomalous.

This is not fixable with better prompting. It’s architectural.

Temperature is just the most visible example of a broader pattern. Every instinct your team has about fixing nondeterminism will reach for a deterministic lever: tighter prompts, stricter validation, more retries etc. These are reasonable engineering responses to a problem that is not an engineering problem. The variance is not a bug you can tune out. It is a property of the system. That distinction matters because it changes what a correct response looks like. You can’t fix it. You need to govern it.

The contract nobody disclosed

Enterprise software has always carried an implicit contract: behavior is repeatable, attributable, and auditable. SOX compliance assumes it. SLAs codify it. Acceptance testing verifies it. When your ERP runs payroll, you expect identical outputs for identical inputs. The entire compliance stack is built on that expectation.

When you deploy an LLM into a business process, you break that contract.

Gartner’s AI Hype Cycle 2025 explicitly calls nondeterminism a core enterprise AI risk and projects that more than 40% of agentic AI projects will be canceled by 2027. A separate Gartner prediction from March 2026 goes further: without explainability foundations, GenAI will be restricted to low-risk tasks only.

The internal consequence is concrete. Your finance agent interprets policy differently on Tuesday than it did on Monday. Your audit trail stops being a record and becomes a probability distribution. This Six Sigma Agent paper cites an MIT GenAI Divide Report finding that 95% of enterprise GenAI implementations fail to meet production expectations. That failure rate is not an implementation problem. It’s a mismatch between what the technology is and what organizations expect it to be.

Manage it like professional judgment, not like software

The right response to this is not to lower expectations. That’s too passive and too vague to be useful.

The right move is to adopt a different governance model entirely. One closer to how organizations manage professional judgment than how they manage software.

When you hire a consultant or engage outside legal counsel, you don’t write an SLA that promises identical advice on every engagement. You build in review gates, approval layers, and accountability structures. You treat the output as a recommendation, not a transaction. The answer might differ based on context and so you have a human in the loop before that recommendation becomes an action.

That is the correct mental model for an LLM. The Stochastic Gap paper from researchers at UMBC and MIT formalizes this. Using a Markov framework applied to a real procurement workflow across 251,000 cases, they show that the mismatch between deterministic enterprise workflow assumptions and probabilistic AI behavior is structural, not incidental.

The consultant analogy is useful but it only gets you so far. The more precise frame is professional judgment under institutional accountability. Law firms, audit practices, and medical institutions have spent decades building governance structures for exactly this problem. Expert output varies by practitioner, by day, and by context. It still needs to be defensible, traceable, and bounded by institutional policy.

Those structures share three characteristics that software governance does not.

Output is reviewed before it becomes a commitment. A legal opinion goes through a partner review before it’s sent. A diagnostic recommendation goes to an attending doctor before it reaches a patient. The expert produces the output; the institution validates it before it has consequences. Most LLM deployments skip this entirely. The model generates; the system acts. There is no partner review layer.

Variance is bounded by policy, not eliminated by process. You don’t solve variance in professional judgment by making every practitioner identical. You solve it by defining the boundaries within which variation is acceptable. Then you build escalation paths for cases that fall outside them. A tax advisor can give different guidance to different clients with different circumstances. That’s appropriate variance. An LLM giving different legal interpretations to identical queries in the same product is not. The governance question is whether you’ve defined the difference.

Accountability is personal and institutional simultaneously. When a lawyer gives bad advice, liability flows in two directions: to the individual practitioner and to the firm. Enterprise AI deployments are constructing a version of this whether they intend to or not. In 2024, a Canadian tribunal held Air Canada liable for incorrect information its chatbot gave a customer. The airline argued the bot was a separate legal entity outside their responsibility. The tribunal rejected that defense. Courts are already resolving institutional liability for AI output, and they are not resolving it in the operator’s favor. The question is whether your governance structure reflects that reality.

This Agent Drift paper introduces a useful concept here: the Agent Stability Index. It attempts to quantify behavioral consistency across model invocations. Not just whether the model gets the right answer, but whether it behaves in a predictable and bounded way over time. That’s the right unit of measurement for governance. Not accuracy but stability.

This reframes the deployment decision. The question is not “is this model accurate enough?” Accuracy is necessary but insufficient. The real question is: is this model stable enough, in a defined scope, with defined escalation paths, for this specific process? That’s how you evaluate a new hire in a professional services firm. It should be how you evaluate an LLM deployment.

Before deploying an LLM into any business process, three questions need answers:

1. Does this process require the same answer every time for legal, compliance, or contractual reasons?

2. Is there a human review layer between the LLM output and the consequential action?

3. If the LLM gives two different answers to the same question, which one are you liable for?

If you can’t answer those, you’re not ready to deploy. But there’s also a fourth question: who inside your organization owns the answer to all three? Software procurement has a buyer, a vendor, and a contract. Professional judgment has a supervising principal who is accountable for the output. LLM deployments currently have neither. They have a model, a prompt, and an assumption that someone else is responsible.

That assumption is what the courts are now testing.

The expectation mismatch has standing in court

This Air Canada case makes this concrete in a way that no whitepaper can.

In Moffatt v. Air Canada (BC Civil Resolution Tribunal, 2024), Air Canada’s chatbot gave a customer incorrect information about bereavement fares. The customer relied on it and tried to claim the discount. Air Canada’s defense was that the chatbot was a “separate legal entity” and not their responsibility.

The tribunal rejected this. Air Canada was held liable for the advice its AI gave.

The company deployed a probabilistic system into a customer-facing context where both the law and the customer assumed determinism. That expectation mismatch wasn’t just an internal governance failure. It had standing in court.

This is the trajectory for any business deploying LLMs into consequential interactions without understanding what they’ve actually built.

We are now in an era where companies are building legal and contractual liability on systems that cannot guarantee the same answer twice. The technology isn’t the problem. The category error is.

I’ve been building AI-powered features for the past several months, and one thing that became very clear very quickly is that token costs can’t be an afterthought. They’re a design constraint. When you’re running a healthcare platform with RAG pipelines, context-heavy prompts, and multiple LLM calls per user interaction, the bill is a direct reflection of how carefully you thought about what you’re actually sending to the model.

Most of the conversations I’ve seen about LLM cost optimization focus on the easy answer: pick a cheaper model. That’s the wrong starting point. Before you reach for a smaller model, you need to understand where your tokens are actually going. Otherwise, you’re optimizing blind.

First, understand your token budget

The single most useful shift in mindset is moving from “which model is cheaper?” to “how many tokens does this job actually need?”

Every LLM call has a token budget made up of roughly four buckets: system prompt, user/conversation input, retrieved context (if you’re doing RAG), and model output. Most engineering teams can tell you which model they’re on. Not many of them can tell you what percentage of their token spend is coming from each bucket, per endpoint, per feature, per user segment.

That’s the first thing to fix. Instrument your app to log tokens in and out at the endpoint level. You’ll almost always find that 20% of your calls are responsible for 80% of your token spend, and those hotspots are where optimization actually pays off. Spending a week shaving 10% off a low-volume admin workflow is a distraction. Shaving 30% off your highest-volume user-facing is much more meaningful.

The input side

Once you know where the tokens are going, the levers on the input side are fairly well-understood. The question is which ones are worth pulling.

Tighten your prompts. This sounds obvious but most system prompts accumulate cruft over time. Redundant framing, hedging language, politeness conventions the model doesn’t need. Audit yours with fresh eyes. “Summarize in 3 bullets” costs fewer tokens than a paragraph explaining what a good summary looks like. Directive beats descriptive.

Split your system prompt. If you have a large system prompt covering multiple capabilities, stop sending all of it on every call. Break it into a core section and optional sections, and load only what’s relevant for the task at hand. A prompt that’s always 3,000 tokens because it contains instructions for five features, when any given call only needs one of them, is something you can fix.

Tune your RAG retrieval. RAG is great, but undisciplined RAG drags enormous amounts of context into every call. Smaller chunk sizes, fewer retrieved chunks, a reranker to prioritize the most relevant passages, and deduplication to eliminate near-identical chunks all compound. You often don’t need half the document. You need the three paragraphs that actually answer the question.

Prune conversation history. Long chat threads accumulate fast. Rather than passing the full conversation on every turn, summarize earlier turns and keep a running state-of-the-world summary plus the last N interactions. The model doesn’t need a verbatim transcript of everything that happened three turns ago.

Preprocess before tokenizing. Strip content before it hits the model. Repeated disclaimers, HTML chrome, email signatures, standard headers. Anything that’s structurally consistent and informationally irrelevant to the task should be removed in code, not sent to the model and silently ignored.

The output side

Output tokens get less attention but they’re directly controllable.

Explicitly bound your responses. “At most 5 bullet points” or “under 150 words” is more effective than “be concise.” The latter is a suggestion; the former is a constraint. Use structured outputs (JSON, fixed schemas) wherever the output is machine-consumed. If you only need a label, a boolean, or a category, don’t ask for prose.

For user-facing answers, consider a layered approach: get a compact answer first, and only fetch a longer explanation on demand. Most users don’t need the full response most of the time.

The infrastructure layer

Beyond prompt engineering, there are a few architectural levers that can meaningfully change the economics.

Prompt caching. If your provider supports it, turn it on. Repeated system prompts and long shared documents shouldn’t be re-billed on every call. This is one of the highest-leverage, lowest-effort optimizations available. The work is mostly configuration, not engineering.

Semantic caching. For support questions, FAQs, and any domain with high query repetition, a semantic cache at the application layer can eliminate a large percentage of redundant model calls entirely. Vector similarity against past queries, reuse or lightly edit the cached response.

Model routing. Not all calls are equal. Route simple, well-defined tasks to smaller, cheaper models by default. Reserve the premium models for complex reasoning, high-stakes content, or tasks where quality differences are genuinely user-visible. The key is measuring this. A cheaper model that needs retries or follow-up calls can easily cost more than the premium model would have.

What doesn’t work

A few things that seem like good ideas and aren’t:

Adding “be concise” to every prompt and expecting large savings. It helps a little. The variance is high and the model will still expand if the rest of your prompt gives it room to.

Blindly truncating inputs. “Just take the first 4K tokens” sounds pragmatic but often drops critical context and degrades quality in ways that are hard to notice until something goes wrong.

Naive string compression: stripping punctuation, collapsing whitespace. This makes text less legible to the model and can actually increase token count due to how subword tokenization works. The model’s tokenizer and your intuitions about “shorter text” don’t always agree.

Swapping to a cheaper model without measuring tokens-per-task and quality first. A weaker model that requires multiple retries or produces output that needs downstream correction can easily end up costing more than staying on the better one.

A note on TOON and structured formats

There’s a newer format worth knowing about called TOON (Token-Oriented Object Notation). The idea is to strip JSON’s syntactic overhead, quotes, braces, repeated keys, by declaring keys once, like a header row. For flat, table-like data (user lists, product catalogs, RAG reference chunks, uniform agent outputs), benchmarks report roughly 30-60% prompt-token reduction with equal or slightly better accuracy on structured retrieval tasks.

It’s a real technique, but it comes with caveats. For deeply nested or irregular objects, the savings shrink or reverse. JSON is deeply embedded in LLM training data; TOON is a format the model hasn’t seen nearly as much, which introduces some fragility without fine-tuning. You’re also adding a serialization layer with converters, validation, and debugging in a nonstandard format that has real engineering cost.

The right framing for TOON: keep JSON in your APIs and storage. Use TOON only at the LLM boundary, and only when your prompts are heavy on structured data and the token savings are large enough to justify the complexity. It’s a specialized tool, not a universal one.

When not to optimize

For small documents, low-volume internal tools, or safety-critical tasks, it’s often better to overspend on tokens than to risk degraded quality or missed edge cases. Token optimization has a cost in engineering time, added complexity, and potential quality tradeoffs. The ROI only makes sense at the hotspots.

The teams that get this right aren’t the ones that optimize everything. They’re the ones that instrument first, identify the real cost drivers, and apply targeted effort where it makes the biggest difference.

Token spend is a design output. If you’re surprised by your LLM bill, the answer isn’t a cheaper model. It’s a closer look at what you’re asking the model to do, and why.

I run a competitive research workflow on a regular cadence. I use the same prompt, same tools, and the same intent. I want to map the landscape, surface new entrants and track what the competition is up to. The first few runs looked good. Then something started shifting. Not the market, our competitors in this space don’t move that fast. The outputs were different. Different companies surfaced. Different framing of the same players. Different conclusions from largely the same inputs.

The agent hadn’t broken. It was still producing plausible, well-structured research. It just wasn’t producing consistentresearch.

The gap between an agent that performs well and an agent that performs reliably is not something I hear talked about very often.

Why the demo always works

LLMs don’t retrieve. They generate. Every output is a sample from a probability distribution shaped by the prompt, the context window, and the model version. In a single-turn interaction this rarely matters. The variation is small and the output is usually close enough.

Agentic systems change the equation. Outputs become inputs to the next step. Tools get called, state accumulates, and decisions made in step two shape what is possible in step six. Small variations compound. The agent that searched slightly differently in step two is now summarizing different sources in step four. It draws different conclusions in step six. This is not a bug in any conventional sense. It’s the nature of the beast.

A demo is a single run. It proves the agent can do the task. It says nothing about whether the agent will do the task the same way tomorrow, after a prompt change, or after the underlying model gets quietly updated by the provider.

Reliability is not following capability

Recent research makes this concrete. A January 2026 simulation study from arXiv (arxiv.org/abs/2601.04170) defines agent drift as the progressive degradation of behavior, decision quality, and coherence over extended interactions. It identifies three distinct forms: Semantic drift is where outputs deviate from original intent. Coordination drift is where multi-agent coherence breaks down. Behavioral drift is where the agent develops unintended strategies over time. Using theoretical modeling across simulated enterprise workflows, the study projects that unchecked drift could lead to task success rates dropping by over 40% and human intervention requirements tripling. These are projected figures from simulation, not measured production outcomes. But the underlying framework for why the failure mode compounds rather than plateaus is well-constructed.

A large-scale empirical survey from UC Berkeley, Stanford, UIUC, and IBM Research (arXiv:2512.04123) gives the clearest picture of how practitioners are responding. Of 306 practitioners surveyed, 68% keep their deployed agents to at most 10 steps before human intervention. 70% adjust the prompts rather than fine-tuning the model. 74% rely primarily on human evaluation. The researchers frame this as a deliberate paradox: reliability is the top development challenge, yet agents are reaching production. The resolution is that teams are not waiting for the reliability problem to be solved. They ship by limiting what agents can do. Constrained autonomy, sandboxed environments, internal deployment first. It works. But it’s a workaround, not a solution.

Not all agents carry the same risk

Agents can be broken down into two main categories. The risk profile for each is different.

Bounded agents are invoked for a discrete task, produce an output, and hand off to a human or downstream process. Example include: A Cursor session writing a function. A one-shot document summary. The scope is defined. The output is reviewable. Failures are localized. These carry constant risk over time that is largely tied to human review.

Ambient agents run continuously and make ongoing judgment calls without a hard stop. For example, inbox triage or continuous competitive monitoring. Basically any workflow where the agent decides what matters and acts on it repeatedly, without a human checkpoint between decisions.

My competitive research workflow sits between these two. It is repeatable rather than truly continuous, but the expectation of consistency is the same. When I run it on Monday and again in three weeks, I expect the differences in output to reflect differences in the market, not differences in the agent. That’s not what happened.

McKinsey’s 2025 global survey found that 62% of organizations are at least experimenting with AI agents. Only 23% are scaling one in production. The gap between experimentation and scale is not a capability gap. It is a trust, observability, and governance gap.

If you’re running long-horizon agents, here’s what helps

The research and the emerging vendor landscape have converged on a set of mitigation approaches. None of them eliminate the problem but they can reduce risk.

Context management. One of the least visible failure modes in long-running agents is context drift. As conversation history grows, reasoning quality degrades before you ever hit a context limit. The industry has settled on episodic consolidation: periodically compressing older context into structured summaries while preserving recent and relevant state. The Agent Drift paper identifies this as one of three mitigation strategies with the strongest theoretical grounding. Anthropic now ships a native compaction API that automates the loop.

Uncertainty-aware memory. A January 2026 paper from Salesforce AI Research calls the core failure mechanism in long-horizon agents the Spiral of Hallucination. A small grounding error in an early step gets committed to the agent’s context. It then becomes a false premise for every subsequent step. Standard self-reflection does not reliably catch this. The model has already accepted the error as ground truth. The proposed fix flags low-confidence steps before they propagate and triggers correction only when needed. Early results showed meaningful reliability improvements on multi-step benchmarks. This is early research. But it is getting at the cause rather than the symptom.

Checkpointing and interrupt design. Orchestration frameworks like LangGraph have built explicit checkpointing into their execution model. Agents are defined as directed graphs with typed state and hard interrupt points. A human can review, approve, or reset to a known-good checkpoint at any of those points. This converts a brittle autonomous system into a collaborative one. Carnegie Mellon benchmarks published in late 2025 found that leading agents complete only 30-35% of multi-step tasks successfully. This shows that uninterrupted autonomous execution is not the right default for complex workflows.

Golden dataset evaluation. This approach maps most directly to my competitive research problem and our product work. Create a set of representative inputs with human-verified expected outputs. Then run your agent against that dataset on a schedule or before any prompt change goes to production. AWS introduced this at re:Invent 2025 with the general availability of Bedrock AgentCore Evaluations: 13 built-in evaluators, CI/CD pipeline integration for pre-deployment gates, and continuous online evaluation against live production traffic. A demo showed the service detecting tool selection accuracy dropping from 0.91 to 0.3 in production. Without continuous measurement, that degradation is invisible.

Pushpay documented a real production implementation of this pattern. Their golden dataset covers over 300 representative queries with validated responses. It is continuously curated from actual user interactions and fed into an engineering dashboard. The key word is continuously. A golden dataset that does not evolve with your actual workload tests against past state not current state.

Beyond AWS, the commercial tooling has matured fast. Braintrust ties production traces and offline experiments to the same scorer library. A production regression automatically seeds the next test cycle. LangSmith integrates human annotation queues with trace replay, letting engineers convert production failures into evaluation cases. Arize offers always-on drift detection at the session and span level. For teams with HIPAA or data residency constraints, Langfuse is the strongest self-hosted open-source option. It was acquired by Clickhouse in January 2026, but the open-source codebase remains active.

None of this is free. Building and maintaining a golden dataset requires human judgment to define what “correct” looks like for open-ended tasks. That is genuinely hard when correctness is partly subjective. Dataset rot is a real risk. The infrastructure to run evaluations continuously has real cost. The tooling can solve the infrastructure problem but the curation problem is still yours.

For my competitive research workflow, the approach is well-suited. The expected output structure is defined even if the specific content varies. I know what a well-formed competitive analysis looks like. I can score for completeness, source coverage, and structural consistency without specifying exact content in advance. That is an easier evaluation target than most ambient agent tasks.

The durability problem

The industry has gotten very good at demonstrating what agents can do. It has not gotten as good, so far, at ensuring they keep doing it the same way.

Gartner projects that 40% of agentic AI projects will fail by 2027. Poor risk controls are cited as a primary cause. That figure will land as a surprise to anyone who has only ever evaluated their agents at a single point in time.

Narrow, monitored, bounded agents are viable today if you build them with that constraint in mind. Always-on autonomous agents are still waiting on better reliability science, better evaluation tooling, and more organizational honesty about the governance they require.

The question worth asking before deploying any agent is not “can it do the task.” It is whether you can tell when it starts doing the task differently than it did before. And whether you would know before your users do.

Are you testing the wrong thing?

The interview process is supposed to predict job performance. When the conditions of the interview bear no resemblance to the conditions of the job, you’re not predicting anything. You’re running a different experiment and hoping the results transfer.

They don’t.

What we actually test when we ban AI

The stated rationale for AI-free coding interviews is reasonable on its surface. We want to see how candidates think. We want to know if they actually understand the problem. We don’t want someone to paste a prompt and copy the solution.

The concern is legitimate. The conclusion is wrong.

Here’s what an AI-free whiteboard session actually measures: the ability to hold syntax, algorithm structure, and edge cases in working memory simultaneously, under pressure, without tooling, in an artificial environment. That is a real cognitive skill. It just isn’t the one that determines whether someone will perform well on your team in 2026.

The cognitive load of software development has shifted. The job now requires a different kind of thinking: decomposing a problem into chunks small enough to prompt effectively, reading AI output critically before accepting it, knowing when the model is hallucinating and why. A candidate who can recite a BFS implementation from memory but blindly accepts a subtly wrong AI-generated solution is a worse hire than one who forgets the exact syntax but immediately spots the flaw in what the model produced.

We’ve been measuring recall. We should be measuring judgment.

The industry has figured this out

In October 2025, Meta began rolling out an AI-enabled coding interview that replaces one of the two traditional coding rounds at the onsite stage. The internal framing is revealing. The format was designed to be more representative of the actual developer environment and also to make LLM-based cheating less effective.

The problems Meta uses are designed with AI assistance in mind. They’re harder than a traditional coding question. The bar for what a candidate is expected to produce is higher precisely because they have help. Candidates work in a multi-file codebase they didn’t write and have to understand quickly. Prompt-and-paste fails immediately in that environment because understanding the existing architecture is the prerequisite for everything else.

CodeSignal launched AI-assisted coding assessments in May 2025 with a feature that matters more than the AI access itself: a full transcript of every candidate-AI interaction alongside a session replay. You’re not just seeing what the candidate produced. You’re watching how they think.

HackerRank moved in the same direction. Candidates work with AI tools in a controlled environment and interviewers get a detailed view of the problem-solving process, not just the output.

What an AI-free interview cannot show you

When I think about what I actually need to know about a candidate, four things matter that a no-AI interview cannot surface.

Prompt quality as a diagnostic. The way someone frames a problem for an AI is a direct readout of how they think about the problem. Vague prompts reveal vague thinking. A candidate who writes “fix my function” tells you something different from one who writes “this recursive function is hitting a stack overflow on inputs above n=1000, here’s the current implementation, what’s the likely cause.” The second candidate has already diagnosed the problem. They’re using the AI to confirm and implement. That’s engineering judgment.

Verification instinct. One E7 candidate at Meta watched Claude Sonnet repeatedly hallucinate on a maze problem. The question isn’t whether the AI was wrong. The question is whether the candidate caught it. Did they know what correct looked like before the model answered? Did they push back? A candidate who accepts wrong output without question is a risk that an AI-free interview will never expose, because you never gave them an AI to accept.

Task decomposition. Candidates who performed well in Meta’s AI-enabled format guided the AI incrementally rather than asking for wholesale solutions. One successful candidate described her approach: start with the core logic as a single function, review it, then build out from there. That instinct, to keep the scope small enough to verify at each step, is exactly how good engineers approach complex problems. It’s invisible in a no-AI environment because there’s nothing to decompose for.

Communication under ambiguity. Meta’s internal evaluation criteria for this round includes a phrase that functions as an answer key: “Should use AI, but need to show you understand the code. Explain the output. Test before using. Don’t prompt your way out of it.” That is a rubric for thinking, not for tool use. The candidate is being evaluated on whether they can narrate a reasoning process in real time, hold a conversation with an interviewer while working with an AI assistant, and remain the accountable decision-maker throughout. That skill matters on the job. AI-free interviews don’t test it.

The gaming problem is real but solvable

The obvious objection is that AI-assisted interviews are easier to game. A fast prompter with shallow understanding can look strong if the evaluation is just the output.

That’s true. But it’s a design problem, not a fatal flaw.

Interaction transcripts solve most of it. When you can see the full sequence of what a candidate asked for and how they responded to what they got, shallow prompters reveal themselves quickly. They ask for too much at once. They accept the first answer without testing it. They can’t explain the code when asked.

Multi-file codebases with staged checkpoints solve the rest. A problem that requires understanding existing architecture before making any change can’t be solved by pasting a description into a chat window. The AI doesn’t have the context. The candidate has to build it, which means they have to understand it.

The gaming risk in no-AI interviews is equally real and far less visible. A candidate who has memorized LeetCode patterns looks identical to one who genuinely understands algorithms. At least AI transcripts expose the reasoning process. A whiteboard session shows you the answer. A session replay shows you how the candidate thinks.

What to change before your next hire

None of this requires overhauling your entire process. Three things make the difference.

Redesign the problem before you change the rules. A standard LeetCode question with AI access is still a bad question. The problem needs to be complex enough that AI assistance is a navigation tool rather than a solution dispenser. Ambiguous requirements, existing codebases, staged checkpoints. Problems designed so that understanding is the prerequisite for prompting.

Make the AI interaction visible and gradable. If you’re using a platform, use one that captures transcripts. If you’re running your own interviews, ask candidates to narrate their prompts out loud and explain what the AI gave them before they act on it. That narration is the interview.

Keep a short no-AI segment with a clear purpose. Baseline fundamentals still matter. A candidate who can’t read a stack trace or reason about complexity without assistance is a real risk. A focused no-AI segment to test that floor is legitimate. Don’t treat that floor as the whole evaluation.

The prediction model you’re running

Every hiring process is a prediction model. The inputs are interview signals. The output is a forecast of job performance. When the inputs don’t reflect job conditions, the model is broken.

Running AI-free coding interviews for engineers who will spend their careers working alongside AI tools is like running driving tests with no steering wheel because you want to assess balance. The rationale sounds defensible. The instrument is wrong.

The industry has started correcting. The question is whether your process has.

If you’ve been using AI coding tools over the past two years, you’ve been getting a great deal. Frontier model access embedded in your IDE, powering your agents, running in your CI pipelines — for prices that don’t actually cover what it costs to serve you.

That’s not a bug. It’s a strategy. And strategies change.

This post is about what happens when the economics of AI-assisted coding get repriced, why that repricing is likely in the next 12 to 24 months, and what your engineering organization should be doing right now before the bill arrives.

The Free Ride Won’t Last

According to internal projections reported by the Wall Street Journal, OpenAI does not expect to reach profitability until 2030. Anthropic projects reaching positive free cash flow by 2027 or 2028. Both companies are growing revenue at extraordinary rates. Anthropic recently reported annualized revenue exceeding $30 billion. But revenue and profit are not the same thing, and right now these companies are very far from the same thing.

The structural problem is inference compute. OpenAI spent roughly 50% of its revenue on inference costs alone in recent years, with training costs pushing total expenditure well above what comes in. Every token you generate costs real money in GPU time. The pricing you see at the API does not reflect what it actually costs providers to serve those tokens.

Open-weight models of comparable capability are anywhere from 17x to 18x cheaper than Anthropic’s API pricing and are from providers who are covering their costs and making margins. That isn’t an indictment of Anthropic’s business model. It reflects real differences in model capability, trust, tooling maturity, and enterprise positioning.

The narrative you’ll hear most often is that inference costs will keep falling and everything will work out. That narrative has been repeated for three years. Inference costs for frontier models have not followed the curve that optimists projected, partly because each new model generation is larger and more capable than the last, which resets the compute baseline. Lower prices on last-generation models don’t help you if you need current-generation capability.

The current pricing environment is a competitive land-grab. It’s not sustainable.

The IPO Pressure Cooker

Both Anthropic and OpenAI are moving toward public markets. Anthropic has engaged IPO counsel and is reportedly discussing an offering as early as Q4 2026, targeting a raise exceeding $60 billion. OpenAI is targeting a similar timeline at a valuation approaching $1 trillion.

Private investors fund growth stories and tolerate long paths to profitability. Institutional fund managers running discounted cash flow models do not. The S-1 filing will contain actual unit economics for the first time. Analysts will model gross margins. Price-to-earnings ratios will matter in a way they don’t when you’re raising from VCs.

This creates a specific incentive. Both companies have strong motivation to show margin improvement before listing, not after. The levers available are cost reduction (hard, because compute costs are driven by usage and model scale) and price increases (a decision that can be made in an afternoon).

There’s also the Inference Trap: You build the best model, usage surges, inference compute explodes, and you face a forced choice between throttling users, raising prices, or cannibalizing the training compute you need to stay competitive. Anthropic experienced five major platform outages in a single month in early 2026. Claude Code users reported burning through usage allocations far faster than the pricing implied. That’s the Inference Trap operating in real time.

The combination of IPO pressure and Inference Trap dynamics makes a repricing event not just plausible but structurally likely. The question isn’t whether it happens. It’s whether you’re ready when it does.

What You’ve Actually Built On

Most engineering teams believe they have less AI vendor lock-in than they actually do. The assumption is: swap the API key, update the model name, done. There’s a lot more to it than that. It gets worse the deeper into agentic workflows you go.

The lock-in profile varies significantly by use case:

IDE-embedded tools like Copilot, Cursor, and Claude Code represent the shallowest lock-in. You could switch IDEs or model backends with little effort. But don’t underestimate soft stickiness. Developer muscle memory, .cursorrulescustomizations, team-shared system prompts, and workflow integrations all add switching friction. A price increase here hits developer productivity budgets, which are visible and politically sensitive.

Agentic coding workflows are where real lock-in begins. Agentic systems don’t just call a model — they build scaffolding around it. System prompts are tuned to a specific model’s personality and failure modes. Tool-calling schemas are optimized for how that model interprets them. Retry logic and output parsing are calibrated to observed behavior. When you switch models, that scaffolding doesn’t transfer cleanly. You’re not changing a config parameter. You’re running a re-evaluation campaign against your own codebase. Industry data suggests migration costs when provider lock-in forces a move average over $315,000 per project, and that figure reflects situations where teams already had some abstraction in place.

CI/CD and automated pipelines carry the highest risk. These are production systems with determinism requirements. Prompts optimized for one model may produce subtly different outputs on another. Those outputs can look similar enough to pass manual inspection but break downstream parsers and validation steps. Model version pinning provides a false sense of stability because providers deprecate models with 90 days’ notice, and there is no guarantee of behavioral equivalence between versions. The fundamental problem is that you cannot treat an LLM call in a production pipeline the same way you treat a deterministic function call. When you switch models, you have to prove the pipeline still works. You cannot assume it.

Open Source is a real option, but it has gaps

The obvious response to pricing risk is to use open source models, self-host, and pay for compute instead of markup. That path is more viable than it was 18 months ago but has real gaps that tend to be underestimated.

The capability gap has largely closed on many dimensions. Open models now match or surpass closed models on knowledge benchmarks, mathematical reasoning, and graduate-level science. The gap that remains is concentrated where it matters most for coding: production-level agentic tasks, multi-step software engineering, and complex tool use. On SWE-bench Verified, the most practically meaningful coding benchmark, the best open models are within a few points of frontier closed models. That gap is still an issue at the tail of task complexity.

The price differential is big. DeepSeek V3.2 is available at roughly $0.28 per million input tokens. Claude Opus 4.7 is $5.00 per million input tokens. That’s a 17x difference. For high-volume workloads the economics are compelling even accounting for operational overhead.

But here’s what the open source advocates undersell: switching models is not the same as switching model providers. The scaffold matters as much as the model. Real-world benchmarks show a 22-point swing on the same task with the same model when you change the agent scaffold and tooling. Switching models requires re-validating your entire system, not just verifying the model output looks reasonable.

The operational burden of self-hosting is a real cost transfer. Inference infrastructure, model serving with tools like vLLM or Text Generation Inference, GPU provisioning, update cadence, and security patching all fall on your team. For most organizations without dedicated ML infrastructure experience, this isn’t a savings. It’s a new operational surface area.

There’s also a geopolitical dimension worth naming directly. The strongest open models right now (DeepSeek, Qwen, Kimi) are Chinese-developed. For organizations with data sovereignty requirements, government contracts, or security-sensitive codebases, the lineage of a model matters. This isn’t a reason to dismiss these models outright, but it’s a factor that belongs in your architecture decision.

The Protocol Layer Is Your Best Friend

The most practical near-term lever against lock-in isn’t switching to open source. It’s building an architecture that makes switching possible.

Model Context Protocol (MCP) is the most significant structural development here. Originally developed by Anthropic and then donated to the Agentic AI Foundation (AAIF). This foundation was co-founded by Anthropic, Block, and OpenAI. MCP has achieved something rare: genuine cross-industry adoption. OpenAI abandoned their proprietary Assistants API and adopted MCP. Google DeepMind, Microsoft, and AWS are all on board. When direct competitors converge on a shared infrastructure standard it signals inevitability.

MCP decouples the agent-tool connection layer from the model layer. Your integrations with databases, APIs, filesystems, and external services are built once against the MCP standard and survive a model swap. That’s the right layer to standardize at.

Pair that with an LLM Gateway such as LiteLLM or Portkey, middleware that abstracts provider-specific API differences behind a single interface, and you get a system where the model backend is genuinely swappable without rebuilding your application logic. The marginal complexity cost of adding this abstraction early is low. The switching optionality it creates is high.

Be honest about what MCP doesn’t solve though. The protocol handles tool integration, not model behavior. When you swap models, your prompts still need re-validation. MCP can also consume 40-50% of available context window before any actual work begins, which creates real production tradeoffs. Standards help. They don’t eliminate the work.

What You Should Do Today

The cost of acting on this now is low. The cost of acting after a pricing shock is high.

For IDE tools: Evaluate whether your current tooling is model-agnostic or model-bundled. Prefer tools that let you swap backends. Baseline your developer productivity metrics now. You need a measurement baseline before any changes hit, not after.

For agentic workflows: Add an LLM Gateway from the start of any new project. Keep your agent orchestration layer architecturally separate from your model API calls. This is the single highest-leverage structural decision you can make. Build evaluation suites against your own codebase, not generic benchmarks. Generic benchmarks tell you how a model performs in the abstract. Your eval suite tells you whether you can safely swap models in your specific system.

For CI/CD pipelines: Treat every LLM call as a third-party dependency with explicit versioning, SLA monitoring, and a tested fallback path. Design for graceful degradation. What does the pipeline do when the model endpoint is slow, unavailable, or has been updated? This should be a documented decision, not an untested assumption.

Across all use cases: Audit your current AI spend and its concentration across providers. Most teams have no idea what this number is. Monitor the IPO timelines. The S-1 filings will be the first time the public sees actual unit economics from these companies, and they will move the conversation. Build internal familiarity with at least one open-weight model family. Even if you never deploy it in production, that knowledge reduces the information asymmetry in any future pricing negotiation.

A Strategy Note, Not a Panic Note

The goal here is not to abandon frontier models. They are genuinely better at certain tasks, the tooling ecosystem around them is more mature, and for many use cases the productivity gains justify whatever they end up costing.

The goal is not to be surprised. More specifically, the goal is to avoid being in the position of needing to move urgently with no alternatives evaluated and no time to build them.

Engineering organizations that have done the architecture work to reduce switching costs will have options when prices move. They’ll be able to make a deliberate choice between absorbing the increase, substituting a capable alternative, or negotiating from a position of real leverage. Organizations that haven’t done this work will face a different situation: urgent need, unknown switching cost, and a vendor who knows it.

The bill is coming. The amount is unknown. The only variable you control is how ready you are to pay someone else instead.

Jack Dorsey and Roelof Botha published “From Hierarchy to Intelligence” on March 31, 2026. It is a serious piece of thinking. The historical framing is sharp, the diagnosis of why hierarchies exist is largely correct, and the argument that AI changes the information-routing constraint is real.

But the essay is about one-third of what managers actually do. It treats that one-third as the whole job, removes the people doing it, and calls the problem solved. The other two-thirds are still there. They just don’t have anyone doing them anymore.

To understand that better, we need to be precise about what managing actually is.

Three Clusters, Not One

In 1973, Henry Mintzberg published The Nature of Managerial Work, based on direct observation of what managers do with their time. His finding was that managerial work falls into three clusters: informational, interpersonal, and decisional.

The informational cluster is what Dorsey and Botha are talking about. Managers monitor what’s happening, disseminate that information across the organization, and represent the team to the outside world. This is the information routing function. It’s the layer that hierarchy was built to support, and it’s the layer that AI can now perform continuously, at scale.

That’s true. But it’s only one of three.

What AI Cannot Route

The interpersonal cluster covers things that depend on trust, relationship, and human accountability. The leader motivates and develops people, figures out who has potential and what they need to grow, and has the difficult conversations.

The liaison role involves building relationships across organizational boundaries, the kind of connective tissue that makes cross-functional work actually work. The figurehead role is about legitimacy and accountability. When something goes wrong, someone needs to be responsible in a way a system cannot be.

A telling data point from the Block restructuring itself: current and former employees told The Guardian that roughly 95% of AI-generated code changes still require human modification. This is in a remote-first, highly digital, machine-readable organization, exactly the environment that Dorsey describes as most amenable to this model. The humans are still in the loop not because the information system failed, but because the work itself requires judgment that isn’t in the data.

The decisional cluster is where this becomes even clearer. Mintzberg’s entrepreneur role involves sensing and acting on opportunities that aren’t visible yet. A world model, by definition, can only reflect what has already happened. It cannot tell you what to build that doesn’t exist yet. The disturbance handler role is about responding to crises and genuinely novel situations, exactly the circumstances where pattern-matching on historical data is most likely to fail. Resource allocation and negotiation involve competing interests, trust between parties, and judgment under uncertainty. In regulated industries, financial services, healthcare, any domain with fiduciary obligations, you can’t delegate these decisions to a system regardless of how good it is.

The Risks Worth Naming

Beyond the functional gaps, there are a few things worth noting:

Post-hoc rationalization?. Block cut 40% of its workforce in February 2026, before the essay was published. The stock jumped roughly 22%. Botha, who co-authored the essay, sits on Block’s board. Morgan Stanley upgraded Block to overweight after the cuts. Goldman Sachs raised its price target. It is fair to ask whether the intellectual framework followed the business decision or preceded it. That doesn’t make the argument wrong, but it does mean the incentive to believe the argument is quite strong for the people making it.

The flat structure graveyard. Zappos tried holacracy. Valve famously ran without managers. The Spotify model has been widely adopted and widely struggled with. These experiments didn’t fail because the idea was wrong in theory. They failed because removing formal structure doesn’t remove the need for coordination. It just moves coordination into informal channels, where it becomes invisible, political, and dependent on whoever has the most social capital. The information routing problem gets solved. The interpersonal and political problems get worse.

Data completeness. A world model built from Slack threads, Jira tickets, pull requests, and performance metrics reflects what was written down. A significant fraction of organizational knowledge is never written down. It lives in the judgment calls that didn’t make it into a doc, the context a senior engineer carries about why a system was built the way it was, the reason a decision was made three years ago that nobody remembers to explain to new people. The model sees the artifact. It doesn’t see the reasoning behind it.

Data quality and drift. This one is distinct from completeness and is arguably more dangerous. Information that was accurate when it entered the system becomes stale. The system continues to present it with the same authority as fresh data. Decisions get made on information that was true six months ago and isn’t anymore. You don’t see the error at the time. It shows up later, in ways that are very hard to trace back to their source.

This is a documented, recurring failure in knowledge management systems generally. It’s not theoretical. Platforms like Guru have built their core product differentiation specifically around the knowledge freshness problem because the industry learned, repeatedly, that drift is the default. Small errors accumulate in decisions that each look reasonable in isolation, until something downstream breaks in a way nobody can explain.

Regulatory reality. For companies operating in financial services or healthcare, the question of whether AI can replace decision-making isn’t just organizational. It’s legal. Explainability requirements, fair lending law, fiduciary duty, these don’t care how good your model is. Decision accountability cannot be delegated to a system in regulated domains. This alone may prevent this type of World Model from being used is certain industries.

Where This Is Actually Going

None of this means the Dorsey/Botha thesis is wrong. I just think it’s incomplete.

The informational cluster is being automated. That is a real and permanent shift. Managers who spent the majority of their time aggregating context, relaying status, and maintaining alignment across teams are already less necessary than they were. That part of the argument makes a lot of sense.

What’s more interesting is what happens to the other two clusters as this plays out. If information routing gets absorbed by AI systems, the interpersonal and decisional work doesn’t disappear. It becomes more visible and it’s value should be more obvious. Managers who survives this transition are the ones who were always doing the work that was hardest to put on a job description.

The open question is whether that work changes in character as organizations become more AI-instrumented, or if it simply becomes more prominent because everything else has been stripped away. Does managing people become fundamentally different when the coordination layer is automated? Or does it turn out that the relational, developmental, and judgment-intensive work was always the real job, and the information routing was just the overhead we confused for the substance?

I don’t think anyone knows yet. Block’s Q1 2026 results will be a first data point. If they hit $12.2 billion in gross profit with 40% fewer people, the thesis gets harder to argue with. If they don’t, their Roman Army comparisons will age badly.

Either way, the question is worth asking more carefully than “can AI replace the org chart.” The org chart was never the point. It was just the structure we built to solve three problems at once. AI can solve one of them. The other two? Let’s see.

I’m reading C. Thi Nguyen’s new book, The Score, and one idea in it keeps coming back to me. He calls it value capture,the process by which a rich, meaningful goal gets quietly replaced by the metric you were using to track it. You start measuring something because it points toward what matters. Then, gradually, the measurement becomes what matters. The original goal doesn’t disappear. It just stops being the thing that drives decisions.

That’s not a philosophy problem. That’s every day in most engineering organizations.

What we actually measure

Let’s be clear about what passes for engineering measurement in most companies.

Output metrics are the default: story points, tickets closed, pull requests merged, lines of code written. They’re easy to collect, easy to visualize, and they feel like signal. The problem is they measure production, not value. A team can close 200 tickets in a sprint and ship nothing a customer cares about. Story points aren’t a unit of value, they’re a unit of negotiated effort, and that negotiation starts the moment someone decides to track them.

Performance metrics try to go one level deeper: code review turnaround time, sprint commitment hit rate, on-call response time, test coverage percentages. These are more interesting because they reflect process health. But they still measure fidelity to a process, not effectiveness of the work. A team can hit 95% sprint commitment every week by sandbagging estimates. Reviews can be fast because nobody’s actually reviewing.

Efficiency metrics are where most engineering organizations have landed recently, particularly DORA, the four-key-metrics framework: deployment frequency, lead time for changes, change failure rate, and time to restore service. DORA is genuinely useful as a diagnostic. The problem is what happens when it moves from a team-level health check to a leadership dashboard. Deployment frequency gets gamed by trivializing deployments. Lead time gets gamed by where you start the clock. You end up with a team deploying 15 times a day that is still six months from shipping anything meaningful.

Every one of these metric categories lives entirely inside the engineering system. None of them has a direct connection to whether the engineering organization is actually doing its job.

The game you didn’t know you were playing

Nguyen’s book makes a distinction that hits differently in an engineering context. He separates striving play from achievement play. In striving play, the goal is to engage fully, the process, the judgment, the craft. In achievement play, the only thing that matters is the score. Great games are designed so that chasing the score also produces striving. You can’t get good at chess by gaming the scoring system; you actually have to get good at chess.

Institutional metrics are the opposite. They strip out the magic circle, Nguyen’s term for the temporary, voluntary frame that makes game constraints feel meaningful rather than oppressive. In a board game, you accept arbitrary rules because you chose to sit down and play. In a work context, those rules aren’t arbitrary and they aren’t optional. The score follows you. It shows up in your performance review. It gets presented to the board.

What’s left, once you remove the magic circle, is a system that rewards achievement play. And engineers, who are, professionally, some of the best problem-solvers in any room, find the optimal path to the score. This isn’t cynicism. It’s a completely rational response to the incentive structure you built.

The consequences are predictable:

Velocity becomes sandbagging. The moment team velocity appears on a leadership dashboard, estimation inflates. Points expand to protect the team. After a few quarters, the number is politically stable and informationally useless.

Deployment frequency rewards triviality. If deploying frequently is the metric, the rational move is to break work into smaller pieces, not because small batches are better (they often are, but for different reasons), but because each deployment ticks the counter.

Commitment rates reward conservatism. Measure whether a team delivers what they promised and they’ll promise less. You’ll see consistently green dashboards and a team that’s quietly becoming slower.

Code review speed becomes rubber-stamping. If time-in-review is visible, reviewers learn to approve fast. Technical debt accumulates invisibly while the metric looks healthy.

This is Goodhart’s Law in action, once a measure becomes a target, it ceases to be a good measure. But Nguyen’s framing adds something important: it’s not just that the metric gets corrupted. It’s that people’s values get reshaped around it. The engineers optimizing for velocity aren’t lying. They’ve internalized the metric. The metric has become, for them, what good work looks like. That’s value capture. That’s the thing that’s actually hard to fix.

What these metrics are actually telling you

This is worth being precise about, because the answer isn’t “nothing.” These metrics have legitimate uses. The failure is the mismatch between what they measure and what leaders use them to decide.

Output metrics, velocity and tickets closed can tell you if work is flowing through the system at all. A team whose velocity drops 40% in two sprints has a problem worth investigating. What they can’t tell you is whether the work matters.

Performance metrics are operational diagnostics. Long code review cycles, high defect escape rates, chronic on-call fatigue, these are real signals about process dysfunction. Treat them as process indicators, not as performance scorecards.

DORA is pipeline health. Lead time and deployment frequency tell you something real about delivery capability. Change failure rate and MTTR tell you something about resilience. But all four are downstream of the question that actually determines whether an engineering organization is performing: are you building things that move the business, fast enough to matter?

None of the standard metrics reach that question. The reason is structural. They’re easy to collect because they live inside the engineering toolchain. Anything that requires a connection to product outcomes, customer behavior, or business results is harder, and that difficulty is precisely why it tends not to get measured.

The missing layer

The data we reach for first, the data that’s easy to collect and easy to present, systematically hides what’s actually going on. The measurement layer that’s missing isn’t more engineering metrics. It’s a feedback loop that closes outside engineering.

A few things that would actually tell you something:

Outcome linkage. Can you connect a shipped feature to a measurable change in user behavior or business results? Not “we shipped it” but “after we shipped it, the thing it was designed to move, moved.” This requires instrumentation, a documented hypothesis before work starts, and a willingness to wait. None of those come naturally to sprint planning cadences. But without them, you’re measuring production, not impact.

Flow efficiency. The ratio of value-added time to elapsed time in your delivery process is more interesting than raw velocity. A feature that takes 12 weeks from idea to production, with only 2 of those weeks involving actual engineering work, has a flow efficiency of 17%. That’s a systems problem — and throughput metrics will never surface it. The bottleneck isn’t in the work; it’s in the waiting.

Technical health as a leading indicator. Complexity trends, dependency staleness, incident frequency — these are imperfect but directionally useful signals about whether your codebase is getting easier or harder to extend. Engineering organizations that ignore these tend to see velocity collapse right when the business needs them to accelerate. It’s not a coincidence.

Team capability over time. Are engineers growing? Are they retaining? Are they increasingly autonomous, or increasingly dependent on a few specialists? These lag badly as indicators, but they’re leading indicators of whether the organization will still be functional in two years. No sprint metric captures them.

Breaking the cycle

Swapping bad metrics for better ones doesn’t solve the problem. If you replace velocity with DORA and keep the same incentive structure, you’ll get gamed DORA metrics in six months. Nguyen is clear on this in The Score, once value capture takes hold, the answer isn’t a better score. It’s rebuilding the conditions under which people can reclaim their own values.

In an engineering context, that means a few things:

Separate diagnostic metrics from evaluation metrics. Deployment frequency is useful when a team uses it to understand their own pipeline. It becomes corrosive the moment it appears on a leadership report as a proxy for team performance. The same number. Completely different effect depending on who it’s for and what it drives.

Measure outcomes and accept the latency. This requires leaders to resist the urge to instrument everything that can be instrumented. Define what you’re trying to move before work starts. Measure it after you ship. Accept that the feedback loop is slower than a quarterly review cycle, and if your performance management cycle is shorter than your product feedback cycle, that’s the real problem to fix.

Make the system visible, not just the throughput. Flow efficiency, incident trends, and technical debt trajectories give teams and leadership a shared picture of systemic constraints. When the conversation shifts from “why didn’t you close more tickets” to “what’s blocking flow and what would it cost to fix it,” you’re at least asking questions that can produce useful answers.

Let the teams define the metrics for their own work. This one is underrated. The people closest to the work know what signals matter and which ones can be gamed. Metrics designed by a team to understand their own performance are completely different from metrics imposed from above to evaluate them. The former creates accountability. The latter creates the conditions for value capture.

Hold leadership accountable for outcome clarity. A lot of engineering metric gaming exists because the business hasn’t clearly defined what success looks like. If product leadership can’t say what a feature is supposed to change, and how they’ll know it worked, the engineering team will fill that vacuum with whatever proxy feels safest. Measurement quality is a leadership problem as much as a measurement problem.

The question the dashboard can’t answer

Look at whatever engineering metrics you’re currently tracking. For each one, ask: what decision would I make differently if this number were 20% better or worse? If the answer is “I’d evaluate someone’s performance differently,” the next question is whether you’re actually measuring what you want to optimize for, or just what’s available.

Nguyen ends The Score with the question the title comes from: is this the game you really want to be playing? Most engineering organizations never ask it. The dashboard is there. The numbers are green or red. The sprint review happens. Nobody stops to ask whether the game itself is worth playing.

Measuring the wrong things precisely is worse than not measuring. It gives you the confidence of data without the benefit of insight. The metric becomes the goal. The goal becomes the metric. The original purpose, building something that matters, gets quietly replaced. And everyone in the room is looking at the dashboard, wondering why the product still isn’t getting better.

When writing code using AI assistance, it’s easy to generate a lot of code in a very short amount of time. As the development process goes on, even more code will be created. This gets complicated when it comes to maintaining and expanding the code base. Over time the person working on the code has to get back into that code to address bugs or add features. Getting to the point where you understand the code well enough, again, takes time. I have seen that while it’s easy to create a lot of code with AI, it’s not as easy to come back into that code after it’s been created.

One thing we need to keep in mind when using AI to generate code is that now there becomes a difference between writing the code and understanding the code. In pre-AI software development, where you needed to manually write the code, there was a certain understanding of what you were writing while you were writing it. With AI, that process goes away. The code is created, and you then need to understand that code and become familiar with it so that you can then make updates.

When you write the code yourself, your brain is creating an internal model of that system that you’re creating. When you step away from that code and come back, some of that that internal model, will persist. Interacting with something, rather than just reading it, tends to be retained in the brain longer. Since you’ve written the code yourself, it’s easier to come back into that code and continue developing. If you have not gone through that process, if you are reviewing code that was written by AI or by another person, then you’re establishing familiarity with the code each time without having that pre-existing internal model. This makes it more difficult and takes longer to come up to speed with that code so you can continue development.

Protecting Comprehension

The answer isn’t to use AI less. It’s to recognize that AI handles code generation but cannot handle comprehension. Comprehension doesn’t transfer. Each of the techniques below are just a mechanism for forcing you to build a genuine internal model of the code, not just review it from the outside. If you spend 2 hours building something with AI that would have taken 2 days manually and then spend 45 minutes re-establishing context every time you return to it, the break-even on that speed gain comes faster than you think.

1. The spec-first inversion

The conventional flow is: prompt → code → review. Reverse the order. Before asking the AI to generate anything, write a 3–5 sentence description of what the code should do and why. Not a formal spec — just externalizing your intent. That document becomes your re-entry anchor, and it forces the model to work from your understanding rather than producing something you have to reverse-engineer.

2. Treat code review as active encoding, not QA

When you review AI-generated code, the goal shouldn’t just be catching bugs — it should be deliberately building the internal model you didn’t get from writing it. Don’t rubber-stamp diffs. Walk the logic path. Rename things that don’t reflect your mental model. The review is the learning; skip it and you’re adding to the context debt, not paying it.

3. The commit message as cognitive snapshot

Write commit messages as if you are explaining the change to yourself six months from now — not what changed, but why and what you understood about the system at the time. This is cheap, async, and gives future-you a scaffold that’s tied to the exact moment of maximum context. Git log becomes a context recovery tool, not just a change log.

4. AI-assisted re-entry

Use AI to rebuild context, not just create code. Paste the module into a fresh session and ask: “Explain what this does, what it assumes, and what would break if X changed.” That AI-generated explanation — corrected where wrong — becomes your working model. You’re using the same tool that created the debt to help pay it down.

What we’re seeing with AI assisted software development is that it is not a panacea. AI is changing the mental effort required to maintain understanding over time. It’s not just about how quickly we can create the code. It’s about being able to sustain that comprehension. The key takeaway here is to use AI to accelerate coding, but don’t outsource the understanding of that code. It needs to be understandable and comprehensible over time by humans after the code is written.

I’ve been doing some thinking about how software development has changed since the introduction of AI for code generation. Having been through a lot of these changes recently with building a new product, I think that it’s become a lot more interesting, but it’s also highlighted some important parts that may not have been given the importance they deserve.

The way I look at software development now is like a bookshelf. You have bookends on either end with books in between. In this metaphor, the LLM or the AI system is the books. This is what generates actual code, whether you’re using Claude Code or OpenAI, Google or anything else. The bookends are the parts that hold up that code.

I see the bookends as follows:

The first bookend is the definition of what you actually want to build. This was always important in traditional software engineering because engineering resources were scarce. The amount of time to develop something was fairly long so there was a lot of analysis and discussion and definition ahead of time, sometimes too much. You would spend months talking about something only to realize that you’ve missed the window and you didn’t need it anymore. That kind of fell a little bit by the wayside as vibe coding came about. People could just type in a one- or two-sentence prompt and get back something that was kind of what they wanted, but not really. The second bookend is the verification and the validation. You’ve given it clear instructions but AI is a probabilistic system so you are going to get something that’s probably very close to, but not exactly what you need. That verification process requires a human to look at the output with human-level context. They needc to look at what’s been built and validate that it did the right thing, that it built what they thought it was going to build. The human must then make changes, maybe small changes, maybe big changes, to get that code or that application really production ready. Without those bookends, if you look at just the way some people have approached coding, you end up with garbage in, garbage out. Without the human aspects on either side, then the AI is going to build something but it may, in fact, probably will not build exactly what you want.

So with all this you might be thinking, well, what are we really getting with AI coding? In a short: speed. We’re also getting a fair amount of accuracy.

If you’ve properly defined the problem, the AI system may generate thousands, tens of thousands, hundreds of thousands of lines of code. Without AI just the physical typing of that can take weeks or months, also taking into account things like typos and people getting tired. This is oversimplifying, but it is the manual work that needs to be done to build an application. What AI is doing is automating that manual piece. Automation allows you to do the right thing really quickly but it also allows you to do the wrong thing very quickly. Even though it can do a lot of that bulk work, we still need that validation piece.

When we talk about the bookend work that humans need to do in this process, this requires a certain amount of domain knowledge, skill, and overall knowledge of how systems work. This generally requires someone who has experience.

The problem that we’re starting to run into is if we look back again at our bookshelf, the book portion, the actual typing of code and debugging typos and logic issues was typically something that was done by entry-level software developers. By doing this, they would gain experience and they would be able to move up to be more senior software developers.

The challenge that we’re going to face is that, with the work that may typically have been done by junior developers now being done more by AI, you end up with a need for fewer junior developers. We’re starting to see now that companies are not hiring as many of those. That is short-sighted because when the current senior software developers are gone, retired, left the company, etc., you don’t have enough of those junior software developers to move up into those senior roles.

This is a problem that has not yet been solved. There needs to be a better way to address this because we still need a pathway for people to enter the industry to do those bookend tasks. It will be interesting to see how the industry evolves around that.

The latest and greatest coding models get better and better at generating code all the time. This is great, and when you look at code generation from this point of view, the frontier models are still helpful. However, a lot of the code that’s going to be generated, assuming you’ve clearly documented the problem and you’re validating the results, may not require as powerful of a model as you might think. This will depend on the code that you need to generate. If you’re doing something like medical diagnoses or tackling physics problems, or dealing with vast amounts of data then yes, you want a very high-power model. I suspect that a simpler model may do just as well in many cases, assuming you have those bookends in place. I don’t have hard data on this but I will be interested to see how things evolve.

As AI-assisted coding gains wider adoption, we need to look at the tools we use to manage the development lifecycle. Historically, the software world has lived in ticketing systems like Jira. This made sense when code was the bottleneck; Jira allowed us to define, plan, and track the manual labor of writing code. When the “act” of coding required significant time and resourcing, tracking velocity and story points provided a necessary although imperfect visibility into progress.

However, with AI now drastically reducing the effort required to generate code, our legacy metrics—points, velocity, and story size—are starting to break down. We have an opportunity to rethink the development process entirely.

While bugs and features both require attention, the impact of AI is most profound in the feature development process. Automation is a double-edged sword: it helps you do the right thing quickly, but it also helps you do the wrong thing faster. To avoid the latter, we must shift our focus to three core phases: Specification, Development, and Verification.

Specification: Raising the Bar

In an AI-augmented workflow, the “size” of a ticket changes. We are no longer limited to small, rigidly defined bits of functionality. We can now deliver much larger chunks of work in a single pass. This shift, however, raises the bar for our upfront definitions.

In a Jira context, the ticket should serve primarily as a tracking mechanism that anchors the specification—whether that spec lives in the ticket itself or is linked via Notion or Confluence. The goal is to create a “source of truth” that defines the feature with enough clarity that an AI can execute it and, more importantly, so a human can verify the result. The planning and point-assigning “middle” of the process is becoming less relevant; the real work is now happening in definition and verification.

Development: Capturing the Thought Process

We cannot treat AI development as a “black box” where a spec goes in and code comes out. While AI-assisted coding frees engineers from the drudgery of syntax and manual typing, the engineer’s role as a “guide” is more critical than ever.

We need to capture the thought process behind the implementation. Tools like Claude Code and Cursor already allow us to export session data that details how an engineer navigated a problem, where the AI stumbled, and how it was corrected. By automatically appending these session summaries to Jira tickets, we can maintain a complete audit trail of the engineering logic without adding manual overhead for the developer.

Verification: The New Bottleneck

If AI has removed the bottleneck of writing code, it has moved it to verification. The temptation with AI tools is to move fast and “test it in production” to see if it works. This is backward.

Because of the potential for AI hallucinations and the risk of building on top of incompletely defined requirements, rigorous verification is now more important than it was in the manual era. If we accept poorly defined or unverified code into our codebase today, we are simply compounding technical debt at an accelerated rate.

Closing the Loop

This shift isn’t something we need to wait for Jira or other vendors to solve. You can implement this workflow today by using existing tools as simple tracking mechanisms for robust specifications and automated session logs.

Over time, I expect legacy ticketing features focused on granular implementation steps to be sunsetted or deprecated as the industry moves toward this new model. By focusing on the feedback loop—using AI to improve specifications and ensuring rigorous verification—we can ensure that “faster” also means “better.”