What Does Good Look Like?
I asked an LLM a question last month and got a technically sound wrong answer.
In Slow Productivity, Cal Newport describes how writers develop taste. They read good writing, produce their own, and take in feedback until they build an internal standard. That standard is what lets them judge a draft before anyone else sees it. LLMs break that loop. The model doesn’t need that standard to produce fluent, confident output, and increasingly, neither does the person reading it. If you don’t know what good looks like, you have no way to know whether what you got is good. That’s what this post is about: not whether the model got the fact right, but whether anyone downstream can tell.
“Correct” and “right” are different failure modes
Two different things can be wrong with an AI answer, and most people are only checking for one of them.
The first is factually wrong. Hallucination, bad training data, an unlucky sample from the probability distribution. This is the failure mode people mean when they say “always verify AI output.” You check the answer against a source of truth and it holds up or it doesn’t.
The second is factually correct and wrong anyway. Ask an LLM to design a network for a mid-size healthcare org and it will hand you sound routing, sensible segmentation, a defensible firewall policy. Nothing in the design is technically incorrect. It might also route PHI traffic through a segment that violates the org’s existing BAA boundary, because nobody told the model that boundary exists. The design is correct and wrong at the same time.
These need different fixes. Fact-checking catches the first kind of error. It does nothing for the second, because there’s no fact to check. The design didn’t fail a lookup. It failed a constraint nobody wrote down. Teaching someone to fact-check AI output teaches half the skill and leaves them confident they’ve covered the whole thing.
Writing used to force the check, now it doesn’t
Before AI, if you needed a network design, you had to write it. Not type it up, construct it. Decide which segment carries PHI. Decide where the firewall rules get strict and where they can loosen. Decide what a vendor gets access to and what they don’t. Every one of those decisions ran through whatever judgment you’d built, whether you were consciously exercising it or not.
That’s the mechanism worth naming precisely. Writing the design didn’t just produce the design. It forced the designer to pass every choice through their own internal check, the same taste Newport describes writers building over years. You couldn’t hand someone a network diagram without first checking it against everything you knew about the org that was never written down. The check wasn’t a separate step. It was welded to the act of producing the thing.
AI output arrives with no check welded to it. The model doesn’t have the organization’s unwritten constraints, so it can’t run the design through them, and it isn’t trying to. That’s not a flaw in the model. Applying that check was never the model’s job. The flaw is in assuming the check still happens somewhere. For years it happened automatically, as a side effect of a human having to construct the thing by hand.
AI didn’t make the check harder or the answers worse. It made the check optional, for the first time, in a part of the workflow where it used to be structurally unavoidable. Whether that optional check gets exercised now depends entirely on whether the person receiving the output already has the judgment writing used to build.
Judgment has only ever come from two places
Two mechanisms have historically built the standard writing used to check against.
The first is tribal. A junior engineer proposes a design, a senior engineer tells them what’s wrong with it and why, and that correction gets internalized over years of repetition. This is apprenticeship. It’s slow and expensive, and it’s remarkably durable, because the judgment gets rebuilt inside a new person each time rather than stored anywhere fragile.
The second is programmatic: documentation. I’ve argued before that most organizational documentation fails at transferring understanding. Reading it was always optional, and the document was never the unit of understanding, the conversation around it was. A document that lists your org’s constraints doesn’t transfer judgment. It transfers information. Judgment is knowing which constraint to check for in a design you haven’t seen yet, and a list doesn’t teach you which situations to reach for it in.
There’s a third mechanism, and most orgs skip past it without naming it: externalized heuristics. Not a knowledge base of everything the org knows, that’s the shallow document problem in a new outfit. A specific, curated rubric of known failure modes for one class of decision. For network design: check PHI segment boundaries against every existing BAA first, check firewall defaults against the last three incidents, check vendor access against least-privilege before anything else. Closer to a pilot’s pre-flight checklist than a wiki page. It doesn’t teach someone to think like a senior engineer. It catches the specific ways this org’s designs have gone wrong before.
That’s also where this connects to something more practical. A checklist like that is cheap to build for decisions where verification is fast and the failure modes are well understood. It’s nearly useless for decisions where checking the output costs as much as producing it yourself. Sorting tasks by how expensive they are to verify, not by how often they come up, is the actual design question underneath all of this. It’s a different question from “does this person have judgment.”
Tribal knowledge is the default most orgs actually run on, whether or not they’ve built a checklist for anything. And it has a structural dependency nobody states out loud: it requires a continuous pipeline of juniors becoming seniors. Every org’s quality control is quietly resting on that pipeline staying intact.
What happens when the pipeline breaks
Layoffs over the last few years have hollowed out entry-level and, in plenty of orgs, mid-level engineering roles. I’m not going to argue about why that happened or how much of it is attributable to AI. I don’t have the data to make that case and it isn’t the point. The mechanism holds regardless of cause: fewer juniors get hired, fewer get mentored, fewer become the seniors who mentor the next round.
Here’s the consequence that matters for this post. A new hire today gets handed an LLM on day one. They don’t get handed the years of correction that used to build the judgment to check what it produces. They inherit the tool without inheriting the taste. They can prompt for a network design as capably as anyone. They have no way to know if the design that comes back is wrong in the second way, correct and wrong at once. Nobody has spent three years telling them what’s wrong with their designs and why.
This is where the two threads in this post meet. The epistemic problem is that AI output arrives without the check that writing used to force. The pipeline problem is that the people receiving that output increasingly don’t have the judgment to run the check themselves. The mechanism that built that judgment is being dismantled at the same time the check became optional. Neither problem is bad enough alone to be the whole story. Together, they compound.
Apprenticeship might not be the floor
The obvious objection: maybe three years of apprenticeship was never the actual floor. Maybe it was just the only delivery mechanism anyone had built, and a faster one is possible.
There’s a real case for that. Apprenticeship teaches judgment through a slow, sparse trickle of failures, one bad design at a time, spread across years. That’s how many real projects a junior actually touches. Curate the actual near-misses an org has hit instead. The design that looked fine and violated a BAA boundary. The migration that passed every test and still took down a dependent service. Put a new hire through deliberate, concentrated exposure to cases like those, and you might build a working approximation of judgment in months instead of years.
I think that’s plausible. I don’t think anyone has proven it. The difference between a checklist of known failure modes and a taught intuition for spotting new ones is exactly the difference between the second and third mechanisms from earlier. The taught intuition has to catch cases that don’t match any entry on the list. It’s not obvious that concentrated exposure closes that gap, rather than just widening the set of cases the checklist covers. Call this a real hypothesis worth testing, not a solved problem.
Generation cost dropped to nearly zero. Verification cost didn’t move. In a lot of organizations it’s about to get more expensive, because the people doing the verifying are the same people the pipeline stopped producing. That’s the actual currency now. Not which model you use, not how good your prompts are. Whether your org can manufacture judgment faster than tribal apprenticeship ever did. Not by writing a better document. By finding a substitute for the correction that only someone who has seen the failure before can deliver.
Writing used to force the check as a side effect of producing the thing. If that’s no longer true, and it isn’t, what forces it now?
Steve, this names something I’ve seen firsthand over three decades of work across healthcare, research, and enterprise strategy: the check never lived entirely in the document or system. It lived in the person, welded to the doing.
Michael Polanyi gave us language for this in The Tacit Dimension (1966): “we can know more than we can tell.” Your “constraint nobody wrote down” is exactly that—the organization’s judgment living in heads and habits rather than on the page.
Your closing question is the design brief for my work, so here is my working answer. If producing no longer forces the check, I design the check into the structure. So, for example, in an enterprise RAG harness, you begin with domain experts verifying an initial set of correct answers, so outputs are measured against their judgment, not the model’s fluency. Human review remains a defined checkpoint where judgment matters. Every “correct and wrong at once” catch is reviewed; under the team’s governance, durable corrections are written back into shared knowledge, while one-time conditions remain case-specific.
I would extend your third mechanism—the curated rubric—with a governed write-back loop. It turns a snapshot of known failure modes into a shared system that continues learning from the people who can still tell when something does not fit.
On your apprenticeship hypothesis, my work in human and organizational learning gives me reason to take it seriously: perhaps the years were the delivery mechanism, not the floor. But what those years delivered was corrective conversation within relationships. Curated near-misses may provide cases more quickly, but people still provide the correction.
This is a hypothesis worth testing and a design problem worth pursuing.