A frontier model, the best in the world by its operator’s own description, inserted a nonsensical 700-millisecond delay to settle something. Challenged, it folded instantly: “you’re right, I was cargo-culting.”
Guillermo Rauch shared that anecdote recently, and I haven’t been able to put it down. His broader point: if you’re not reading AI-generated code (yourself, or through agents doing the reading for you), then one of a short list of things is true: you’re a beginner, the software is throwaway, you’re prototyping, you have no users or revenue, you’re knowingly taking on debt and risk, your problems are basic.
All of which, he adds, is fine. Models still make rookie mistakes and go down bad architectural paths; they’re not at full autonomy yet.
I agree with the conclusion. Review is necessary, and anyone shipping agent-written code to real users without it is running an experiment on their customers.
I want to argue with the autopsy.
“Rookie mistake” is a capability diagnosis. It locates the failure in the weights, and it implies a capability remedy: wait for better models, review harder in the meantime. But there’s a cheap experiment that breaks that diagnosis. Open a fresh session with the same model: no history, no project, no pressure. Ask it: ”I have a race between two services. Should I add a 700ms sleep to settle it?”
It will refuse. Not ambiguously; it will lecture you. Sleeps mask timing bugs instead of fixing them; the problem returns under load; here is what a real fix looks like. The knowledge that a 700ms delay is cargo-culting is emphatically in the model: the word it used to confess is proof it knew the pattern by name. And remember: this was already the best model available. Even at the frontier, having the knowledge isn’t the same as using it.
So whatever went wrong that day happened somewhere between the weights and the transcript, and it wasn’t a shortage of knowledge.
01 — The wrong autopsy
I don’t know what happened inside that session. But I run an agent pipeline every day against a codebase of my own, I read the transcripts, and I can tell you what this failure shape looks like from the inside. When an agent of mine does something it demonstrably knows better than, the transcript almost always shows one of a few conditions upstream:
The context was crowded. The reasoning that would have caught the mistake was competing with a whole session’s worth of accumulated survey results, tool output, and half-finished threads. Attention starts failing long before the window does.
The fix was nested inside another task. The agent set out to implement feature A, hit a surprise, hit a surprise inside the surprise, and the timing bug was three levels down a detour. Nobody, human or agent, scoped that fix. It inherited the momentum of the parent task instead of getting the skepticism a fresh task would have received.
The agent inferred urgency. Somewhere in the accumulated context was a signal (a tone, a deadline mention, a string of quick confirmations) that it read as settle this fast and move on. I’ve found models exquisitely sensitive to what they think you want. Sycophancy is the loud version of that sensitivity; the quiet version is an agent deciding you’d rather have it fast than right.
These aren’t hypotheticals; they’re what the run records show. One case: a review flagged a vacuous test: an absence assertion on an element id with no positive anchor anywhere in the suite, so a rename would leave the test green while the feature it guarded went dark. Inside the very fix round for that finding, the agent applying it wrote a fresh absence assertion on an id that doesn’t exist in production at all. The run record’s own words:
reintroduced the identical bug while fixing itCaught before commit, that time. Rauch’s session I can’t inspect; this one is on my disk. The rule wasn’t merely in the weights. It was in the context window, freshly stated, and it still didn’t fire on an edit nested one level inside the fix.
Before calling any of that a malfunction, notice that the same conditions bend human judgment too, and there we don’t call it a rookie mistake, we call it a judgment call. A strong engineer at a two-person startup, coding at midnight with demo day at nine, adds the 700ms sleep, writes TODO: real fix, and ships. In that context the delay isn’t even wrong.
Which exposes what the capability framing hides: whether the delay is a mistake at all is decided by the context, not by the code. An agent’s judgment isn’t merely degraded by what it arrives with; it’s constituted by it. The difference between the demo-day engineer and Rauch’s model is that the engineer knew what they were trading. The model made the trade without knowing a trade existed: the context handed it the urgency, but not the awareness. Managing how agents arrive at a task, with what in mind and under what inferred pressure, isn’t a nicety. It’s the job.
Now, a technically careful reader should object to the experiment above, and they’d be right: it proves the knowledge exists in the weights, not that the model could retrieve it that day; retrieval under a crowded context is itself a capability. Conceded, fully. Better retrieval will retire some of these failures, and I’ll be glad to see them go. But no point on the capability curve is error-free. The mistakes get more sophisticated, show up deeper into longer contexts, arrive more rarely and more plausibly. And whenever one arrives, every transcript of this shape shows one more thing, and it’s the thing that matters: nothing else in the system was positioned to object. The objection a fresh session would have raised had no second chance to fire. Whatever generated the bad line (capability, crowding, inherited urgency), the system decided whether it survived.
So the interesting question about that 700ms delay is not why did the model err? It’s: what happened to the objection?
In agent systems, a mistake matters less because it was made than because it went unchallenged.
The claim is about where the engineering leverage sits. Something incorrect gets proposed in every long session, by every model, at every capability level, the way it does in every human engineering organization: a bad proposal is inevitable; a production failure is not. Distributed systems learned this shape long ago: you don’t get reliability by assuming components won’t fail, you get it by designing what happens when they do. But import the analogy with care. Replicas buy reliability only because their failures are independent, and the first failure story below is about reviewer failures that are anything but. What transfers isn’t redundancy; it’s decorrelation, the discipline of engineering failures apart.
The same move works on judgment. You don’t make an agent system trustworthy by making every agent right. You make it trustworthy by making it hard for a wrong decision to go unchallenged, which turns the question into an engineering one: where are the points at which a bad proposal can be opposed, do they exist at all, and are they starved of the context they need or drowned in context they don’t?
This reframing matters because of what it changes about the remedy. A capability diagnosis leaves you one move: wait for better models, and review harder in the meantime. A systems diagnosis hands you levers that exist today: the crowding, the inherited urgency, and the missing second chance to object are conditions of the pipeline, and the pipeline is yours.
If that’s right, then reading the code, necessary as it is, inspects the last artifact of a process whose real variable sits upstream: the conditions under which objections fire and survive.
That is a thing you can engineer, and the rest of this essay is one long attempt to do it, told through its failures: the architecture I built to manufacture objections, the fabrication it let through anyway, and the day the objection machinery itself went quiet without anyone noticing. I’ve spent the better part of this year on it, in a personal laboratory: a side project (open-domain event extraction from news articles) that I build almost entirely through agents. The product is the excuse; the pipeline is the experiment.
02 — Designing for “unchallenged” as the failure mode
The pipeline is spec-driven: an idea becomes a written proposal through its own guided flow (an interview, a codebase survey, an adversarial critique), and the proposal is reviewed to zero blockers, then implemented task-by-task by specialist agents, verified, reviewed again as a diff, and opened as a PR with the accumulated review evidence attached. Spec-driven development is not a new idea, and it is not the point. The point is that a spec is the artifact an objection can attach to before code exists. Almost everything else in the pipeline is machinery for manufacturing objections and giving them somewhere to land.
Three design decisions carry most of the weight.
Nothing forces a plan to face the rules
Like most serious codebases, mine has accumulated rules: architectural invariants, naming conventions, hard-won “never do X” entries with incident numbers behind them. And my tooling is generous with them. Every rule file is loaded into the working context, whole, in every session; the planning agent is never missing a rule. It holds them the way Rauch’s model held the case against sleeps: present, ambient, unconsulted, competing with a session’s worth of survey results for the same attention. The run-record case in the first section already showed what that’s worth: a rule freshly stated in the window still failed to fire. Presence is not a check, and the author of a plan is not a reviewer of it. So the pipeline includes a rule-compliance checker: a fresh agent whose only job is to hold the proposal against the applicable rules and deliver a verdict, at plan time, before implementation is allowed to start, told explicitly not to audit from memory but to re-read the rules it is checking. It turns “the rules are in context” into “the rules have been checked,” in the window where wrongness is still cheap. (Hold that thought: the rule checker stars in the second failure story.)
The devil’s advocate is deliberately starved
Every proposal is challenged by an adversarial agent before it’s accepted. The design constraint that took me longest to learn: this agent must know less than everyone else. My orchestrating agent spends its early phases interviewing me, surveying the codebase, finding all the reasons the idea fits. The skill file that governs it now states the resulting hazard in one line: the coordinator “will unconsciously steelman the idea while pretending to challenge it.” Enthusiasm travels through context. Bias travels through context. So the advocate runs as a fresh agent with a minimal brief (the idea, not the survey; the problem, not the solution) precisely so it can’t inherit the coordinator’s investment.
There’s a companion rule I resisted longer than I should have: when a session arrives already deep in an investigation, the temptation is to skip the independent critique, because so much analysis already exists. The skill mandates the opposite: the more prior context, the more the fresh critic matters, because the prior context carries the most bias.
Witnesses don’t get the defendant’s story
After every pipeline run, a retrospective agent writes an honest record: what went wrong, which agent erred, what it cost, what rule change would prevent recurrence. The naive design has the orchestrator write this: it was there, it knows everything. That’s exactly why it can’t. An orchestrator narrating its own run smooths its own errors out of the record, not maliciously, but the way any narrator does.
So the retro is written by a fresh agent that did not do the work and is structurally denied the coordinator’s self-report: it reads artifacts (git history, state files, review evidence), and one of its schema fields says, in so many words:
reversals: derived from git, not self-reportNotice the shape of all three: none of them is “add more context,” and none of them is “add more agents.” They are decisions about who is given the full history and who is protected from it. The implementer gets everything. The critic gets a clean room. The witness gets the evidence but not the story. And each of these is a role, not a headcount: implementation fans out across specialist subagents task by task, and a critique or a retrospective can split the same way. The information boundary is drawn around the role; inside it, the work can spread across as many agents and sessions as it needs.
Independence, it turns out, isn’t achieved by putting agents in different boxes; it’s achieved by controlling what flows between the boxes. The clean room and the withheld self-report are decorrelation devices: they don’t add reviewers, they make the reviewers’ failures independent.
Most of the multi-agent architectures I see discussed, mine included for months, are org charts: boxes, titles, arrows. The titles turned out to be theater. What carries the weight is who knows what and, just as deliberately, who doesn’t.
And with all of that in place (a plan-time checker, an adversary, independent witnesses), the system still let a fabrication through three gates. Here are both stories.
03 — The failure the reviewers couldn’t see
Case 1 — the premise that wasn’t true.
A design document justified a schema decision with a claim about the codebase that was structurally impossible, while the conclusion it supported was correct. Three review gates passed it. It was caught two weeks later, when implementation began, by the one reviewer with a reason to open the file, and the fix became a machine-checked citation rule.
A few weeks ago, a design document in my pipeline justified a schema decision with a technical claim about the existing codebase. The claim was not just wrong. It was structurally impossible: it asserted that a federation directive sat on a GraphQL enum, a position the spec doesn’t even permit, and a thirty-second repo-wide grep confirms nothing of the kind exists. But the conclusion it supported was correct: the change genuinely was expensive, just for a different reason.
The proposal passed a rule-compliance check, an architecture review, and schema validation: three separate review gates. The fiction sailed through all of it.
It surfaced two weeks later, and not through any of them. When implementation began, the pipeline’s entry review dispatches domain specialists alongside the generic gates, and the GraphQL specialist, the one reviewer with a concrete reason to open the schema file, flagged it under a heading that says exactly what it is: corrections to stated facts. The run halted and raised it to me.
A fair question at this point: if the conclusion was right anyway, why does the falseness of the premise matter? Three answers, all from this one incident’s paper trail.
First, the premise wasn’t decorating the cost; it was concealing it. Stated correctly, the real mechanism exposed a hazard the plan had no answer for: mid-rollout, an old replica of a service can be asked to serve a value that only the new replica understands, and the task list had no two-phase plan for that. With the true cost finally on the table, the schema change was withdrawn from the proposal entirely and deferred. The false premise had been quietly mispricing the decision it supported.
Second, design documents outlive decisions. A conclusion gets implemented once; a premise gets read forever: by the next task, the next agent, the next maintainer reasoning from the recorded claim instead of from the code.
Third, and I wouldn’t have believed this without the paper trail: the premise reproduced after being refuted. The correction was recorded in the proposal itself. Later in the same run, an implementing agent wrote a code comment restating the original claim, the one already killed in review. That comment survived the post-implementation review, the coordinator’s own read, seven specialist reviewers in total, and was finally caught by a bot after the PR opened. The run record’s own words:
a correction captured in a proposal did not propagate into code written later in the same runFalse premises don’t just survive review. They reproduce.
The general form of the problem goes well beyond agents:
A false premise that supports a correct conclusion is invisible to every reviewer who agrees with the conclusion.
Review, human or LLM, of code or of plans, is overwhelmingly normative: is this good? does it comply? will it break? The descriptive question (”is what this document says about the world actually true?”) gets asked only as a side effect of diligence. A conscientious human sometimes opens the file; a well-prompted review agent sometimes greps. But nothing makes them. Premise-checking is nobody’s designed job, and it’s the first thing to decay as the diff grows. A reviewer who agrees with the conclusion has no structural reason to check the premise: my fabrication was eventually caught by exactly the one reviewer whose domain gave them such a reason, and it took two weeks to reach them.
Worse, adding reviewers of the same kind compounds the illusion of coverage without raising the odds of detection: seven reviewers asking does this plan look good? are not seven chances to catch a false fact; they’re one question asked seven times, with correlated blind spots.
Detection stays a matter of luck while confidence grows, and every “looks good” is another attestation laminated over the unchecked premise, so it reaches the human who merges better credentialed than a claim nobody reviewed at all. In the terms the opening section set, the check was starved: for two weeks, the claim never reached a reader equipped to doubt it.
What worked, in the end, was almost embarrassingly primitive. Load-bearing factual claims about existing code now must carry a citation the machine can re-execute: file, line, and the quoted text at that line. A path and a line can be fabricated plausibly; quoted text has to actually be there. From a real proposal:
**Evidence:** `k8s/base/settings/deployment.yaml:12` — ` replicas: 2`Claims of absence carry the search that establishes them, and the search is re-run. If you cannot cite it, you may not assert it: the claim moves to an “Open questions” section instead. A deterministic checker verifies every citation before any reviewer reads a word. No model judges truth anywhere in the loop.
Be clear about the guarantee, because its narrowness is the point. The checker certifies that the cited evidence exists: the quoted text really sits at that line, the claimed search really comes back empty. It does not certify that the evidence entails the conclusion; a real-but-selectively-quoted line passes. That division of labor is the design: the machine holds the facts still, and reviewers judge what the facts support, a job worth doing once the ground under it stops moving. That division of labor still has an empty seat, a premise checker: a fresh agent handed the document’s claims stripped of the conclusions they support, verifying each against the repo, unable to agree with a conclusion it cannot see. Anchors would cut its workload; it would cover the entailment gap the citation checker deliberately doesn’t claim.
The deeper mechanism is behavioral, and it fires before the checker ever runs: to write the citation, the agent has to open the file, and opening the file is what kills the fabrication. I’ve since extracted the convention and checker into a small open-source tool, nullius, because nothing about the problem is specific to my pipeline.
The mechanism half of the incident got its own discipline, too: compatibility risks must now name when they bind, from a closed vocabulary of moments, because a risk framed on a mechanism that cannot occur produces a mitigation that does nothing, while reading as diligence.
04 — Who checks the checkers
Case 2 — the reviewer that wasn’t there.
The rule checker went silent for a week of runs and nothing looked wrong. Four confident root causes later, the fault was a cost no file records: the harness was attaching every connected integration’s tool catalog to every dispatch, and the checker’s small model had no headroom left. The fix was a two-line allowlist, and a pipeline that now counts dispatches and delivered reports separately, so silence shows up as a zero.
The second story is the one that keeps me humble, because it’s about the review machinery itself.
The rule checker had been quietly doing its job for months. Then, one week, it went silent. Here’s the uncomfortable part: nothing looked wrong. The pipeline ran, implementation proceeded, PRs opened; the system produced success-shaped output around the hole. You had to read the run records to notice lines like:
review dispatches: 5
review reports delivered: 0I had a diagnosis ready, and I loved it. Weeks earlier I had done something responsible: the backend rules file had grown long enough that my coding tool complained, so I split it into focused files (GraphQL rules here, permission rules there), and the checker’s instructions still cited the file that no longer existed. My cleanup had broken the reader. Tidy, self-deprecating, satisfying. The ledger killed it with a date: the split had landed ten weeks before the first failure, and the checker had produced substantive findings the whole time in between. A cause present through ten weeks of success cannot explain a failure that begins on a specific day.
So I “fixed” it three more times. The dispatch brief was too fat: tightened three separate times, once down to 450 words, no effect. The checker read too much: I cut its discovery reads tenfold, measured the cut, and the validation dispatch died anyway. Fixed overhead: a probe agent told to read nothing and reply OK survived next to a real audit that died, the fresh-session probe from the top of this essay turned on my own machinery. I misread even that clean result, blaming the reading list. A commit in the record is literally titled ”third root-cause also refuted.” The investigation that followed concluded the failure was not diagnosable from the agent’s definition.
It was right, and that was the clue. The cause didn’t live in anything the agent could read, so no amount of reading the agent’s inputs could find it. My harness attaches the tool catalog of every connected integration to every subagent it dispatches: about 165 tool schemas, injected before the agent reads a single byte, recorded in no file and visible in none of the artifacts the investigations examined. The checker was one of two agents pinned to a smaller, cheaper model, and the invisible catalog had consumed its headroom; every real audit died of arithmetic before it began. The crowded context from the first section, in its purest form: content the agent never used, occupying the room its judgment needed. The first story starved a check; this one drowned one. The same failure, run in opposite directions.
The record even held a cure that worked for the wrong reason. One desperate run had re-dispatched the checker on a larger model and gotten back a full, clean report; the bigger window simply absorbed the bloat. Had I shipped that as the fix, everything would have looked resolved, permanently, at several times the cost, with the mechanism still wrong: a working fix on a false premise, the first story’s disease wearing a green check. The actual fix was an allowlist. Two lines, declaring which tools the reviewers actually need. They stay on the small model.
Two lessons, one uncomfortable, one useful.
The uncomfortable one: count the diagnoses. The refactor story, the fat brief, the reading list, the model tier. Four confident root causes, several of them “verified” by measurement at the time, all four wrong, in a pipeline whose entire premise is that unverified claims are dangerous. Epistemic discipline isn’t a property you install; it’s a property you keep losing and re-earning, and the person it protects you from most reliably is yourself.
The useful one: in an agent system, silence is indistinguishable from success unless you make absence loud.
A reviewer that returns nothing looks, from the orchestrator’s seat, exactly like a reviewer with nothing to say. Three different states hide in that blank (I checked and found something, I checked and found nothing, nobody checked), and orchestration layers love to collapse all three into “no issue reported.” Absence is layered, too: the run records were complete about everything the checker read and silent about everything the harness injected. The most dangerous cost is the one no artifact records.
So the pipeline now treats absence as a first-class, checkable fact. Run records count dispatches and delivered reports, separately, so a zero is visible. And every review-evidence entry the orchestrator writes must end with a section titled ”Coordinator corrections since last append”: what it got wrong since the last entry, how it was caught, what changed.
The contract has a line I’d defend anywhere: writing “None.” is a valid answer; writing nothing is not. An explicit “None.” is a claim someone can later disbelieve and check. An omission is nothing at all. The whole game, over and over, is converting silent gaps into assertions that can be wrong out loud.
05 — Open ledger
In the spirit of all that, here is what the pipeline does not do well right now, from its own retrospectives rather than my memory of them:
It learns faster than it applies. The retrospective agent proposes concrete rule changes with evidence. Some get filed in near-identical words two runs apart, because the first one was never applied. The loop from lesson recorded to lesson enforced still runs through me, and I am the slowest component in the system.
Green checks launder later mistakes. In one recent run, a specialist agent ran a destructive probe on a live file and terminated without reverting it. Type checks had passed earlier, so nothing looked wrong. It was caught only because the orchestrator re-read the file instead of trusting the earlier green. “Verified once” quietly became “verified,” and nothing in the machinery objected to the tense change.
Silence handling is still crude, and it corrupts the measurements. In the same run, three healthy subagents finished their work and their reports simply never arrived; an infrastructure failure, not an agent failure, but the orchestrator treated the recoverable silence as terminal and reconstructed the work from the file tree. That run’s ledger credits five caught defects to the orchestrator’s own re-checking and zero to specialist review. Dead, silent, and withheld still look too much alike, and a zero is a zero.
All of this costs real money and real latency. Every asymmetry I described is another dispatch, another context, another model call. Some runs the devil’s advocate changes a design decision and the record says so explicitly; some runs it earns nothing. A disciplined pipeline is slower than an undisciplined one every single day, and pays for itself on the days something would have gotten through.
A fair pushback I’ve heard more than once: this is a lot of machinery, a lot of tokens, all spent just in case a model gets something wrong: the cost of paranoia. But nobody calls a three-node cluster the cost of paranoia; it’s the cost of availability, priced against the outage rather than against the single node. The records give a rough denominator, too: across twelve retro’d runs, about a hundred defects are attributed to review before merge (an attribution the witness itself flags as softer than the ordering it implies), against fifteen recorded reversals and fifty-one proposed rule changes. Not a controlled study; a ledger. But it is the difference between “two catches and a year of overhead” and a yield. What the objection does get right is scope, so let me draw it explicitly: none of this is a prescription for a developer shipping features with an assistant at their elbow. In that setting, the human reading every diff is the objection machinery, and most of this would be redundant weight. The discipline is built for the regime the industry says it’s headed toward: pipelines where agents carry more of the work and a human watches less of it. The less often a human reads the transcript, the more the objections have to be manufactured, and the more absence has to be made loud. That’s the transition this laboratory is rehearsing.
And by Guillermo’s own list, a personal project is exactly the place where I’m allowed to skip all of this. I run it anyway, because the discipline is the thing under test: I’d rather learn what trustworthy agent work costs on a codebase where mistakes are cheap than discover it later on one where they aren’t.
06 — Where review goes
Guillermo’s post ends where I’d start the next argument: he’s in the camp that the need for this reading diminishes over time, that most code becomes something like assembly (an artifact humans rarely read, produced by a toolchain they trust), while global software infrastructure riding on these models deserves respect in the meantime.
He’s right that some review genuinely disappears. Nobody proofreads compiler output anymore; whole layers of scrutiny evaporated as toolchains earned trust, and layers of today’s code review will evaporate the same way. But compilers earned that trust through determinism, and models don’t have determinism to offer. So the review that remains doesn’t shrink so much as migrate: toward whatever layer is currently the least trustworthy.
In my pipeline it has already moved once: the highest-leverage question stopped being is this diff correct? and became is what this plan claims about the world true? Code, then plans, then premises.
And it’s visibly moving again, to the layer underneath: the pipeline itself. The same asymmetries as before, one level up, plus the question the second story forces: is the objection machinery even alive? That is not a regress of reviewers reviewing reviewers; it’s the same monitoring problem every operated system has, and it takes the same answer: instrumentation and visibility. The counters, the run records, the witness reports exist so a developer can see the pipeline’s health without re-reading everything it does. The regress terminates because each layer down needs less judgment than the layer it watches: a reviewer can have a bad day; the counter that reports whether the reviewer ran cannot. The ladder ends at arithmetic, not at another mind. That last review has no finish line, because review systems fail in ways success-shaped output conceals. I know because mine did.
A mistake matters less because it was made than because it went unchallenged, and silence is what unchallenged looks like from the outside.
Which is why the anecdote stays with me. The delay was never the interesting bug.
The silence was.