Hosted Code Review: The YAML Parsed and Nothing Read It

Contents14 sections

The change that parsed and did nothing

One entry went into the configuration of a hosted code reviewer in August. (A hosted reviewer is a commercial service that reads a pull request, meaning a proposed change to a codebase, and comments on it the way a colleague would.) That entry pointed the service at a set of instruction files. Three separate graders would now work from one shared body of criteria instead of three private ones. The configuration parsed. A test went green. Everything shipped.

It did nothing at all.

Entries of that form are directory-scoped, a detail buried in the vendor's own documentation and easy to read past. Such a guideline governs one folder and everything below it, and nothing outside. Every rule inside those files aimed at the application trees. The files themselves sat in a configuration folder somewhere else entirely. So no rule reached a path it had been written for. Three graders carried on reading what they had read the week before.

A comment sits above that entry now, and it carries the whole lesson: the test we validated it with only proved the YAML parsed, which is not the same claim.

That shape of failure is the one worth being afraid of. Something opens the first file, reports success, and never touches the rest. The board stays green. In the defect registry it files under surrogate metrics — a measurement that stands beside the thing you care about and then moves independently of it. Two of the three graders flagged the entry once it reached review. That is at least small evidence that two of them were reading something.

What follows is the fix, and it starts with the size of what the graders were supposed to read.

Two bounds, and they fail differently

Every automated reviewer is bounded twice. The two bounds get discussed together; they behave nothing alike.

Table 1. The two ceilings on a hosted reviewer, and why only one of them gets noticed.

A token is the unit a language model counts in: roughly three quarters of an English word. Size is a hard number, published by the vendor, and it announces itself the first time a request crosses it. Shape is a set of filters, mostly written by the customer, and it announces nothing ever. Take two reviewers: one never opened the file, and the other opened it and found nothing wrong. They hand back the same artifact, which is silence.

Size cost about a week. Shape cost the August change above, and it would have cost considerably more if two graders had not caught it in the act.

The size, measured

One grader loads a single instruction file into every review it runs. The published ceiling for the whole prompt is 110,000 tokens. That file had reached 365 KB — roughly 90,000 tokens — so about eighty percent of the budget was gone before a line of the proposed change had been read. Reviews from that lane failed with a complaint that the prompt was too big.

Diagnosis arrived by accident. The same failure fired on a documentation-only change whose diff ran to a handful of lines. That ruled out the size of the change and put the blame where it belonged. A large diff would have been the comfortable explanation; the uncomfortable one is that the instructions had eaten the window.

Here is what the governing files weighed once somebody measured them instead of estimating.

Table 2. The five governing files, measured after the carve described below. The instruction file stood at 365 KB before it.

Set the registry against that ceiling and the arithmetic stops looking like a tuning problem: the registry alone runs to one and a half times the entire window. It is also the number most likely to keep climbing, since a window is fixed and a record of defect classes only ever grows. Each entry holds a kind of bug that already shipped here and already failed. It carries three things: the symptom, the root cause, and the check that would have caught it. Nothing leaves. Where a class can be mechanised it becomes a lint rule and the entry retires itself. That trims the growth rate; it never reverses it.

An append-only record and a fixed window diverge. That is arithmetic rather than a vendor defect, and past some point the two simply part company.

One repository makes the pressure look gentler than it is. The same standard governs a fleet of them. Each tree accumulates its own classes and carries its own copy of the criteria, so the total climbs on two axes at once: the number of repositories, and the age of each. A single repository crossing a ceiling is a scheduling problem. A fleet crossing it at different times, quietly, is what keeps a maintainer awake.

The shape, which is worse

Eight constraints stand between a change and a hosted review here. Five came out of a configuration written in this building. That is the part that stings.

Table 3. Every gate between a change and a hosted review. The first three were written here.

Read the third row on its own and the design problem stands out. A hosted grader reads exactly one file of ours. The registry stays shut, and so do the standing rules and the document index. Everything a reviewer ought to know about this codebase has to fit through that hole, and the hole keeps its width while the knowledge grows.

Four moves

Carve

Take out of an instruction file everything that does not need to be there. A 219-row index of documents accounted for 55.4 percent of that file: more than half a review budget spent on a table of contents. It moved to a file of its own at the top of the repository, so every relative link inside it kept resolving. What remained came down to 175 KB.

A system test holds the line now, and it asserts the budget rather than a byte count (a byte count invites somebody to raise it by one). Half the grader's window is the ceiling. Its failure message ends with a sentence aimed squarely at whoever reads it at eleven at night: do not raise this limit to make the test pass.

Compile

No reviewer can be made to read more files. Any reviewer can be handed more inside the one file it already reads.

So a generator walks every class heading in the registry and rewrites a marked block inside the criteria file. Two hundred and thirty classes land there as 35,766 characters, deterministic, with no model anywhere in the path. Copying entries across by hand was the alternative; an append-only registry copied by hand drifts inside a week. Compilation turns a file the reviewer will not read into a section of a file it will.

Pin

A generated file nobody checks is a stale file with extra steps.

One test runs the generator in check mode and fails with the exact command needed to regenerate. A second pins that the parser sees every heading in the registry, so a new heading style cannot quietly drop classes out of the reviewer's working set while every check stays green. That second test exists because the first alone would have caught nothing.

Pinning also has a shape inside the build, learned at cost. Both the registry and the criteria file are classified as code rather than documentation, which keeps them off the documentation-only fast path. One change added a single bullet, took that path, skipped the test shards, and merged green. It left the local reviewer's model definition stale on the main branch — briefing it on criteria that no longer matched their source. The next change along ate the red build that should have fired on the first.

Duplicate, deliberately

Three graders read three different files, and no configuration path joins them. August settled that question.

So a new defect class goes into the registry, into the criteria file, and into the third grader's instruction file, by hand, every time. Only the registry block inside the criteria file is generated. That duplication is a stated discipline rather than a mechanism, and it is written down as one; a discipline somebody believes is a mechanism is a discipline that quietly stops being performed.

Three claims that get conflated

Underneath the sizes and the filters sits a harder question. How do you confirm a reviewer read what you gave it?

Table 4. Three different claims that a single green check is regularly asked to stand in for.

August was a first-row test being read as a third-row guarantee. Schema satisfied, everybody moved on.

What a receipt looks like

On the hosted side the only honest receipt is a reviewer's own tool trace, and exactly one of the three services emits one. Its review body on a single pull request carried the traces inline: a knowledge-base load returning 13,343 bytes, then two repository-wide searches returning 8,272 and 38,100. On that run the service demonstrably went past the diff and read the tree. Stated precisely, that is evidence about one run rather than a guarantee about every run. No equivalent exists from the other two. One offered only its complaint about prompt size; the other offered a quota error.

A softer signal came out of the same change. That service raised a finding. Shown contrary evidence in a reply, it re-ran its own inspection, withdrew the finding, and wrote down what it had learned. Any lane that can be moved by evidence is reading something.

Locally the receipt is structural instead of lucky. A report header states five things every run: which model answered, over what window, how many findings came back, how many registry entries the brief carried, and whether each supporting piece loaded (the teaching corpus, the structural context, the review slices, the build notes).

Two details there carry the weight. Registry entries render as a count rather than a flag. A failed read shows as unavailable, never as a quiet zero, because "the brief held all 230 classes" and "the brief loaded" are different claims. Every line renders whenever it is supplied, including the cheerful ones. An ungoverned context window went unnoticed here for months for a single reason: nothing ever printed it. Anything able to stop loading quietly has to be made unable to do it quietly.

Sitting under that is an honest-empty rule, which matters more than it looks. An unreadable ledger degrades to a brief that states it is empty; it never degrades to an empty string. Splice in an empty string and a section heading renders with nothing beneath it, which a model reads as other reviewers having found nothing rather than as a collection that has not happened yet.

Which leaves the strongest claim available, and it is narrower than anyone would like. Nobody here can prove a vendor read all 230 classes. Anybody here can prove, on every commit or the build fails, that all 230 sit in the one file the vendor says it reads.

What none of this fixes

Table 5. The limits that survived the fix, each stated as the strongest true sentence available.

The mismatch underneath

One shape shows up at every layer here once you have seen it once. Something of ours grows; something of theirs is fixed.

Defect classes accumulate; a context window does not. Output rate climbs, because AI-assisted development moves the number of pushes and leaves the quota exactly where it was. A per-push meter stays put. Repositories multiply, and the one file a hosted grader will read stays one file. Each time the answer is the same move wearing a different name: measure what is actually being sent, take out whatever fails to earn its place, generate whatever can be generated, then pin the result so the next person cannot quietly undo it.

None of which settles the question that August entry asked. The receipt problem has no vendor-side answer, so it gets solved on the side of the machine that can be opened — which is why the reviewer worth running first is the one that can be made to show its work, rather than the one most likely to be right.

Glossary