Context Windows and Code Graphs: The Error Past the Cut

Contents11 sections

A defect at its own boundary

The local reviewer reported a syntax error. It named a line, described an unclosed construct, and sounded entirely confident. Nothing was wrong at that line. The line was the last one the model had been given, and the fault it found was the edge of its own input.

That's a truncation phantom, and there's a mechanism behind it rather than a mystery. A diff — the set of lines a change adds or removes — was capped at 60,000 characters before it went into the prompt. Past the cap the text stopped mid-construct. A model can't see past a cut. What it can see is a fragment that looks broken, so it reports the fragment, and the check went red on a defect that existed only inside the prompt.

Two more failures were running at the same time, and all three reached an operator as the same complaint: too big. All three trace to a window nobody chose and a prompt nobody sliced, and a deterministic graph of the code fixes the half that a bigger window can't.

Table 1. Three failures that all surface as the same operator complaint.

Failure 2 produced the most plausible output of the three, which is why it deserves a second look. A reviewer handed a diff and nothing else sees the change and nothing the change touches. Nobody calls into it. Nothing calls out of it. No test is visible anywhere. So the model does what a student does with an unfamiliar passage: it paraphrases. Four confident findings came back, and every one restated a comment sitting in the diff — including the comment that described the bug.

Nobody chose the window

Failure 3 is the counter-intuitive one.

The window was never too small; nobody picked it. A context window is the number of tokens a model can attend to in one request (prompt and answer together), and a token runs roughly three quarters of an English word. Left unspecified, the server reached for the model's native ceiling of 256,000.

What that costs stays hidden until you know where the memory goes. A transformer keeps a key-value cache for every token in its window, and the allocation follows the window size rather than how much of it you fill. So a prompt of about 20,000 tokens was paying for 256,000. The tax landed on every review. It showed up as latency, and the timeouts it fed are what eventually got the lane disarmed altogether.

An ungoverned setting fails in both directions. Nothing announced the 256,000. Had a server configuration lowered the default instead, the prompt would have been truncated in silence and the check would have gone green. A number nobody states is a number nobody can check.

Two halves of a fix

Govern the window. The request now carries an explicit 65,536, with the arithmetic written down beside it: roughly 10,000 tokens of system text and repository brief, leaving roughly 55,000 for the diff and the answer. That number prints in the header of every review report, so it's a visible lever rather than a server-side mystery. On an older client that rejects the option, it's the first rung shed in the existing degradation ladder — and the review still runs.

Stop sending one big document. This is the half that needed a code graph, and it's the half that changed what the reviewer could actually say.

A code graph is the output of static analysis: every symbol in the tree as a node, and every call, import, inheritance and reference as a typed edge between them. The library that builds it is pure Python over an incremental parser. No model sits in the path; nothing touches a network; the same commit in gives the same graph out. Determinism is what makes everything downstream checkable instead of probabilistic.

With a graph in hand the diff stops being one document. It partitions into at most three slices, grouped by which neighbourhood of the code each hunk touches. Every slice carries its own hunks plus a structural block: the enclosing symbols, one hop of callers, one hop of callees, each edge marked as read from the syntax tree or inferred by the producer. That block is hard-capped at 8,000 characters — structure can never crowd out the diff it competes with.

Four things follow mechanically, and only the last of them is about quality. No slice reaches the 60,000-character cap, so the phantom has nowhere left to form. Smaller prompts generate faster. A deadline now costs one slice rather than the whole review, because the slices are independent and a partial answer is still an answer. The model also sees the code around the change at last.

One graph, five consumers

Building the graph is the expensive part — the one that takes real seconds. Once it exists, everything else is a query.

Table 2. Five consumers off one build, and what each one asks the graph.

The fourth row is the quiet one. A ledger of past findings has to decide whether a new finding matches an old one, and line proximity is the obvious way to do it. It's also wrong the moment a file moves. Anchoring on the enclosing symbol retired a whole class of false matches, including one where a deleted file's hunks were confirming an unrelated file's imports.

Freshness is commit-pinning rather than a schedule. The graph is a derived artifact of exactly one commit, and it carries that commit wherever it travels; every consumer asks for the one it is working on. A mismatch degrades in the open. The reviewer notes that structural context is stale and leaves it out, and test selection falls back to the full suite. New tests arrive as new nodes in the very commit that adds them, so there's no separate index to fall behind — and that missing index is what makes development speed the fuel here rather than the threat.

Why a deterministic tool

"Which tests reach this symbol" is a graph walk. Asking a 27-billion-parameter model to guess it is how you get a confident wrong answer; asking a parser gets the same answer twice.

The measurements come off this tree rather than off a vendor page. A cold build of the full backend takes 73.4 seconds across 1,337 files; a warm one takes 20.2, a little under a third of that. The result holds 31,807 nodes and 65,156 edges. Yet the decisive probe was narrower than any of those numbers: one module, asked for the tests that reach it, returned exactly its four test files.

Two design choices sit behind that. The producer lives behind a seam of our own, since the library is pre-1.0 and moving fast; the version is pinned and no consumer imports it directly, because swapping the producer must not touch a consumer. Over-selection is the second. Where the graph widens through a shared fixture it hands back too many tests, never too few. A suite that runs long is a cost; a suite that misses the changed code is a hole.

What it doesn't solve

Table 3. What the graph does not fix, stated before anybody has to discover it.

The fence has to be in the code

The review prompt carries the repository's full diff. That single fact makes the choice of model host an egress decision rather than a configuration preference, because one mistyped variable would ship every diff to a third party, quietly, behind a green check.

So the reviewer refuses any host that isn't loopback — an address that can't leave the machine. The refusal lives in code rather than in a convention somebody documented, and it sits on the probe path and the chat path both, since the check that matters is the one on the path the data actually takes. Hosts are parsed rather than substring-matched. An address like http://127.0.0.1.evil.com/ contains the loopback address and is entirely remote.

Two more choices follow from treating the lane as a guard rather than as an oracle. A field recording whether the review ran is load-bearing, since a dog that is asleep must never be mistaken for a dog that saw nothing, and an exit code of zero has to mean the review happened. And the lane stays inert until somebody arms it. With no local model available it would otherwise report that it hadn't reviewed, forever, and a permanently red check is one people learn to scroll past. Absent is more honest than red.

The cost ladder, as the code states it

The roster of lanes is a typed table in the reviewer's own source, and the tier field encodes cost-of-catch directly.

Table 4. The lanes in run order. A reviewer produces findings; a gate produces only pass or fail.

Every defect caught at tier 1 or tier 2 is a tier-3 request never made, and a tier-4 minute never burned. Tier 2 costs nothing to run, so it can afford several focused passes where a metered lane runs once and gets rationed. The per-slice fan-out is exactly what that freedom buys.

The argument for keeping a lane on the box rests on the lane being there, whatever the local model's quality. On one evening in August all three hosted services went unavailable at once, for three unrelated reasons, on a day with six pushes; later that evening the build service stopped creating runs at all. Underneath the availability floor sits a compatibility one. A vendor ships its reviewer as a container action, and the build service runs those on Linux only, so on a Windows workstation that lane can't run whatever the configuration says.

What a fleet does to all of this

The same discipline runs across several repositories, and exactly one artifact is byte-identical between them: a shared core block inside the standards file, authored in one repository and carried verbatim by every sibling. Each downstream repository records the block's digest and fails on a local edit.

That proves there was no local edit, and it proves nothing else. The copies may still have parted company, because the block and its digest live together and a sync updates both. Only the authoring repository can see more than one tree at once, so its comparison gives an explicit three-way verdict in which "cannot compare" reads as unknown rather than as in sync.

That distinction stopped being theoretical one morning in August. A sibling sat 155 lines behind, with every check inside it green. Not one local edit had been made. Upstream had added 155 lines of rules and nobody had told the sibling, and what found it in the end was somebody cloning the two trees and diffing them by hand.

Why it matters

A reviewer reading past its own input, a window nobody chose and a copy drifting behind a green board all fail the same way: quietly, with a confident result. The cost lands on whoever trusts the check, and the fix each time was a number or a fence written down where a person can see it.

A false clean is worse than no check at all. No check is a known gap, and somebody eventually closes a known gap. A false clean is the state that lets drift accumulate behind a green board for exactly as long as nobody happens to look.

Glossary