Pareto in Practice: Eighty Percent from Three

Contents11 sections

The rule is a finding you measure

The 80/20 rule gets quoted as a planning heuristic. It's a property of a distribution, and you either measured it or you didn't.

The folklore drops three things. It's an observation rather than a law, so some distributions are skewed and some are flat, and "80/20" is a finding somebody made rather than a licence anybody holds. The two numbers need not sum to a hundred: ours come out at 80/3 for one population and 53/19 for another. And it shows where the mass sits rather than what matters, because a test that runs in four tenths of a millisecond can be the one that catches the outage.

"Focus on the 20% that matters" becomes a plan only when somebody says which 20%, and how they measured it.

Both populations come from one repository, and every number comes off this tree.

The sharp one: where the clock goes

The backend suite holds 14,078 tests and runs 24.0 minutes serially; the median test finishes in 11.2 milliseconds. The slowest one takes 153.5 seconds — roughly 13,700 times the median.

Sort by duration. Accumulate. The shape arrives before you have finished reading the first few rows, and it isn't the shape the rule predicts.

Eighty percent of the runtime comes from 3.3 percent of the tests. Call it 80/3. The difference between that and 80/20 is the difference between a project and an afternoon: 462 tests to examine instead of 2,815. A suite-wide push to make the tests faster would spend most of its effort on the 96.7 percent that jointly cost a fifth of the clock.

It also means the median lies, and lies badly. Half the suite finishes under 11.2 milliseconds; one test runs for two and a half minutes. Any decision made on the average test is wrong about nearly every actual test.

One level higher, where the tests are grouped by what they prove, the same sharpness shows up again.

Table 1. Runtime by tier. One tier is three quarters of the bill.

One tier carries three quarters of the cost. That single row is what makes the draft-scoping decision worth having — and it's also what made the version we got wrong look so attractive.

The flat one: where the defects come from

The defect registry is an append-only record of bug classes that already shipped here and already failed. Nothing ever leaves it. If Pareto held, a few families would dominate, and the sensible move would be to pick those few and harden them until the recurrence rate fell.

Two hundred and ten entries spread across 104 families. The largest accounts for 12.4 percent of the registry, and 96 of the 104 hold one entry or two. Nothing dominates. There's no single cause available to eliminate, which is an unwelcome answer and still the answer.

Table 2. The same rule tested against two populations in one repository.

Measuring both populations is what made the real point visible. Pareto is an understatement for runtime and simply false for defects. Had we assumed 80/20 for bugs (the way a roadmap usually does) we would have built a few deep defences. The evidence calls for the opposite: broad, cheap, mechanical gates — a lint that sweeps 1,509 files, a documentation check, a manifest check, one syntax-level pin per contract.

So the rule that survives both cases is a conditional rather than a ratio, and it fits on one line. Where the distribution is sharp, concentrate. Where it's flat, automate cheaply and broadly. Measure first, because the two answers are opposites and picking the wrong one costs a quarter of a year.

Spending the leverage on the build

Sharding buys latency rather than cost

The unit suite splits four ways by recorded duration rather than by file count.

Splitting by count puts roughly 3,500 tests in each shard and drops the 153-second test beside 3,499 others; one shard then runs long while three sit idle. Splitting by duration packs the heavy 3.3 percent deliberately, so the four shards finish together. That's the Pareto-aware choice, and it exists only because somebody recorded the durations first.

Sharding doesn't make the suite cheaper; four shards cost the same machine minutes plus four times the fixed setup. What they buy is latency (the thing a working session actually waits on).

Tier scoping, and the part we got wrong

A draft change runs a reduced set of tiers, and the full pyramid fires when the change is marked ready.

Table 3. Three draft scopes, and the one that got retired.

Contract was the aggressive play: drop seven eighths of the test time and keep the floor. On the arithmetic it's the obvious winner — and the arithmetic was measuring the wrong thing.

Under that scope neither the unit tier nor the integration tier ran on any draft push. One change spent a whole session in that state. It carried four broken fixtures, three failing unit tests and a two-shard hang through three green draft checks, and every one of them surfaced only on the ready-flip pyramid, cycles later than a unit-tier draft run would have caught them.

The lesson generalises well past a build configuration. A Pareto cut optimises the metric you measured, and we had measured minutes. What mattered was minutes until a break becomes visible. By that measure the cheap gate was the expensive one, because a defect found three cycles late costs a rebuild, a re-review, and a cold-start diagnosis by somebody who has since loaded another problem into their head.

The unit tier is back on drafts: 75 percent of the test time and, on this evidence, most of the signal. Integration, routes and topology still wait for the ready flip, and that part of the cut survived, because nothing in the record shows it hiding defects the same way.

The floor the rule may not touch

Ring 0 is 11.4 percent of the clock and exempt from every optimisation, by written rule. It holds the build guard, the topology pins, the manifest check and the wiring checks, including the tests that check the build configuration itself.

Read that last clause twice; it's the whole reason for the exemption. A scoped run that dropped Ring 0 could green-light a change to the very file deciding what gets scoped. Some 20 percent is load-bearing. Pareto shows where the mass is rather than which parts hold the roof up.

Coverage, and how it invites abuse

Coverage turns into self-deception faster than any other number here, because it moves whether or not anything was verified. An inert assertion executes perfectly and reports full coverage. Two habits keep it honest.

The first habit is mutation verification. A test earns its place only once breaking the code breaks the test. From the change that prompted this article: neutering one six-line guard failed five of six new tests, and the sixth was the control proving the healthy case still passed. Without that check the six prove only that they run.

The second is fringe detection. Rather than chase a percentage, the build computes the blast radius of a change and asks one question: is any changed symbol reached by no test at all? That aims straight at the zero-coverage tail, which is where the risk actually concentrates.

Table 4. Three ways a coverage number gets used against the people relying on it.

The same shape at runtime

The logic runs on in the live system. An always-on service keeps a roster of health probes under a Pareto-shaped constraint. A check that fires constantly gets disabled inside a week, so the roster has to report healthy for the designed quiet state and spend its alerting budget on genuine divergence.

A microphone probe is the worked example. A closed microphone is the correct state most of the time — the system arms it only when a client connects — so "microphone closed" can't be the alarm.

Table 5. Four signals, of which only the disagreement is worth waking somebody for.

That divergence is the 20 percent worth a person's attention, and it stayed invisible for months for exactly the reason the table shows. Each signal, read alone, looked entirely normal.

Why it matters

Table 6. Six places the rule was tested, and what each measurement licensed.

Skipping the measurement has a price, and it lands on the team that skipped it.

A team that assumes 80/20 for its defects builds a few deep defences and leaves a hundred small families open. A team that cuts its build on minutes per run ships breaks that surface three cycles late, after the author has moved on to something else. And a team that never writes down its exempt fraction will one day watch a reasonable-looking optimisation remove the checks that would have caught it.

The rule tells you where the weight is. It has never told anybody which part is holding the roof up.

Glossary