Security Scanning in Practice: The Red Check Was a Full Disk

Contents11 sections

The check named Security

A pull request came back with a red check called Security. Nothing insecure had been found. Nothing had been scanned at all.

The job ran two scanners, and both scan steps carried continue-on-error, so a genuine finding could never fail anything. The steps around them carried no such protection. A runner filled its disk during install; the install step failed; the job went red under the name of the thing it was meant to measure.

That is worse than running no scanners at all, and the reason has to do with reading rather than with security. A red check named Security gets read as a finding. Somebody opens it expecting a vulnerability, finds a disk quota, and next time opens it a little slower. Three rounds of that and the check has trained the team to ignore it, which is the exact outcome it existed to prevent. The tool worked fine. The wiring taught people not to look.

That job file was written here, and the tool choice was fine. The error was letting one signal carry two meanings, which is a mistake in wiring rather than in judgment about scanners, and it took a full disk on a build runner to make it visible.

Everything below assumes the decision to adopt scanning has been made and the only remaining question is where to put it. Where that decision is still open, the companion piece on what SAST and SCA actually read, and on the list of vulnerabilities that neither of them will ever find, is the one to start with.

Three loops, three budgets

The commonest failure is treating this as one deployment decision. There are three of them, they have different budgets, and a control designed for one fails in the others.

Table 1. Cost rises left to right, and so does authority. The commonest mistake is running only the middle column.

Each loop is cheaper than the one to its right and catches less. None replaces the others.

What the inner loop can afford

One requirement really binds here, and it is speed. A check that adds two seconds to a commit is invisible. One that adds ninety comes out inside a week, and no amount of policy survives that removal, for the simple reason that the person doing the removing is the same person the policy was written to protect.

Everything else follows from the budget. Scope to the changed files rather than the tree. Run SCA only when the lockfile moved. Cache the advisory database so the network is never in the path.

Put the fastest rules in the editor, where the feedback lands while the code is still in working memory and the fix costs an order of magnitude less than it will an hour later. Put secret scanning in a pre-commit hook especially: a credential caught before the commit costs nothing, and one caught after is a rotation and possibly a history rewrite.

Then fail on new findings only. On any existing codebase the first full run returns dozens of pre-existing findings, and blocking on those means an engineer's first experience of the tool is being stopped for something they did not do. The hook comes off that afternoon. Baseline the existing debt on day one; fail only on additions.

What continuous integration has to decide

CI is where the answer becomes authoritative, and four design decisions matter more there than the choice of tool does.

Pin the tool version. An unpinned scanner is a gate that changes definition without a commit: a new release adds a rule, and yesterday's green branch turns red today for a change nobody made. Upgrade it as a reviewed diff that re-baselines deliberately. Version drift is not a security event.

Ratchet against a committed baseline, for the same reason as the inner loop and with more force. A gate that is red on arrival gets bypassed, and a bypassed gate is worse than an absent one because it still reports. Everything downstream of it then inherits a signal that somebody has quietly agreed to ignore, which is how a control becomes decoration without anybody deciding that it should.

Distinguish a finding from a failure to look. A scanner that crashed, timed out or never installed has produced no verdict, and no verdict is a different thing from a clean one. Give them different exit codes and different messages, because the two send a reader to two different places: one to the code, and one to the machine that was supposed to run the check. Collapsing them is how the opening happened.

Never make the whole job non-blocking. Report-only attached to the scan steps while everything around them can still fail is the worst arrangement available, and it is the one from the opening. The finding cannot fail the build. The infrastructure can. So the check goes silent exactly when it matters and loud when it does not.

What release records

Release asks a different question: what is in this artifact, and what do we know about it today. Today carries the weight in that sentence. It keeps changing after you ship, and the artifact does not.

Generate an SBOM at build and store it with the artifact, in CycloneDX or SPDX. Reconstructing a bill of materials months later from a tag and a lockfile is guesswork. Capturing it at build is bookkeeping.

Re-scan that SBOM continuously. This is the step most often missed and the one that pays, because a build-time SCA result is a point-in-time answer while advisories keep being published against versions that shipped perfectly clean, which means the question worth asking is never the one you asked at build. Scanning the stored SBOM on a schedule turns "are we affected by this morning's disclosure" from an investigation into a query.

Keep an exception register with expiry dates. Every real programme accepts some findings; an acceptance with no expiry is one nobody revisits, while a dated one comes back as a decision. Record reachability as data rather than as folklore, which is what VEX is for: it lets "present and not exploitable in this product" survive the analyst who worked it out.

Sign the build and record provenance, so the artifact you scanned is demonstrably the artifact you shipped. An SBOM for a different build is a document. It is not a control.

The queue is the real product

Installation is a day's work. What decides whether the programme is alive in a year is the queue: whether the thing it produces every morning is one a human being can actually finish before lunch, or a list that grows faster than anybody reads it.

An unfiltered SCA report on a large tree runs to hundreds of findings, a handful of which matter, with no signal separating them. What happens next is predictable and it is nobody's fault: a queue that cannot be finished teaches everybody who opens it to scroll past. Heroism does not enter into it.

Filter to what ships first, because it costs nothing. A build-only or test-only dependency never reaches the artifact, and scoping the queue to what actually ships removes more rows than any other single move available. Then filter to what is being exploited in the world, which is nearly free. Reachability comes last. It is the only expensive filter, and skipping the cheap ones to reach it is the more common mistake.

Severity is not priority

The default sort is CVSS, and CVSS answers how bad this would be if exploited. That is a question about consequence, asked in the abstract, with no knowledge of your deployment. Using it as a priority score produces a queue where a theoretical critical in a test-only dependency outranks a medium being exploited this week.

Table 2. Four signals, only one of which is a priority score.

A workable default: filter to what ships, rank by KEV first and EPSS second, then treat CVSS as a tiebreak. Reachability analysis earns its cost at the top of that list and almost never earns it in the tail.

Suppression needs an expiry date

Every real programme suppresses findings, and a tool with no supported way to do it gets suppressed wholesale instead, by removing the step. So the design question is never whether to allow exceptions. It is whether an exception is reviewable: who accepted it, on what reasoning, and when it comes back.

Suppress in a file rather than in a comment. An inline annotation is invisible in aggregate, so nobody can list them, count them or review them; a baseline file is a diff, and a diff gets reviewed.

Fingerprint on the code rather than on the line number. A suppression keyed to a line number silently detaches the first time somebody adds an import above it, and then either re-fires or, worse, quietly keeps suppressing a different finding that moved into its place.

Require an expiry. An acceptance with no date is permanent by default, and permanence is exactly what nobody meant to decide.

Then count them. The number of active suppressions is the best leading indicator available of a gate about to be abandoned, and it is better than any measure of the findings themselves, because a team routing around a control leaves its fingerprints on the exception list long before it leaves them anywhere else. Rising fast means the gate is mis-tuned rather than that the code got worse.

Three outcomes, and the middle one

A gate has three possible answers, and implementations collapse the middle one.

The rebuild that followed the incident in the opening involved no better scanner. It came down to four decisions.

Fail on new findings only, measured first. The tree carries forty-three known findings, so blocking on all of them would have reddened every pull request and been bypassed within a day.

Exit with a different code when a scanner produces no verdict, because unknown is not clean. Fingerprint findings on the code rather than the line number, so that an edit elsewhere in the file does not arrive as a new finding, which is the difference between a gate that tracks the code and one that tracks the diff.

Make the install step name its own failure. A full disk can never again be read as a vulnerability.

A fifth thing turned up late. Adding that gate created a check that lived only in CI, which the local runner could not reproduce, so a contributor had no way to see a failure before pushing it. Being blocked by something you cannot run yourself is how tools acquire their reputations. Every blocking check should be runnable on a laptop with one command.

Four numbers worth watching

Most security dashboards report the count of open findings, which is the least informative number on offer. It moves when the scanner is upgraded. It moves when a dependency is added. It moves when nobody has done anything at all.

Table 3. Four measures that separate a working programme from a decorative one.

Run the scans in three places with three different budgets. Ratchet all of them against a recorded baseline, and never let one report an unknown as a pass. Spend the triage effort on filtering to what ships and what is actually exploited before spending any of it on reachability. Make suppression a reviewable file with an expiry date, and watch the size of that file, because the gate that gets switched off is almost never the one that found too little.

Glossary