← plumbing.sh

What plumb does, and how we know

Five claims about what plumb does to a repository. Each one names the evidence behind it and the boundary it stops at — because a claim without a boundary cannot be checked, and one that cannot be checked is worth nothing to you.

This is the discipline plumb applies to a repository, applied to plumb. Where a number would flatter us and we cannot derive it, it is not here.

plumb finds the checks your repository already declares.

Not checks it invents. Checks your Makefile, your lockfile and your CI already name.

Established· VERIFIED
On repositories it had never seen, plumb recovered nearly all of what their own CI and build files declare, and can cite where each came from.
Evidence
  • —93.5% of the verification commands in CI workflows — 62 repositories, held out from development
  • —59 / 59 declared Makefile targets — 5 repositories, 59 targets
  • —47 build systems recognized, 42 of which emit commands
  • —4 more recognized as a signal and correctly told to emit nothing
  • —every command carries the file and line that declares it
Where it stops
That your repository declares anything worth finding. Across 92 repositories only 22.6–31.1% carried a runnable declared check at all — and of seven of the author's own projects, five yielded nothing. plumb reports that honestly rather than inventing something. One known gap: Taskfile.yml.

plumb reads your prose and returns the obligations it states.

Not only package scripts and build configuration — CONTRIBUTING.md, AGENTS.md, CLAUDE.md. The requirements written in sentences.

Established· VERIFIED
On repositories never used in development, plumb recognizes 70.6% of the provisions an independent third party labeled as verification obligations — sentences that bind a contributor but appear in no config file anywhere.
Evidence
  • —scored against a labeled corpus published by a third party, not by us
  • —split by repository, seed frozen in a commit that precedes the change
  • —held out: 48 repositories never used in development, scored exactly once
  • —41.2% → 70.6%, against a ≥65% threshold committed in advance
Where it stops
A finished number. This measures the admission gate, which is a ceiling: classification filters further, so what you see is less. The development split scored 87.1% — that figure describes fit to the sentences used to design the fix, and is quoted nowhere for that reason.

plumb tells you when it has no answer, instead of guessing.

Five verdicts, and three of them are refusals.

Established· VERIFIED
A verdict from plumb distinguishes 'your tree is wrong' from 'I could not run this' from 'there was nothing here to run.' A tool that answers PASS or FAIL to all three is lying in two of them.
Evidence
  • —PASS · FAIL · BLOCKED · INCONCLUSIVE · NOT-RUN
  • —BLOCKED names what stopped it — a missing binary, an uninstalled dependency
  • —NOT-RUN is returned in full when a repository declares nothing runnable
  • —five of the author's own seven projects return exactly that
Where it stops
That the command plumb selected is the canonical or complete check for your repository. It never establishes that, and does not claim to — it reports what was declared, where, and what happened when it ran.

plumb does not accuse your repository of things it cannot support.

The failure mode of a tool like this is confident nonsense. It was measured for.

Established· PARTIAL
Across the repositories that executed, plumb produced one questionable alarm — and the evidence rule behind that gate is explicit: absence is only evidence when the namespace was enumerated.
Evidence
  • —a held-out corpus of 21 repositories; 15 executed a check
  • —1 of 15 raised an alarm not attributable to the tree — a count, not a rate
  • —that single alarm is itself disputed, suspected to be a shallow-clone artifact
  • —an accusation gate that requires an enumerated namespace before naming a defect
Where it stops
A percentage. One event in fifteen supports a 95% interval of 1.2%–29.8%, which is far too wide to quote as a percentage, and the numerator is not trusted. It is reported as a count on purpose.

plumb is fast enough to run on every single turn.

A gate you notice is a gate you disable.

Established· VERIFIED
Discovery finishes fast enough to sit in an agent's loop without anyone choosing to remove it. The cost of running plumb is the cost of the check it runs, not of plumb.
Evidence
  • —discovery median 93 ms across 19 repositories
  • —JabRef — 4,450 files — 182 ms cold, 39 ms warm
  • —no model call, no network round trip, nothing to wait on
Where it stops
Anything about your check. If your test suite takes nine minutes, plumb takes nine minutes. And these are discovery timings on these repositories — a monorepo an order of magnitude larger has not been measured.

One capability ships marked experimental

plumb drift compares what your documentation claims against what your build actually does. It was reviewed by hand, finding by finding, on repositories excluded from development, against a rubric frozen before the review began. It did not clear the bar — so it was stopped rather than tuned until it passed, it ships marked experimental, and no license pays for it. No maintainer was ever sent a finding, because the findings were not good enough to send. You learn that here rather than from your repository.

And the binary that does it

plumb is closed source, so everything above is only worth as much as the artifact you install. Every release is signed and notarized, and every claim on this page was re-earned on the exact binary you download rather than inherited from the one before it.

You do not have to take the offline claim on faith — run plumb with your interface down, or watch it under tcpdump. It is the rare claim that is stronger tested than sourced.

Questions about any of this: support@plumbing.sh