Operations produce evidence. Evidence produces numbers. Numbers become assertions. We determine whether those assertions deserve to be believed — and we publish what we could not check. Five surfaces, one instrument that refuses to emit a report without stating its own blind spots.
Every link is somewhere a number can stop being checkable, and the last one is where it usually happens: an assertion outlives the evidence that produced it. The chain is what the name is for.
Not a wrong number — an unfalsifiable one. A retrieval score with no audited test set, a campaign lift below its own detection floor, a saving with no written counterfactual. Each is defensible and none is evidence.
The same four steps on every surface. The fourth is enforced in code: our report object refuses to construct with an empty limits list, an unstated resolution floor, or an empty findings list.
One decision procedure, five kinds of artifact.
Each dial is a measured proportion, not an illustration. The distinction inside the badges matters: one surface is calibrated against labelled data — 59 suites over seven real corpora — and three against generators, which is a weaker kind of evidence and is labelled as such on every one of them. The fifth has no reference standard at all and its dial is empty.
What was tested, against what, and how wrong it was.
Calibrating the middle three found seven defects in them, including a check that had never once fired in the life of its surface and a rule that excluded the charges it existed to find. Every figure below came from a run that can be repeated in about a minute.
| Instrument | Reference standard | Measured error | State |
|---|---|---|---|
| Nullius documents | 59 labelled suites, 7 real corpora, human-authored queries | 38/38 caught 0/21 false positives | Field data |
| Auctus growth | 8,000 simulated experiments, known true effects | 85.6% recall 17.0% false positive | Simulated |
| Fiscus money | 7,200 merchants, 600 synthetic statements | 100% recall 99.4% precision | Simulated |
| Rima code | 38 labelled fixtures, constructed | 11/11 · 7/7 0 false alarms | 38 fixtures |
| Arbol assistant | — no reference standard established | — | Prototype |
Three of the four calibrations are against generators — simulated experiments, synthetic statements, constructed fixtures. Only the documents surface was measured on real corpora with human-authored queries. Those are not the same kind of evidence and the reports say so in as many words.
The section a competent buyer reads first.
Every report ends with its own version of this, generated from the run rather than written by hand — the validator refuses to construct one without it. These are the standing limits, true of every engagement.
The assertion each one interrogates.
Your retrieval score is measuring your wording.
Does the assertion “our retrieval is good” have proof behind it — or a test set written from the documents it is testing?
A RAG system scores 0.82 and nobody can say what that means. The test set was written by the same people who wrote the documents, so the queries reuse the documents' own vocabulary and the retriever is being rewarded for string matching. Twelve of the forty questions are answerable from one paragraph. Half the corpus has never appeared in a single test case. The score is real; it is just not a measurement of the thing anyone cares about.
Most campaign wins are smaller than the experiment could see.
Does the assertion “the campaign worked” have proof behind it — or a lift below the smallest difference the experiment could see?
A campaign reports +11.9% and gets rolled out. The test ran on 16,000 users against a 2% baseline, which means the smallest lift it could distinguish from noise was +33.4%. The result was checked six times before anyone stopped, which puts the real false-positive rate at 26.5% against the 5% printed on the dashboard. Nothing here is fraud. Every step was reasonable. The number is still not evidence.
Finding the savings is the easy half. Proving one happened is the other.
Does the assertion “we saved $40,000” have proof behind it — or a counterfactual nobody wrote down?
Somebody reports saving $40,000 last year. Against what? A saving is a difference from what would otherwise have happened, and if the counterfactual was never written down the number cannot be checked by anyone — including the person who produced it. Meanwhile three subscriptions have crept up 30% without a single approval, and two vendors are billing for the same job.
Two checks, chosen because they are high-precision and commonly missed.
Does the software’s operational behaviour support what it claims — or does it report success while swallowing the errors that say otherwise?
The second check exists because of a bug in our own repository. A calibration script wrapped its corpus loaders in a broad try/except that printed one line and continued. A schema mismatch silently dropped an entire corpus from the evidence base for a whole working session. The script said so, in a wall of output, and nobody read it. A swallowed exception is not a style problem — it is a measurement problem. The system reports success while doing less than it claims, and every number downstream inherits that without a footnote.
The layer that answers from your documents, and shows you which ones.
It makes the assertions, and shows its sources. Then it audits its own evaluation set, and withholds the score when that set cannot support one.
Every company that buys an internal assistant asks the same question six weeks in: is it actually right? The vendor shows a retrieval score. Nobody audited the test set that produced it. Arbol is the same product with that loop closed — the assistant and the instrument that checks it come from the same codebase, and the instrument was built first.
Why this is one product and not five brochures.
Every surface reads different artifacts. What they share is a decision procedure, and it is the same one every time.
Fixed scope, fixed price, and a report you can argue with.
The documents surface, running in your browser — the real engine, installed into the page. Nothing is uploaded, because there is no server to upload it to. Or send one artifact and we will look at it: you get the finding either way, including if the finding is that nothing is wrong.
One product run against your artifacts, with a written report: measures, findings by severity, the resolution floor, and the limits. The price is fixed before the work starts and quoted from the size of your company, not from how the conversation goes.
Where it gets interesting — the findings on one surface routinely explain the numbers on another.
A test set, an experiment, six months of statements, a repository. The first look costs nothing, and if we find nothing we will say that too.
rishabh@op2ra.com