Nullius — documents auditing by Op2ra

The situation

Your retrieval score is measuring your wording.

A RAG system scores 0.82 and nobody can say what that means. The test set was written by the same people who wrote the documents, so the queries reuse the documents' own vocabulary and the retriever is being rewarded for string matching. Twelve of the forty questions are answerable from one paragraph. Half the corpus has never appeared in a single test case. The score is real; it is just not a measurement of the thing anyone cares about.

$ nullius audit ./corpus queries.jsonl documents audit ============================================================== vocabulary overlap (median) 0.834 (lift 4.1x over a size-matched random draw) longest verbatim span 11 tokens corpus coverage 38% effective queries 19 of 40 submitted resolution 0.061 — differences below this are noise -------------------------------------------------------------- FINDINGS -------------------------------------------------------------- [CRITICAL] LEXICAL_CONTAMINATION: queries reuse their gold document's wording 4.1x above chance [WARNING ] BELOW_RESOLUTION: the +2.1% you reported last sprint is smaller than this test set can resolve

What it checks

Four questions, asked in the same order every time.

Chance-corrected vocabulary overlap — how much of a query's wording is lifted from its own gold document, measured as lift over a size-matched random draw of the same number of documents — not the raw overlap, which rises with corpus size
Verbatim span containment — the longest contiguous run of query tokens that appears intact in the gold document. Containment rather than Jaccard: an overlapping chunk IS contained in its source, and the union denominator hides it
Coverage and effective-n — how much of the corpus is never tested, and how many of your queries are actually independent rather than restatements of each other
Resolvable delta — the smallest score difference this test set can separate from its own noise. Most reported improvements are below it
False misses — gold labels that are wrong because a near-duplicate chunk was the correct answer and got marked incorrect

Calibrated against labelled data

public and private corpora

labelled suites

38/38

contaminated suites caught

0/21

false positives on clean suites

Corpora: SQuAD 2.0, Cranfield, CISI, the Rust book, PEPs, and two private document sets. The threshold was set once and never retuned per corpus — the last two corpora were added after it was fixed, and the first of them produced a false positive that forced a bug fix rather than a threshold change.

What this audit cannot see

The section a competent buyer reads first.

Every report this product emits ends with its own version of this list, generated from the run rather than written by hand. A report cannot be constructed without one — the validator refuses.

whether your gold labels are correct in the first place — the audit measures a proxy for it and reports the proxy, not a verdict
whether the retriever is any good. This audits the ruler, not the thing being measured
anything about production traffic. A test set can be clean and still bear no resemblance to what users type

Engagements

Fixed scope, fixed price, and a report you can argue with.

Free

Browser audit

The documents surface, running in your browser — the real engine, installed into the page. Nothing is uploaded, because there is no server to upload it to. Or send one artifact and we will look at it: you get the finding either way, including if the finding is that nothing is wrong.

Fixed

One surface, two to three weeks

One product run against your artifacts, with a written report: measures, findings by severity, the resolution floor, and the limits. The price is fixed before the work starts and quoted from the size of your company, not from how the conversation goes.

Quote

Multiple surfaces

Where it gets interesting — the findings on one surface routinely explain the numbers on another.

The rest of the suite

Five surfaces, one decision procedure. Each deploys separately, so one product's failure cannot take another down.

Product	Surface	In one line
Auctus	growth	Most campaign wins are smaller than the experiment could see.
Fiscus	money	Finding the savings is the easy half. Proving one happened is the other.
Rima	code	Two checks, chosen because they are high-precision and commonly missed.
Arbol	assistant	The layer that answers from your documents, and shows you which ones.