Auctus — growth auditing by Op2ra

The situation

Most campaign wins are smaller than the experiment could see.

A campaign reports +11.9% and gets rolled out. The test ran on 16,000 users against a 2% baseline, which means the smallest lift it could distinguish from noise was +33.4%. The result was checked six times before anyone stopped, which puts the real false-positive rate at 26.5% against the 5% printed on the dashboard. Nothing here is fraud. Every step was reasonable. The number is still not evidence.

$ nullius growth --n-control 8000 --x-control 160 \ --n-treatment 8000 --x-treatment 179 --looks 6 growth audit ============================================================== observed relative lift +11.9% p-value as reported 0.2969 smallest detectable lift +33.4% (at n=16,000, baseline 2.00%, 80% power) true false-positive rate 26.5% (after 6 looks, against the 5% printed) -------------------------------------------------------------- FINDINGS -------------------------------------------------------------- [CRITICAL] UNDERPOWERED: a +10% effect needs 161,364 obs [CRITICAL] BELOW_RESOLUTION: +11.9% < the +33.4% floor [CRITICAL] PEEKING: 6 looks, alpha is 26.5% not 5%

What it checks

Four questions, asked in the same order every time.

Minimum detectable effect — the smallest lift the test could have found at its actual sample size. A result below it is not a small win — it is unmeasured
Required sample — how many observations the effect you hoped for would have needed. Usually an order of magnitude more than was run
Peeking correction — the true false-positive rate after k looks, 1−(1−α)^k. Six looks at a 5% test is a 26.5% test
Resolution floor — stated before the findings, so a reader can tell which numbers are bigger than the experiment's own noise

What it produced on a reported win

+11.9%

lift as reported

+33.4%

smallest it could detect

26.5%

true false-positive rate

161,364

sample the claim needed

The test ran on 16,000. Three critical findings on a result that had already shipped. The mathematics is standard two-proportion power analysis — the contribution is that it is applied to the number before the number is believed, not after it fails to replicate.

What this audit cannot see

The section a competent buyer reads first.

Every report this product emits ends with its own version of this list, generated from the run rather than written by hand. A report cannot be constructed without one — the validator refuses.

attribution versus incrementality. No arithmetic on four numbers can detect whether your control arm was still seeing the channel — that is a property of assignment, not of the data. It is a declared limit and never a finding
whether the metric is the right metric. A perfectly powered test of the wrong outcome is still a perfectly powered test of the wrong outcome
the calibration is against SIMULATION, not field data. 8,000 experiments with known true effects, under a binomial model that assumes independent users, no novelty decay and a stable baseline. Real conversion data breaks all three, and Nullius — measured on seven real corpora — is not the same kind of evidence
recall is 85.6% and the false positive rate is 17.0%. Recall is quoted against the spurious wins it can see: a false positive from a single honest look is a 1-in-20 fluke that no arithmetic on four numbers separates from a real win, and 40% of the simulated fakes were exactly that. Both numbers are published because the second one is the unflattering one

Engagements

Fixed scope, fixed price, and a report you can argue with.

Free

Browser audit

The documents surface, running in your browser — the real engine, installed into the page. Nothing is uploaded, because there is no server to upload it to. Or send one artifact and we will look at it: you get the finding either way, including if the finding is that nothing is wrong.

Fixed

One surface, two to three weeks

One product run against your artifacts, with a written report: measures, findings by severity, the resolution floor, and the limits. The price is fixed before the work starts and quoted from the size of your company, not from how the conversation goes.

Quote

Multiple surfaces

Where it gets interesting — the findings on one surface routinely explain the numbers on another.

The rest of the suite

Five surfaces, one decision procedure. Each deploys separately, so one product's failure cannot take another down.

Product	Surface	In one line
Nullius	documents	Your retrieval score is measuring your wording.
Fiscus	money	Finding the savings is the easy half. Proving one happened is the other.
Rima	code	Two checks, chosen because they are high-precision and commonly missed.
Arbol	assistant	The layer that answers from your documents, and shows you which ones.