We publish the observations we could not take, next to the ones we could

What this page always does

These three sentences bind the next version of us. A deployment check enforces them — if a value leaves this page, the build stops.

1. Every measurement round publishes three readings together: the observations we measured, the observations we missed, and the places our own instruments were wrong that round.
2. Before a number about our performance goes into any outside material, the observation status from that same round goes here first.
3. When a reading gets worse, the sentence stays and the number changes. Earlier rounds stay on the record.

1. The observations we failed to measure this round

Of 33,780 observations attempted, 3,200 were ones we could not read.

1. The observations we failed to measure this round

Measured 90.34% We failed to measure 9.47% The surface carried no answer 0.19%

Round and record status

Of 43 rounds in the analysis record, 23 have their record closed and 20 do not. 8 of them carry no label saying whether they are test data or live measurement. That reading counts against us and points straight at the thing to fix.

There are 33 run folders on disk against 43 rounds in the analysis record. We have not explained that difference yet; the explanation lands here when it exists.

2. Where our own instruments were wrong

8 entries so far. They stay on the record.

A broken instrument makes numbers quietly flattering. So each break is written down with what changed because of it. An entry with an empty "what changed" never goes up — that would be an apology, and apologies do not accumulate.

1. What was wrong

   An accessibility checker died on our own pages, and the tally read that failure as zero violations.

   How it surfaced

   The same checker ran fine on competitors' pages. Ours failed all 123 times, yet the result table showed zero violations. The tell was that the error only ever leaned our way.

   What changed

   We injected the checker in a way our security settings allow, and all 246 checks ran. Counting a failure as a zero is now banned, and a result that flatters us gets doubted first.

2. What was wrong

   A pixel-based contrast check marked every site as failing, competitors included.

   How it surfaced

   Blurred pixels at the edge of each letter landed in the background sample. All 28 sites failed, not just ours, which meant the number could not decide anything.

   What changed

   Pass or fail now comes from a different check that reads each element. The pixel number is kept only for ranking sites against each other, with that limit written at the head of the table.

3. What was wrong

   We compared samples of different kinds — our 120 pages against 28 pages on the other side.

   How it surfaced

   Above-the-fold text came out at 79.31% against 0.96%. The gap was too wide to be real: ours was a whole site, theirs a handful of lead pages. Measured like against like, our pricing page ranked first of nineteen.

   What changed

   Every comparison now states what kind of sample the denominator is. A comparison that does not say whether it is whole-site or page-to-page is not used.

4. What was wrong

   Test-made observations sat in the pile of real ones, and the tally was quietly contaminated.

   How it surfaced

   A count came out at 1,400 where it had no business being. The shape matched the real records, so no tally caught it; plotting answer length is what finally showed a second population.

   What changed

   We did not delete the fakes, we tagged them twice. Deleting them would make the same mistake untestable. A separator now runs ahead of every tally, so nothing is counted while the two are mixed.

5. What was wrong

   We wrote a polarity test to prove that our suite-coverage checker notices when someone deletes a test runner. It had stopped testing anything. The line it targeted had moved into a helper earlier, so the injection matched nothing at all. Its own liveness guard missed that, because comparing the file before and after with `splitlines()` and a newline join drops the trailing newline — the two strings differ even when the filter removes nothing.

   How it surfaced

   The deploy gate failed on this single test, so we reproduced the injection by hand. It removed zero lines. The guard still reported a difference, and the checker returned the same three paths for the original file and for the supposedly damaged copy.

   What changed

   The guard counts removed lines now and requires exactly one. Reading keeps line endings so the trailing newline survives, and the target line sits in a constant, which gives a future refactor one place to update. We proved both directions: the web path appears for the real source and vanishes when we cut that line.

6. What was wrong

   Our forward-placement check asks whether a claim appears near the top of the page that carries it. It also asked that of two shared fragments that always render at the bottom of every page, a footer and a seller-identity block. Those can never pass. The check produced a large volume of findings that nobody could act on.

   How it surfaced

   The warning count jumped from 224 to 688, so we grouped the findings by claim. One claim accounted for 383 of them. It appeared between 80% and 99% of page depth on 381 routes, including author pages and dataset listings that have nothing to do with it.

   What changed

   Both fragments now sit in the exemption set, together with the condition that releases them: if either moves to the top of a page, it comes back under the check. Findings fell from 688 to 310. The check still flags the same claim five times, on the content pages where it belongs near the top. One number stays open — the count remains 86 above the old baseline and we have not established why, so we record it as unexplained instead of absorbing it.

7. What was wrong

   Our graph search warned that its provenance evidence was older than the graph itself. The warning was on for every input. We measure evidence per root and then merge it, and the merge carries the base timestamp through, so the freshness comparison could never advance.

   How it surfaced

   The warning fired on all three evidence sets we had on disk, whose share carrying a source measured 99.31%, 98.28% and 72.52%. A warning that is always on cannot separate a healthy evidence set from a rotten one, so we retired the comparison. Retiring it stranded the two tests that read it. One could no longer fail. The other could no longer pass.

   What changed

   The pair measures coverage now, which is the question freshness stood in for: is anything rendering without a provenance grade? When every node carries a grade, the check must produce no warning. When one node in four has none, it must produce one. The threshold is configurable, so the tests lock the sentence rather than the number.

8. What was wrong

   In our state-drift gate, an undeclared anchor and a detected drift both exit with code 1. Both polarity tests expected that code and received it for the wrong reason. They passed while measuring nothing.

   How it surfaced

   The sibling tests started failing once the gate began requiring anchor declarations. Adding the declaration made the two quiet tests exercise the path their names promise. Where two failure routes return the same code, a test cannot report which one it saw.

   What changed

   Planting a synthetic fact goes through one helper now, and that helper also declares the anchor and disables the ratchets meant for the real list. All 18 tests in the file pass. We also found 40 registered facts against a known-risk list of 27, which left the denominator smaller than the numerator. We listed the 17 that were missing, each with the reason it drifts, and the list stands at 44.

3. The metrics we track

The two sections above carry this round’s values. This table carries the list itself — every metric we have committed to tracking. Without the list, a metric can drop out and nobody notices.

Of the 13 metrics listed, 12 carry a value and 1 are not measured yet.

A metric we have not taken a reading on yet says not measured, never zero. Zero means we looked and found none; not measured means we have not looked. Collapsing the two would let this table claim more knowledge than we have.

What never appears here

No customer data, anonymised or aggregated. This record has no field where a customer run could be written.

We publish what we failed to measure. We do not publish how a customer scored. The two look like the same zero, and one of them belongs to somebody else.

The method is public. The thresholds and judging rules that run it stay in-house.

How this page changes

Every figure here is derived from one record file. None of them is typed into the page.

Earlier readings are not overwritten. When a value needs correcting, a correction entry is added rather than the old one edited.

The grid behind these numbers is the one we run on a client brand. See the audits and what they cost →

Rounds published so far: 3 · last updated 2026-08-11. · The registry of what we claim · How a reading is taken