Created: 19-10-2026 · Last updated: 19-10-2026

title: Benchmark v2 · held-out tasks toc: false

Benchmark v2 · held-out tasks

This is the public scoreboard for three review-driven atlas tasks. It reports only held-out, double-keyed gold; a reviewer approving the machine's own proposal is not a benchmark label. Read the task cards, datasheet and licence before citing anything here.

Honest status

Task splits

Blockers and recruitment need

Baselines and negative control

The canary is synthetic machinery verification, not evidence about the data. It deliberately predicts the opposite label and must fail; a passing canary score would mean the scorer is broken.

Frozen split manifest

The sampling algorithm, seed and every held-out reviewId are in src/data/benchmark/split-manifest.json. Rebuild with npm run build-benchmark-report; score with npm run score-benchmark.

What this page does not do

• It does not print the old overallPrecision: that was the machine-verdict accept rate over self-reviewed rows, not classifier precision.
• It does not train on the held-out split (trainRows is 0 everywhere).
• It does not turn a single-key review into gold.