Calibration record

1Instrument

Instrument

loading

Predicts with

loading

Readings sealed

loading

Not yet resolved

loading

2Procedure

1. A pull request is read only while no human other than its author has responded. Bots don’t count.

   the exact rule is grailb_policy.py; section 8 shows how to hold any reading to it

2. The reading is sealed: hashed into an append-only chain, one reading per pull request. A second run on the same pull request is rejected, not kept.

3. The chain tip is anchored to Bitcoin through OpenTimestamps, usually within an hour or a few.

4. Once the pull request merges or closes, the reading is scored against the outcome by a model judge, alongside two baselines that saw the same pull request.

   calls on pull requests still open stay sealed until they resolve, so publishing them can’t steer the outcome

3Traceability

Every reading is traceable to a clock nobody here controls. These checks ran on our server and in your browser; section 8 runs them on your machine.

Hash chain

loading

Bitcoin anchors

loading

4Results

Resolved pull requests only. Intervals are 95%: Wilson for rates, normal approximation for score differences. Recall has to beat the arm’s flag rate, which is what flagging at random would catch; precision has to beat the base rate.

5Arms compared

Each arm makes one call per resolved pull request: close or merge. Arms are scored over slightly different sets of pull requests; the like-for-like comparisons are in section 4.

6Limitations

• Readings 1 to 1,051 were sealed before Bitcoin anchoring began. they can’t have changed after 2026-07-14 17:56 UTC, but for most of them the claim that they came before the first human response rests on our git history and our word
• A fast pull request can’t be proved, however honest the seal. anchoring takes from under an hour to a few, and Bitcoin’s block timestamps carry about two hours of tolerance, so a pull request that got a human response within roughly three hours of its seal can’t be shown to have been read first
• The analyses themselves are not public. they name individual reviewers; each content_hash commits to one, so it can’t be swapped, and anyone with a serious reason can check them under NDA, choosing the readings themselves
• Most Baseline v2 readings were back-runs.
• Baseline readings aren’t sealed. we keep them for our own record; they aren’t hash-chained or anchored, so we can’t prove they weren’t altered or rerun, and every comparison against a baseline rests on our word in a way GRAILB’s own readings don’t
• For many earlier readings, the model wasn’t recorded. a bug in our logging left it out; those can’t be recovered now, and sections 1 and 5 show how many per arm; from late September 2026 on, GRAILB’s model is recorded inside its sealed trace
• The scores are only as good as the judge. a model scores every arm after the outcome, against a fixed rubric; where two arms were scored by different judges, section 4 says how many
• The record shows nothing was removed, not that nothing was skipped. the chain proves no reading left the log; it can’t show which pull requests the instrument never read
• None of this shows the readings are any good. that is what section 4 is for

7Readings

Every sealed reading, newest first. Open a row for its hashes, the models behind each arm, and the command that checks it against GitHub and Bitcoin.

8Independent check

Nothing on this record has to be taken on trust. These run on your machine, against the public log, GitHub and Bitcoin.

1. Clone the log, then replay the chain and every Bitcoin anchor. No Bitcoin node needed.

   git clone https://github.com/grailb/calibration-log
   cd calibration-log
   pip install opentimestamps-client
   python verify.py

   Copy

2. Hold any reading to the sealing rule, using GitHub’s own timestamps.

   python verify.py --pr kubernetes/kubernetes#143825

   Copy

3. Or let chance pick. Choose your own seed, so we can’t choose the sample.

   python verify.py --sample 25 --seed your-own-words

   Copy

   set GITHUB_TOKEN to any personal access token to check more than about fifteen pull requests an hour

4. Compare against this record. Every reading in section 7 carries its chain_hash; it must match the same chain_seq in your clone. If it doesn’t, this page is wrong, not the log.