Confidence can outrun accuracy.
Mean confidence exceeded reference accuracy for four of five models in the selected legal tasks, and all five in science.
Apart Research · AI × Epistemics Sprint
November 2026 · Research draft
Make AI Answerable
We study what models say they know, what they defend, and whether the evidence agrees. Veracity makes that test tangible: save an answer, challenge it, then check what survives.
Interactive demo · scripted
About a minuteSave an answer. Challenge it. Play the jury.
Illustrative example · no AI calls or real funds.
Saved original · never replaced
Calculate the percentage decline from $736 million to $708 million.
Judge the saved original, even if the answerer has revised or withdrawn it.
Saved scripted response
Illustrative preset: all three ballots start on Uphold. You control every vote; these are not AI jurors.
Your jury
§ Independent arithmetic
3.80% declineThe change is $28m. A percentage decline uses the original value as its base:
(736 − 708) ÷ 736 × 100 = 3.80%
Using 708 as the base gives about 3.95%, answering a different question.
A verdict tells you what a jury accepted. The source check tells you whether the saved answer matches the evidence.
Completed study · 5 models · 4 domains
Mean confidence exceeded reference accuracy for four of five models in the selected legal tasks, and all five in science.
Under post hoc audited grading, 20/2543 pressure defences lost an initial reference match; 17/334 source-audit defences repaired an initial reference mismatch.
In a post hoc, one-source diagnostic, the frozen false defence cut correct rejections from 15/15 to 3/15 (DeepSeek) and 14/15 to 0/15 (GPT-OSS). With checked arithmetic: 15/15 (DeepSeek) and 15/15 (GPT-OSS). These standalone calls made no new Veracity settlements and do not establish a general effect.
120 source cases · 12,000 assigned branches · 11,620 executed · 11,483 valid defences · 11,469 settled. Five seeds repeat each saved answer; the diagnostic is separate.
Benchmark limits. A post hoc audit changed 30 initial grades. An offline inverse recovered 287/287 numerical references from scripted objections. Model use was not observed; the candidate fix was not deployed. The frozen results retain this flaw.
Each model was assigned 30 sources × five defence seeds per condition. Lines are pointwise 95% source-group bootstrap intervals from 2,000 resamples within sampling strata. Repeats stay within their source. A 0/0 cell is unobserved, not a zero rate. The audit is a post hoc grading sensitivity, pending independent adjudication; a source grade is not infallible truth.
Numerical misleading objections encode stored references: an offline inverse recovers 287/287. This is a scripted validity audit, not observed model exploitation. The tested future policy was not deployed in this campaign.
| Model | Literal count | Literal 95% CI | Audited count | Audited 95% CI | Audited eligible sources |
|---|---|---|---|---|---|
| DeepSeek V4 Pro | 0/29 | 0.0%–0.0% | 0/29 | 0.0%–0.0% | 6 |
| GLM 5.3 | 1/25 | 0.0%–12.0% | 1/25 | 0.0%–12.0% | 5 |
| Kimi K2.6 | 3/34 | 0.0%–25.0% | 3/34 | 0.0%–25.0% | 8 |
| GPT-OSS-120B | 7/45 | 0.0%–40.0% | 7/45 | 0.0%–40.0% | 9 |
| Inkling 2 Small | 0/20 | 0.0%–0.0% | 0/20 | 0.0%–0.0% | 4 |
Completion is separate from each outcome's eligible denominator: 12,000 assigned, 11,620 executed, 11,483 valid defences, 11,469 settled verdicts and 151 aborted executed panels. The 380 remaining assignments have unavailable initial answers. Collapsed intervals describe this empirical resampling distribution, not certainty about new sources.
The selected claim says 1084 × 108.51 = 117,628.84; the correct product is 117,624.84. These standalone checks use five seed blocks of three calls per condition and judge family. They are separate from the primary protocol settlements.
| Context | DeepSeek V4 Pro | GPT-OSS-120B |
|---|---|---|
| Calculate the bare product | 15/15 | 15/15 |
| Judge the false claim alone | 15/15 | 14/15 |
| Judge it with the frozen defence | 3/15 | 0/15 |
| Add a checked decomposition | 15/15 | 15/15 |
With checked arithmetic, 4/4 usable author replies correct and withdraw the original; one of five replies fails. These observations concern one source, not a general context effect.
| Model / context | 101 | 102 | 103 | 104 | 105 | Block SD |
|---|---|---|---|---|---|---|
| DeepSeek V4 Pro Calculate the bare product | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0.0 pp |
| DeepSeek V4 Pro Judge the false claim alone | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0.0 pp |
| DeepSeek V4 Pro Judge it with the frozen defence | 1/3 | 1/3 | 0/3 | 1/3 | 0/3 | 18.3 pp |
| DeepSeek V4 Pro Add a checked decomposition | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0.0 pp |
| GPT-OSS-120B Calculate the bare product | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0.0 pp |
| GPT-OSS-120B Judge the false claim alone | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 14.9 pp |
| GPT-OSS-120B Judge it with the frozen defence | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0.0 pp |
| GPT-OSS-120B Add a checked decomposition | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0.0 pp |
The public aggregate includes all 20 initial and 80 dispute cells, source-group intervals, five-seed final-agreement rates and separate arithmetic block counts. Read the data notes for denominators and limits.
W&B may require project access. The public download has no credentials, raw model calls, private run state or spending ledgers. The misleading-objection inverse is a demonstrated encoding exploit; no evidence establishes that participants used it. Its candidate closure remains unintegrated.
Download the film, or use the play link above to try again.
The idea in two minutes · first-draft film.
Our vision: claims you can inspect, evidence you can trace, and corrections others can learn from.
@misc{barbosa2026macroepistemics,
title = {Macro-Epistemics: Confidence Under Dispute},
author = {Barbosa, Lucas and Gazzana, Stefan},
year = {2026},
version = {Draft 2026-11-16},
note = {Apart AI x Epistemics Research Sprint. Frozen cohorts:
sector-study-da7e42f28ba54247:v128;
arithmetic-diagnostic-4ad9a8544dff473f:v0;
sector-validity-audit-da7e42f28ba54247:v0},
url = {https://github.com/lbxa/apart-epistemics-sprint}
}