The finding
A diffusion model's answer hides its doubt.
The journey there keeps it.
At every denoising step the sampler commits the positions it is most sure about and re-masks the rest. The final distribution is therefore sharpened by the sampler itself — it looks equally certain whether the answer is right or wrong. But the step-by-step trajectory still records which positions were contested, and for how long.
Read the two traces left to right. They separate immediately and stay separated — hallucinated answers travel through higher, longer-lived early entropy. Then read the third readout: the endpoint confidence is identical to three decimal places. That single row is the paper's argument. A score computed from the destination cannot tell these two apart; a score computed from the journey can.
Watch a hallucination form
These are real POPE examples: a real photograph, the real question the model was asked, and the real per-step entropy captured at the answer position. Press play and the model answers in front of you. The large token settles as the measured entropy falls; the meter under it is that entropy. Nothing here is simulated.
Start with the pair below. Both end at a final confidence of 0.9999. One is right and one is a hallucination, and the endpoint cannot tell you which.
Question asked—
All 60 captured examples
Orange marks the ones the model got wrong. Click any of them to run it through the viewer above.
Both at once
The single viewer makes you remember the other case. This one runs two examples on the same clock, so you can watch them diverge and then converge. Pick any pair — the defaults are the two that end at the same confidence to four decimal places.
—
—
Output confidence is not just weak here. It inverts.
The score almost every abstention method consumes is output confidence. On an autoregressive VLM it works. On LLaDA-V it sits at chance on POPE and below chance on GQA — meaning a confident answer is, if anything, slightly more likely to be wrong.
Where in the trajectory the signal lives
Drag through the steps to see individual captured traces rather than the mean. The answer locks in early; after it locks, the endpoint tells you nothing, while the steps before it already did.
From a score to a guarantee
Ranking errors is useful; a deployment needs a promise. Learn-then-Test turns the score into a threshold that certifies a risk budget. Abstaining by the trajectory answers the most reliable half of POPE at 95% accuracy. Abstaining by confidence leaves 21% error on that same half — worse than the 17% base rate you started with.
The gate, over real photographs
Risk–coverage curves are abstract. This is the same idea made literal: all sixty captured examples, ranked by the detector and cut into what a deployment would answer and what it would refuse. Orange rings mark the eleven the model actually got wrong. Drag the gate, then flip the ranking — the photographs never change, only the order they are trusted in.
Notation and abbreviations
Reproduce it
Every per-example record behind every table is served from this site, so the headline numbers can be recomputed without a GPU, without downloading a model, and without asking us for anything. This script fetches the released POPE capture and prints the two AUROCs from Table 1.
All analysis code
The experiment and analysis scripts, exactly as submitted. Pick one to read it here, or download it.
Select a script.
Released records
The per-example signal values and correctness labels behind every table and figure. These are the files the script above reads.
Tables from the paper
Figures from the paper
What it does not do
The signal diagnoses; it does not repair. We tried: re-ordering the sampler's commits, voting over steps, and preference training. None recovers the answer, and the reason is measurable — for most hallucinations the correct answer is never the internally favored option at any layer, so a decode-time rule has nothing to select. The advantage is also conditional: it appears under dynamic decoding, and every dynamic cell we have is LLaDA-initialized. On models that commit in roughly one step, plain confidence is already fine.