SEISMO Reliability for masked-diffusion vision–language models Anonymous · under review

The finding

A diffusion model's answer hides its doubt.
The journey there keeps it.

At every denoising step the sampler commits the positions it is most sure about and re-masks the rest. The final distribution is therefore sharpened by the sampler itself — it looks equally certain whether the answer is right or wrong. But the step-by-step trajectory still records which positions were contested, and for how long.

Chart recorderanswer-position entropy · 32 denoising steps LLaDA-V · POPE · n=400 · real captured means
correct answers hallucinated answers shaded band = the gap the detector reads
How to read itLeft to right is the 32 denoising steps the model takes to write its answer. The blue line is the average entropy at the answer position across correct answers, the orange line across hallucinated ones, and the shaded band between them is the gap the detector reads. Higher means the model was still torn between options at that step. The vertical axis uses a square-root scale on purpose: the answer settles by about step 4, and on a linear axis everything after that would collapse onto the baseline and hide the gap that persists.
Entropy · step 1
—
Entropy · steps 6–32
—
Final confidence
—
Detects errors at
—

Read the two traces left to right. They separate immediately and stay separated — hallucinated answers travel through higher, longer-lived early entropy. Then read the third readout: the endpoint confidence is identical to three decimal places. That single row is the paper's argument. A score computed from the destination cannot tell these two apart; a score computed from the journey can.

Watch a hallucination form

These are real POPE examples: a real photograph, the real question the model was asked, and the real per-step entropy captured at the answer position. Press play and the model answers in front of you. The large token settles as the measured entropy falls; the meter under it is that entropy. Nothing here is simulated.

Start with the pair below. Both end at a final confidence of 0.9999. One is right and one is a hallucination, and the endpoint cannot tell you which.

Denoising viewerstep 01 of 32 LLaDA-V · POPE · real capture
The photograph the model was asked about
POPE image · case 30

Question asked—

··· settling
measured entropy at this step: —
—
How to read itThe photograph and the question are exactly what the model was given. The large token is its answer: while the measured entropy is high it churns between candidates, and it settles once the entropy falls. The bar directly underneath is that entropy, and the chart plots it across all 32 steps, with the part already decoded drawn solid and the rest left faint. The four readouts at the bottom give the true answer, what the model said, how confident it ended up, and the average entropy over the whole trajectory — which is the detector's score for this example.
Gold answer
—
Model answered
—
Final confidence
—
Mean step-entropy
—

All 60 captured examples

Orange marks the ones the model got wrong. Click any of them to run it through the viewer above.

Both at once

The single viewer makes you remember the other case. This one runs two examples on the same clock, so you can watch them diverge and then converge. Pick any pair — the defaults are the two that end at the same confidence to four decimal places.

Side-by-side viewerstep 01 of 32 same clock · real captures

—

···
——

—

···
——
How to read itTwo examples running on one clock, so you can compare them directly instead of having to remember the first while watching the second. Each side colours itself by outcome: blue if the model got it right, orange if it hallucinated. Watch the two entropy bars diverge while the two final confidences end up the same. Either dropdown will swap in any of the 60 captured examples.
—
Entropy now · left
—
Entropy now · right
—
Final confidence
—
Mean step-entropy
—

Output confidence is not just weak here. It inverts.

The score almost every abstention method consumes is output confidence. On an autoregressive VLM it works. On LLaDA-V it sits at chance on POPE and below chance on GQA — meaning a confident answer is, if anything, slightly more likely to be wrong.

Error detectionAUROC for predicting a wrong answer · 0.50 = chance
How to read itFour bars in two pairs. In each pair, orange is the model's own output confidence and blue is the trajectory score, both scored on how well they predict that an answer is wrong. The dashed line at 0.50 is chance, so a bar at that height carries no usable information. Read the pairs against each other: confidence works on the autoregressive model and fails on the diffusion one, while the trajectory works on both.

Where in the trajectory the signal lives

Drag through the steps to see individual captured traces rather than the mean. The answer locks in early; after it locks, the endpoint tells you nothing, while the steps before it already did.

Step inspectorstep 04 of 32
How to read itThe same measurement as the chart at the top of the page, but showing individual captured examples instead of averages, so you can see how much they vary around those averages. Drag the slider to move the marker along the trajectory; the three readouts underneath give the average entropy at that step for correct answers, for hallucinated ones, and the ratio between them.
Correct · entropy now
—
Hallucinated · entropy now
—
Ratio
—

From a score to a guarantee

Ranking errors is useful; a deployment needs a promise. Learn-then-Test turns the score into a threshold that certifies a risk budget. Abstaining by the trajectory answers the most reliable half of POPE at 95% accuracy. Abstaining by confidence leaves 21% error on that same half — worse than the 17% base rate you started with.

Risk–coveragelower is better · dotted line = 50% coverage
trajectory · AURC 0.072 confidence · AURC 0.141
How to read itEach point is one possible operating choice. Moving right means answering a larger share of the questions; moving up means a larger share of the answers you give are wrong. Lower is better, so the blue curve lying beneath the orange one means the trajectory leaves you with less error at every level of coverage. The dotted line marks the point where you answer half the questions and abstain on the rest.

The gate, over real photographs

Risk–coverage curves are abstract. This is the same idea made literal: all sixty captured examples, ranked by the detector and cut into what a deployment would answer and what it would refuse. Orange rings mark the eleven the model actually got wrong. Drag the gate, then flip the ranking — the photographs never change, only the order they are trusted in.

Abstention gate— 60 real POPE examples · LLaDA-V
How to read itAll 60 captured examples, ordered by how much the detector trusts them, then split into what a deployment would answer and what it would refuse. Orange rings mark the 11 the model actually got wrong, so a good ranking pushes those into the greyed-out block. Drag the slider to move the cut. Then switch the ranking from trajectory to confidence and watch the orange rings migrate up into the answered block — same photographs, same cut, worse ordering.
Answered
Abstained
—
Coverage
—
Errors still answered
—
Risk among answered
—
Base error rate
—

Notation and abbreviations

Reproduce it

Every per-example record behind every table is served from this site, so the headline numbers can be recomputed without a GPU, without downloading a model, and without asking us for anything. This script fetches the released POPE capture and prints the two AUROCs from Table 1.

quickstart.pyno GPU · no dependencies · ~2 seconds

    

      
Prints
0.7781

      
Paper, Table 1
0.778

      
Prints
0.5040

      
Paper, Table 1
0.504

    

  


  

    
All analysis code

    
The experiment and analysis scripts, exactly as submitted. Pick one to read it here, or
    download it.

  


  

    
Analysis code—

    

      

        
        

          

            —
            
              
              Download
            
          

          

          
Select a script.

        

      

    

  


  

    
Released records

    
The per-example signal values and correctness labels behind every table and figure. These
    are the files the script above reads.

  


  

    
Per-example records—

    

  


  

    
Every number on this page is recomputed from the released per-example records —
    the captured trajectories, the risk–coverage curves and the certified-coverage tables all come
    from the same JSON files shipped with the paper, so any of it can be checked line by line.

  


  
  

    
Tables from the paper

  

  


  
  

    
Figures from the paper

  

  


  
  

    
What it does not do

    
The signal diagnoses; it does not repair. We tried: re-ordering the sampler's commits,
    voting over steps, and preference training. None recovers the answer, and the reason is
    measurable — for most hallucinations the correct answer is never the internally favored option at
    any layer, so a decode-time rule has nothing to select. The advantage is also conditional: it
    appears under dynamic decoding, and every dynamic cell we have is LLaDA-initialized. On models
    that commit in roughly one step, plain confidence is already fine.