The question
A model that outputs the right label has not yet told you anything about why that label obtained. Accuracy answers a different question from explanation. One is about performance on a distribution. The other is about the legitimacy of a story we tell ourselves, a regulator, or a person affected by the decision.
An explanation is not a visualization. It is an epistemic act: something is asserted, and something is refused.
The trouble with much of explainable AI is not that the tools are useless. It is that they quietly change the question. A heatmap can feel like understanding while committing to almost nothing. A counterfactual can sound actionable while describing only the model's geometry. A causal narrative can sound deep while resting on an unexamined graph.
A shared example
A toy underwriting model scores an applicant and outputs Deny. Confidence is high. Four explanation styles will now speak about the same decision.
Keep the decision fixed. What changes, as we climb the ladder, is not the output — it is the strength and kind of claim we are allowed to make about that output.
The ladder
Step through four explanation styles. For each rung, read the claim, look at the toy visual, and notice the “does not entail” list. The interesting part is usually what the explanation refuses to say.
Interactive · Explanation ladder
Same decision, four claims
What the model attended to
ClaimsThese input dimensions most strongly influenced the local score for this applicant.
Does not entail
- That changing a highlighted feature would flip the decision
- That the feature caused the outcome in the world
- That the explanation remains valid off this neighborhood
Use ← → keys
What each rung claims
Saliency
Attribution methods answer a local sensitivity question: which coordinates mattered for this score. They are often good at directing attention and bad at supporting intervention talk. A bright feature is not yet a lever.
Counterfactual
Counterfactuals answer a model-edit question: what nearby input would flip the label. They are closer to action than saliency, but still silent about feasibility in the world and about whether the edit path is stable for other people.
Causal
Causal explanations answer an intervention question under assumptions. The graph is doing real work. If the graph is wrong, the story can be fluent and false. Conditioning is not intervening; that distinction is the whole point.
Formal guarantee
Certificates answer a robustness question inside a stated region. They are narrower and, for that reason, often more honest. Stability is not fairness, wisdom, or understanding — but it is a claim that can be checked.
Faithfulness versus plausibility
Humans like explanations that sound like the ones we give each other. Models do not owe us that genre. A plausible story can be unfaithful to the computation; a faithful description can be psychologically unsatisfying. Confusing the two is how interpretability becomes theater.
One useful discipline: for any explanation artifact, ask what would count as a counterexample. If nothing could falsify it, it was never a claim — only a mood.
Closing
Explanation, on this view, is typed. Saliency, counterfactuals, causal stories, and certificates occupy different positions in a space of commitments. Climbing the ladder is not automatically progress; it is a change of speech act.
The practical moral is modest. Before asking a model to explain itself, decide what kind of answer would be allowed to change your mind — and what kind would only decorate a decision you had already made.