Explainable AI is the set of techniques that make a model’s decision inspectable by a person. It does not reconstruct the computation; it produces a claim about the decision that a human can check, argue with and act on. The useful question is therefore not whether a model is explainable, but what each method establishes and where it stops.
Quick Overview
What you’ll learn from this article:
- What an explanation is for, and why the answer decides which method fits
- How local surrogates, Shapley attributions, saliency maps and counterfactuals each work
- Where every one of these methods stops being true, and how that failure looks in practice
- When an inherently interpretable model is the better answer than an explained opaque one
- How the choice changes across finance, healthcare, manufacturing and customer-facing systems
Who this article is for: data scientists and ML engineers choosing a method, analysts and product owners who consume the output, and risk and compliance specialists who have to judge whether an explanation is adequate.
Reading time: 14 minutes
What an Explanation Is Actually For
An explanation has an addressee, and the addressee decides everything about its form. An engineer debugging a model wants to know which inputs the model is leaning on and whether that is defensible. An affected individual wants to know what about their case produced the outcome, and what would have to differ for it to change. An auditor wants evidence that the system behaves as documented across a population, not a compelling story about one case. These are different artefacts, and a single explanation format serving all of them is usually serving none.
The Four Principles of Explainable Artificial Intelligence published by the National Institute of Standards and Technology makes this concrete by separating properties that are easy to conflate: that a system produces an explanation at all, that the explanation is meaningful to its intended audience, that it is accurate as an account of the system’s process, and that the system operates only within the conditions it was designed for. Meaningfulness and accuracy pull in opposite directions — the simplification that makes an explanation legible is the same simplification that makes it less faithful — and every method below sits somewhere on that trade-off.
Two further distinctions organise the field. An explanation is local when it accounts for one prediction and global when it characterises the model’s behaviour overall; the two answer different questions, and a local explanation aggregated carelessly does not become a global one. And a method is model-agnostic when it treats the model as a function it can query, or model-specific when it reads the model’s internals. Agnostic methods are portable and can be applied after the fact; specific methods see more, at the cost of only working for one architecture.
Local Surrogates: Explaining One Prediction at a Time
The surrogate idea is the most intuitive entry point into the field. Rather than trying to describe a complicated decision boundary everywhere, it describes it in the immediate neighbourhood of the case you care about, where a simple approximation is usually good enough.
The mechanism is straightforward. Take the instance under examination, generate a set of perturbed variants around it, ask the model to score each one, weight them by how close they are to the original, and fit a simple, readable model — typically a sparse linear one — to that weighted sample. The coefficients of that simple model are the explanation: these features, in this direction, with this weight, produced this outcome. The approach was introduced as “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, and its appeal is that it needs nothing from the model except the ability to score inputs.
What it buys is a per-case account expressed in features a domain expert recognises, produced without touching the model’s architecture. What it costs is stability. The explanation depends on how the neighbourhood was sampled and how locality was defined, and two runs with different random draws can return noticeably different attributions for the same case. In practice this means an attribution should never be presented as a single authoritative reading without checking that it holds under repetition, and that a decision defended in front of a regulator on an unstable explanation is a decision defended badly.
The parameters that govern that stability are worth understanding rather than accepting as defaults. How wide the neighbourhood is set decides whether the surrogate describes the immediate decision boundary or smooths across a region where the model behaves differently. How many perturbed samples are drawn decides how much of the variation between runs is noise. How the perturbation is generated decides whether the sampled points resemble real cases at all — perturbing features independently on correlated data produces a neighbourhood populated by instances that could not exist, and the surrogate then explains behaviour the model was never asked to exhibit. Recording those settings alongside the explanation, and repeating the fit with a different seed before anything is shown to a decision-maker, costs very little and converts a suggestive chart into evidence.
Additive Attribution and Shapley Values
The second family answers a sharper question: how should credit for a prediction be divided among the input features in a way that is internally consistent? Borrowing from cooperative game theory, it treats each feature as a participant and asks how much the prediction changes when that feature joins the coalition, averaged across the orders in which the coalition could form.
A Unified Approach to Interpreting Model Predictions showed that several previously separate attribution methods can be understood as members of one class of additive explanations, and that the game-theoretic solution is the only one in that class satisfying a small set of desirable properties at once — the contributions sum to the difference between the prediction and a baseline, a feature the model ignores receives nothing, and features that behave identically receive the same credit. Those guarantees are the practical reason this family became the default in regulated settings: the attribution is reproducible and defensible in a way a fitted surrogate is not.
The costs are real and worth stating plainly. Exact computation is expensive as the number of features grows, so implementations rely on approximations whose own error has to be understood. Attribution is measured against a baseline, and the choice of baseline is a modelling decision that changes the answer. And correlated features distort the division of credit, because the procedure evaluates combinations that could never occur in the data — which is how a plausible attribution can be assigned to a feature that plays no causal role at all.
Seeing What the Model Looked At
Where the input is an image, attribution can be shown rather than tabulated. Gradient-based visualisation computes how sensitive the output score is to each input pixel and renders the result as a map over the image, so that the regions the classification actually rested on become visible. The technique was set out in Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, and it remains the fastest way to catch a model that has learned the wrong thing — a scanner artefact, a watermark, a background texture that happens to correlate with the label.
The diagnostic value is high and the interpretive risk is equally high. A saliency map shows where the output was sensitive to change; it does not show why that region mattered, or what the model concluded from it. Read as evidence that the model is “looking at the tumour”, it invites a confidence the map cannot support. Read as a screening tool that flags cases where the model is clearly attending to the wrong part of the image, it is very effective.
For tabular data the equivalent visual instrument is the dependence plot, which sweeps one feature across its range while holding the rest fixed and traces what happens to the prediction. It reveals shape — thresholds, plateaus, non-monotonic responses — that a single attribution number hides. Its blind spot is the same as the Shapley method’s: where features are correlated, holding the others fixed constructs combinations that do not exist, and the curve describes a population that was never observed.
Explanation by Contrast
The methods above answer “what drove this outcome”. A different family answers the question people actually ask, which is “what would have had to be different”. A counterfactual explanation names the smallest change to the input that would flip the decision: a specific amount of additional income, a shorter gap in employment history, one fewer missed payment.
Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR argues that this form has a particular advantage in a legal setting. It gives the affected person something actionable and comprehensible without disclosing the model’s internals, which sidesteps the tension between an individual’s interest in understanding a decision and an operator’s interest in protecting a system from gaming and from disclosure of trade secrets. Under Regulation (EU) 2016/679 (General Data Protection Regulation), decisions based solely on automated processing that produce legal or similarly significant effects for a person carry obligations of information and of human intervention, and a contrastive account is one of the few explanation formats that is genuinely usable by a non-specialist recipient.
The engineering caveats are specific. A counterfactual must be plausible — proposing a change to an immutable attribute is worse than useless — and it should respect what a person can actually alter. Many valid counterfactuals usually exist for the same case, so the selection rule is itself a design decision that has to be documented rather than left to whatever the optimiser returns first. And a counterfactual describes the model, not the world: it says what would change the decision, not what would change the underlying outcome the decision is trying to predict.
When the Right Answer Is an Interpretable Model
Every method above adds a layer of explanation on top of an opaque model. There is a standing argument that for consequential decisions this is the wrong architecture, and it deserves to be met rather than assumed away.
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead makes the case directly: a post-hoc explanation is by construction an approximation of the model, so it can be faithful or it can be simple, and where it is simple enough to be useful it is not guaranteed to be faithful. The paper also challenges the assumption underneath most of these projects — that opacity is the price of accuracy — arguing that on structured data with meaningful features, a constrained interpretable model frequently matches the opaque one closely enough that the difference does not justify the loss of scrutiny.
The practical consequence is a decision that belongs at the design stage rather than after deployment. Where the features are engineered and meaningful, the decision affects individuals, and the system must be auditable, a scoring model or a shallow rule set is often the cheaper answer over the system’s life, because the model is the explanation and there is no faithfulness gap to defend. Where the input is unstructured — images, audio, free text — the interpretable option does not exist in the same form, and post-hoc methods are the only instrument available. Deciding this late is what produces the familiar situation in which an explainability tool is procured to compensate for an architectural choice nobody revisited.
Where the Explanation Stops Being True
Every method here has a boundary, and the boundary is not a footnote. An explanation is a model of a model, and treating its output as ground truth is the characteristic failure of this field.
- Attribution is not causation. These methods describe what the model responds to, not what causes the outcome in the world. A feature can carry high attribution because it proxies for something the model was never meant to use.
- Correlated inputs corrupt the division of credit. Where features move together, methods that vary one at a time evaluate impossible combinations, and the resulting attribution is arithmetically valid and substantively misleading.
- Instability undermines defensibility. An explanation that changes materially between runs, or under a trivial perturbation of the input, cannot be the basis of a decision the organisation will have to defend later.
- Fluent explanations manufacture confidence. A clear attribution chart is persuasive whether or not it is faithful, and reviewers reliably discount that risk. This is the reason explanation quality needs its own validation rather than being assumed from the tool’s reputation.
Validating explanations is a discipline in its own right: checking that attributions are stable under resampling, that removing the features an explanation credits actually degrades the prediction, and that domain experts recognise the reasoning as plausible. The Artificial Intelligence Risk Management Framework (AI RMF 1.0) treats explainability as one measurable characteristic of a trustworthy system among several — alongside validity, reliability, safety and fairness — which is the right framing: an explanation is a component that can fail, and it needs testing like any other.
What Changes by Sector
The method that fits is decided by who receives the explanation and what obligation sits behind it.
In financial services, the recipient is frequently the applicant, and the demand is contrastive and individual. Attribution supports the internal decision record; counterfactual reasoning supports what is communicated outwards. Because these decisions are individual, automated and consequential, the data protection obligations above apply directly, and the explanation format has to survive contact with a complaint rather than only with a model review.
In healthcare, the recipient is a clinician who carries the responsibility for the decision, and the explanation’s job is to support their judgement rather than substitute for it. Visual attribution on imaging is diagnostic in both directions — it can show that the model attended to the relevant structure, and it can show that it did not — and the clinician’s ability to reject the suggestion is the control that makes the system safe.
In manufacturing and predictive maintenance, the recipient is an engineer, the features are sensor channels with physical meaning, and global attribution over time is often more useful than a per-case reading, because the question is which signals precede a fault rather than why this particular alert fired.
In marketing and recommendation, the stakes per decision are low and the value of explanation is mostly internal: understanding what the system has learned to optimise for, and noticing when that diverges from what the business intended. Understanding why an algorithmic recommendation was made, and being able to say so publicly, is also what keeps automated personalisation acceptable to the people subject to it — a theme developed further in AI in business and society: the future of artificial intelligence.
Across all of these, Regulation (EU) 2024/1689 (Artificial Intelligence Act) sets a floor for systems classified as high risk: they must be designed so that deployers can interpret the output and use it appropriately, and so that natural persons can oversee them effectively while in use. That obligation is about the whole system, not about installing a particular library, and it is the reason explanation choices belong in the design record rather than in a tooling decision made late.
What This Article Deliberately Leaves Out
This is an article about the mechanism of explanation — what each technique produces, what it establishes and where it fails. Two adjacent subjects have their own treatment in this knowledge base and are not duplicated here.
The first is governance: who owns an AI system, what documentation is required, how an ethical audit is run, what roles a governance function needs and what the consequences of non-compliance are. That is set out in the AI governance and ethics practical guide, which treats explainability as one principle to be implemented among several. This article assumes that decision has been taken and asks the next question: given that an explanation is required, which one, and what is it worth.
The second is the normative frame — the principles of responsible AI, how values translate into design commitments, and how the regulatory landscape is navigated as a whole. That is the subject of ethics and responsibility in AI. The regulatory references here are deliberately narrow: they appear only where a specific provision constrains the choice of explanation, and every such requirement is cited from the text of the act rather than from commentary about it.
Also outside the scope: the detailed obligations of the AI Act for high-risk systems as a compliance programme, and the competency planning that follows from them. Both are covered separately, and the boundary is worth keeping sharp — knowing which article of a regulation demands transparency does not tell you which method will deliver it, and knowing the methods does not discharge the obligation.
Frequently Asked Questions
Does adding explainability reduce a model’s accuracy?
Post-hoc methods do not change the model at all — they interrogate it from outside, so predictive performance is untouched. The trade-off appears earlier, if you choose an inherently interpretable architecture instead of an opaque one. Even there the gap is often smaller than assumed on structured data with meaningful features, and it should be measured on the actual problem rather than accepted as a general rule.
Which method should you start with?
Start from the recipient. If the answer has to be given to an affected person, a contrastive explanation is usually the right form. If the audience is an engineer diagnosing behaviour, attribution and dependence plots come first. If the audience is an auditor, the global picture and the evidence that explanations are stable matter more than any single case. Choosing the tool before the recipient is how organisations end up with dashboards nobody uses.
Can an explanation be wrong?
Yes, and this is the most important thing to understand about the field. Every post-hoc explanation is an approximation, and it can be unstable, distorted by correlated inputs, or faithful to the model while misleading about the world. Explanation quality has to be validated — for stability, for fidelity, and against domain expectations — and never inferred from the tool’s reputation.
Is explainability legally required?
It depends on the system and the jurisdiction. In the European Union, high-risk systems carry transparency and human-oversight obligations under the AI Act, and automated decisions with legal or similarly significant effects on individuals carry information and intervention obligations under data protection law. Neither mandates a named technique; both require an outcome that a specific explanation choice has to deliver, which is why the choice belongs in the design record.