Explainable AI in finance means a lender or insurer can say why a model reached a specific decision, in terms both a customer and a regulator can check. In practice it comes down to one question: when the model declines a loan, can you name the principal reasons it declined, and are those reasons true? In the US, consumer credit law already requires that answer. In the EU, the AI Act adds its own version for high-risk systems such as credit scoring.
This guide covers what regulators expect, which explanation techniques hold up under review, and where AI agents and language models make the job harder.
What explainable AI in finance actually means
A model is explainable when a person can trace an output back to the inputs that drove it. For a credit model, that means a ranked list of reasons attached to each individual decision: debt-to-income ratio too high, or a recent missed payment.
There are two routes to that list. An interpretable model, such as a logistic regression or a scorecard with points per attribute, carries its explanation inside it. You can read the weights. A complex model, such as a gradient-boosted tree ensemble or a neural network, has to be explained after the fact with a separate method that estimates how much each input mattered.
Banks have used both for years. The scorecard dominated retail credit because a credit officer could defend it line by line. Machine learning arrived with better accuracy on thin-file applicants, and with a harder conversation at the compliance desk. The model got smarter and the paperwork got longer, which is the usual trade in regulated industries.
The distinction matters because the two routes fail differently. An interpretable model can be wrong, but it is honestly wrong: the explanation is the model. A post-hoc explanation is a second model describing the first, and it can drift from what the first model really did.
Explanations also come at two scales. A global explanation describes the model as a whole, for instance that payment history carries more weight than income across the portfolio. Model risk teams and auditors want that view.
A local explanation covers one decision for one person, and that is the view a customer is legally owed. A model can pass the first test and still fail the second, so plan for both from the start.
Why AI compliance in finance starts with the adverse action notice
Regulators care about explanations because customers have a legal right to them. The clearest case is the adverse action notice, the letter a US lender sends when it denies credit or offers worse terms.
Under the Equal Credit Opportunity Act and Regulation B, that notice must give specific principal reasons. In Circular 2022-03, published in May 2022, the Consumer Financial Protection Bureau answered the obvious follow-up. A creditor using a complex algorithm still owes the applicant accurate, specific reasons, and the bureau stated that a model too opaque to explain is no defence. The circular also notes that post-hoc methods approximate the model, so a lender has to be able to validate those approximations.
Europe reaches the same place through the AI Act. The European Commission's summary lists credit scoring among high-risk uses and sets out duties that include activity logging and human oversight among others. Article 86 adds a right for affected people to obtain clear and meaningful explanations of the AI system's role in a decision. After the AI Omnibus amendment, the high-risk rules apply from 2 December 2027.
Treat both regimes as one design requirement. Every automated decision that affects a customer needs a stored, reviewable reason, produced at decision time. We covered the ownership side of this in why AI transformation is a problem of governance: someone has to own the model before anyone can sign off on its explanations.
SHAP and counterfactuals: explainable AI techniques that hold up
Most production teams settle on two families of technique. Feature attribution says which inputs pushed this decision up or down. Counterfactuals say what would have had to change for a different outcome.
Feature attribution with SHAP
SHAP, short for SHapley Additive exPlanations, is the default attribution method in credit modelling. Scott Lundberg and Su-In Lee introduced it in a 2017 NeurIPS paper that showed six earlier explanation methods belong to one class of additive feature attributions. It borrows Shapley values from cooperative game theory: each feature is a player, and the prediction is the payout split between them.
For a declined applicant, SHAP returns a signed contribution per feature. Sort the negative ones, map each to an approved reason code, and the top entries become the reasons on the notice. That mapping table deserves as much review as the model, because the bureau expects reasons that describe the factors actually scored.
Two cautions from teams that have shipped it. SHAP values depend on the background dataset you compare against, so fix that dataset and version it with the model. Correlated features also split credit between them, which can bury a real driver in the middle of the list. A model with forty income variables will report forty modest income reasons and leave the applicant to work out that the underlying issue is income.
Counterfactual explanations
A counterfactual answers the customer's real question: what do I change? "Approval would have followed with a debt-to-income ratio below 36%" is actionable in a way a ranked list is not. The constraint is plausibility. A counterfactual that suggests the applicant become ten years older is mathematically valid and useless as advice.
Neither technique fixes a model you cannot validate. If the explanation and the model disagree on held-out cases, the explanation is the part that has to change. Our post on convolutional neural network stock market prediction shows the other end of the scale: a model so opaque that explaining a single trade signal is an open research problem.
AI agents in finance and the explanation gap
An AI agent, meaning a language model that plans steps and calls tools on its own, breaks the neat picture above. There is no single prediction to attribute. There is a chain of tool calls and generated text that ends in an action.
Feature attribution does not transfer to that sequence. The workable substitute is a trace: a log of every tool call the agent made with its arguments, plus the inputs and final output, stored against the case ID. When an auditor asks why the agent flagged a transaction, the trace is the explanation. It is a long one, roughly the length of the agent's diary, but it is reviewable.
Four design rules keep agent explanations defensible:
- Keep the decision with a scoring model. Let the agent gather documents and draft the summary, and let an interpretable or SHAP-explained model make the credit call.
- Require citations. Any claim the agent puts in front of a customer or an analyst should link to the document it came from.
- Log the prompt and model version with every case, so a reviewer can reproduce the run.
- Route low-confidence or adverse outcomes to a human before they reach the customer.
The same applies to natural language processing in finance more broadly. A model that reads earnings calls or complaint emails produces a classification, and the explanation is the passage that drove it. Store the passage. Language models also misquote their sources more often than vendors suggest, a problem we measured in LLM web search is inaccurate more often than you think.
An explainability checklist before the model ships
Explainability is cheapest at design time and most expensive after a regulator asks. Before a finance model goes live, confirm each item below:
- Every automated decision stores its reasons at decision time, with the model version.
- Attribution outputs map to an approved, reviewed list of reason codes.
- The explanation method is validated against the model on held-out cases.
- The SHAP background dataset is fixed and versioned.
- Counterfactuals are limited to features a customer can change.
- Agent workflows log every tool call and output against a case ID.
- A named owner signs off on explanations, and can pause the model.
For the business side of taking a system like this from pilot to production, including who owns it and what it costs to run, AI Systems in Business covers the ground for managers. Start with the adverse action notice. If you can write that letter accurately for every declined applicant, most of the rest of explainable AI in finance follows.