Why Most AI Explanations Are Fake

Here's something I've watched happen dozens of times.

A data science team builds a machine learning model. It predicts customer churn, or production defects, or loan defaults. The model performs well on the test set. Accuracy is high. The team is excited.

Then they show it to the executives who need to actually make decisions based on it.

"Why does the model think this customer will churn?"

The data scientist pulls up a SHAP plot. "Well, you can see here that the top features are contract length, monthly charges, and tenure. The red bars push toward churn, the blue bars push away from it."

The executive stares at the colored bars. "So... what should we do about it?"

Silence.

This is the moment when most enterprise AI projects die. Not because the model doesn't work-it does. But because the explanation isn't actually an explanation. It's a statistical summary dressed up to look like understanding.

And the executives know it. That's why they fall back on regression models and Excel spreadsheets-because those they can actually understand and defend to a board.

The Explanation Theater

SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) are the two most popular methods for "explaining" machine learning models. They've become industry standards. Every major ML platform includes them. They've been cited thousands of times in academic papers.

They're also fundamentally incapable of answering the questions that matter.

Here's what SHAP actually tells you: for this particular prediction, how much did each feature contribute to pushing the output away from the average prediction?

Here's what it doesn't tell you:

• Why that feature matters
• What would happen if you changed it
• Whether the relationship is causal or spurious
• What you should actually do about it

A SHAP plot for a churn model might show that "contract length" is the most important feature. Great. What does that mean? Should you offer longer contracts? Would that reduce churn, or do people who sign longer contracts just happen to be less likely to churn for entirely different reasons?

SHAP can't answer this. It's not designed to. It's designed to decompose a prediction into feature contributions. That's a mathematical operation, not an explanation.

The same is true for LIME. It fits a simple model locally around a prediction and shows you the coefficients. This tells you how the black box behaves in that neighborhood. It doesn't tell you anything about the underlying reality the model is supposed to represent.

The Fundamental Problem

The issue isn't that SHAP and LIME are technically flawed. They do exactly what they claim to do. The issue is that what they do isn't what enterprises actually need.

What enterprises need is answers to questions like:

• "If we increase marketing spend by 20%, what will happen to sales?"
• "Why did production yield drop last quarter?"
• "What's causing customers to leave, and what can we do about it?"

These are causal questions. They're about interventions and their effects. They're about understanding mechanisms, not just patterns.

Feature importance scores don't answer causal questions. They answer a different question entirely: "What patterns did the model learn from the historical data?"

I've watched a data science team present a model showing that "number of support tickets" was a top predictor of churn. The executive asked, "So should we make it harder to submit tickets?"

Obviously not. The support tickets aren't causing churn-they're a symptom of the underlying problems that cause churn. Intervening on the symptom doesn't fix the cause. A SHAP plot can't tell you this. You need to understand the causal structure.

The Regression Trap

I spent a lot of time in the early days of RootCause watching enterprise analytics teams work. One pattern kept emerging: when the ML models couldn't be explained, teams would fall back to regression.

"Let's just run a regression with these variables as controls."

The reasoning is understandable. Regression coefficients are interpretable. You can say "for every unit increase in X, Y goes up by β, holding everything else constant." Executives understand this language. It feels like science.

But here's the problem: "holding everything else constant" is a fantasy. In the real world, everything affects everything else. When you increase marketing spend, you don't magically hold consumer sentiment constant. When you change your pricing, you don't hold competitor behavior fixed.

More importantly, regression gives you association with controls-not causal effects. If ice cream sales and drowning deaths both go up in summer, a regression will show a positive coefficient. It will dutifully tell you that "controlling for other factors, ice cream sales are associated with drowning."

This is useless for decision-making. It's not even wrong-it's just not answering the question you actually have.

I'll give you a real example, though I've changed the details for confidentiality.

A client was convinced their new digital channel was cannibalizing their traditional business. They ran regressions. They controlled for everything they could think of-macroeconomic factors, seasonality, demographic shifts, you name it. The regression showed a negative coefficient between digital and traditional revenue. Case closed, right?

Except when we built a causal model of their business, we found the arrow pointing the other direction. The traditional channel was declining due to secular trends, and customers were shifting to digital as a result. The new channel was backfilling lost customers, not stealing existing ones.

The regression told them to restrict the digital channel. The causal model told them the opposite. One of these recommendations would have cost them millions in foregone growth while doing nothing to stop the underlying decline.

Why Executives Don't Trust AI

Post-ChatGPT, everyone wants AI. But trust in AI is collapsing. Gartner reports that 70% of AI projects fail. Most enterprises I talk to have a graveyard of abandoned ML initiatives.

The common diagnosis is "implementation problems" or "data quality issues" or "organizational resistance." These are real. But there's a deeper issue that rarely gets named:

You can't take a machine learning output to make a company-wide decision.

Executives need evidence. They need models they can understand and defend. They need to answer "why" when the board asks. A SHAP plot doesn't give them that. A feature importance ranking doesn't give them that.

What gives them that? Models they can trace. Relationships they can verify. Mechanisms they can explain to a non-technical audience.

That's why they fall back on regression models and Excel spreadsheets. Not because Excel is better technology-obviously it isn't. But because the relationships are transparent. You can see the formula. You can argue about whether the assumptions make sense. You can explain it to someone who wasn't in the room when the analysis was done.

Black-box ML with post-hoc explanations bolted on top doesn't provide this. It provides the illusion of explanation while leaving the actual mechanism hidden.

The Causal Alternative

There's a reason I spent years building causal AI systems instead of better explanation tools for existing ML. The explanation problem isn't a UI problem or a visualization problem. It's a fundamental problem with the underlying approach.

When you build a causal model, the explanation is the model. Every relationship in a causal graph has a meaning: A causes B through C. This isn't a post-hoc rationalization-it's the structure you learned from the data.

When you simulate an intervention in a causal model, you're not asking "what does the model think?" You're asking "what would happen if we actually did this?" The math is different. The answer is different. And crucially, the explanation is built in.

"If you increase marketing spend by 20%, revenue will increase by 8%, primarily through the mechanism of increased website traffic leading to higher conversion rates. Here's the causal pathway. Here's the uncertainty. Here's what else would be affected."

This is an explanation an executive can work with. They can ask follow-up questions. They can challenge the assumptions. They can trace the logic. They can explain it to the board.

Not because we added a better visualization layer. Because the underlying model is actually modeling causation, not just correlation.

The Audit Problem

Here's another angle on this that matters increasingly: regulation.

The EU AI Act is coming. Financial services have always had explainability requirements. Healthcare is heavily regulated. Enterprises need to demonstrate that their AI systems make decisions for defensible reasons.

SHAP and LIME were designed for a different era-an era where "we can show you the feature importances" was enough. That era is ending.

A regulator asking "why did your algorithm deny this loan?" won't be satisfied with "the SHAP values show that income was the primary factor." They'll want to know: what's the relationship between income and default risk? Is that relationship causal or merely correlative? Could it be proxying for something else? Is the model using income appropriately, or is it just memorizing patterns in the training data?

Feature importance can't answer these questions. Causal models can-because the structure itself encodes the relationships, and those relationships can be audited, challenged, and validated.

This is where the industry is heading. Not better visualizations of black boxes, but transparent models where the explanation is the mechanism.

What Actually Helps

If you're building ML systems for enterprise decision-making, here's what I've learned:

1. Distinguish prediction from decision-making.

Prediction is about forecasting what will happen given current trends. Decision-making is about choosing interventions to change what will happen. These require different tools. A model that predicts churn is not the same as a model that tells you how to reduce churn.

2. Ask "what should we do?" before "how accurate is it?"

If your model can't answer intervention questions, it's a forecasting tool, not a decision-support tool. That's fine-forecasting is valuable. But don't pretend it's something it's not.

3. Be suspicious of any "explanation" that doesn't tell you about mechanisms.

Feature importance tells you what the model learned. It doesn't tell you how the world works. These are not the same thing. One is useful for debugging your model. The other is useful for making decisions.

4. Build models that are transparent by construction, not by post-hoc interpretation.

The best explanation isn't a visualization layer on top of a black box. It's a model whose structure is itself the explanation. Causal graphs, decision trees, rule systems-these have tradeoffs, but they share the property that the reasoning is visible.

5. Remember who has to defend the decision.

The executive presenting to the board doesn't have the luxury of saying "the SHAP values indicate..." They need to say "here's why we believe this will work, here's the evidence, and here's what we're assuming." Your explanation needs to survive that translation.

The Uncomfortable Truth

The enterprise AI industry has a dirty secret: most deployed ML systems aren't trusted by the people who make decisions. The models run in the background. The predictions get generated. And then humans override them based on intuition, politics, and spreadsheets.

This isn't because executives are Luddites or because data scientists are bad communicators. It's because we've been optimizing for the wrong thing.

We optimized for prediction accuracy on held-out test sets. We should have optimized for decision-relevance in the real world.

We built sophisticated models and then bolted on explanation tools as an afterthought. We should have built models where the explanation is inherent in the structure.

We asked "how do we explain what the model is doing?" We should have asked "how do we model what's actually causing what?"

SHAP and LIME aren't the problem. They're symptoms of a deeper issue: we've been building AI systems that can predict without understanding, and then wondering why decision-makers don't trust them.

The solution isn't better explanation tools. It's better models-models that actually capture the causal structure of the systems they're meant to help us understand.

That's a harder problem. But it's the right one.