Science Education Systems · Article 108. Maya, Jia Jun, Hana and Ethan are fictional learners used to make scientific reasoning visible. This article owns one distinct scientific job: model interpretability—understanding which inputs a fitted model uses, how those inputs influence predictions under the model, and where explanation tools become unreliable. It does not replace causal inference, mechanism discovery or model validation. Its central boundary is crucial: explaining a model is not the same as explaining the world.
The 50-second parent route
A predictive model can be accurate and still be difficult to inspect. Interpretability asks what the model learned, which signals it uses, how predictions change, and whether the explanation itself survives scrutiny.
The route is:
model → scientific question → global behaviour → local prediction → feature contribution → dependence and interaction → explanation method → stability → subgroup comparison → shortcut check → causal boundary → external validation
The fastest diagnostic is to ask: Is this explanation describing the model’s behaviour, or are we accidentally turning it into a claim about what causes the real-world outcome?
This article extends How Scientific Feature Selection Works, How Scientific Distribution Shift Works, How Scientific Causality Works and How Scientific Mechanisms Work.
1. Interpretability begins with a purpose
Do we need scientific understanding, debugging, regulatory explanation, user trust, failure diagnosis, feature audit or model comparison? Different purposes need different explanation methods.
2. There is no single universal definition of interpretability
Some people mean transparency of the model structure. Others mean post-hoc explanation of predictions. The intended meaning should be stated explicitly.
3. Maya’s first error is “feature importance proves causation”
Her repair is to remember that a predictive model can rely strongly on a proxy, collider, confounder or measurement artefact without identifying the true cause.
4. Jia Jun’s first error is trusting one explanation plot
His repair is to compare methods, check stability and ask whether correlated features or extrapolation distort the explanation.
5. Hana’s first error is explaining training data only
Her repair is to interpret model behaviour on held-out and deployment-like cases, because shortcuts often become visible only outside training.
6. Ethan’s first error is assuming a simple model is automatically truthful
His repair is to distinguish interpretability from correctness. A transparent wrong model is still wrong.
7. Intrinsic interpretability comes from model structure
Linear models, small decision trees, rule lists and sparse scoring systems can often be inspected directly.
8. Post-hoc interpretability explains a fitted black box
Permutation importance, partial dependence, SHAP, LIME and counterfactual explanations are common examples.
9. Intrinsic and post-hoc methods solve different jobs
A transparent model exposes its structure directly; a post-hoc explanation approximates or summarises behaviour after fitting.
10. Simplicity is relative
A linear model with 50,000 interaction terms is technically linear but not practically interpretable.
11. A small decision tree can be globally interpretable
The path from root to leaf exposes the rules used for each prediction.
12. Deep trees quickly become unreadable
Thousands of branches destroy practical transparency even though every split is explicit.
13. Sparse linear models are often easier to inspect
Fewer coefficients reduce cognitive load and can make sign and magnitude easier to discuss.
14. Coefficient sign is not always straightforward
In models with interactions, nonlinear transforms or correlated variables, a positive coefficient does not imply universally increasing predictions.
15. Scaling affects coefficient magnitude
A coefficient per metre cannot be compared directly with a coefficient per millimetre without considering units.
16. Standardised coefficients improve comparability but change meaning
They express effects in standard-deviation units rather than natural measurement units.
17. Logistic-regression coefficients operate on log-odds
A coefficient is not a direct probability change unless transformed through the logistic function and interpreted at a specific baseline.
18. Odds ratios can be interpretable
Exponentiating a logistic coefficient gives a multiplicative change in odds per unit predictor change under model assumptions.
19. Odds are not probabilities
A doubling of odds does not mean a doubling of probability.
20. Global explanations summarise the model overall
Examples include global feature importance, partial dependence and aggregated effect curves.
21. Local explanations target one prediction
They ask why this particular case received this score rather than another.
22. Local and global explanations can disagree
A feature can be important globally yet irrelevant for one specific case.
23. Global importance can hide subgroups
A variable may drive predictions only for one region of feature space.
24. Local explanations can hide systematic behaviour
Inspecting one case does not reveal how the model behaves across the population.
25. Permutation importance measures performance dependence
Shuffle one feature, break its relationship with the outcome and other features, then measure how much model performance degrades.
26. Large performance drop suggests the model relies on that feature
But the interpretation depends on the validation dataset and chosen performance metric.
27. Permutation importance is model-specific
A variable can be scientifically important yet unused by a particular fitted model.
28. Correlated features weaken ordinary permutation importance
One feature is shuffled, but its correlated partner still carries similar information, so importance appears artificially small.
29. Shuffling can create unrealistic feature combinations
If temperature and humidity are tightly linked, independent permutation can produce combinations never seen in reality.
30. Conditional permutation methods preserve dependence better
They shuffle a feature within comparable contexts or model its conditional distribution.
31. Impurity-based tree importance is convenient but biased
Features with many possible split points or high cardinality can receive inflated importance.
32. Gain-based importance reflects training decisions
It measures how much split criteria improved when the feature was used, not necessarily how much held-out performance depends on it.
33. Drop-column importance retrains without a feature
Compare the full model with a model trained from scratch after removing the variable.
34. Drop-column importance is expensive
Each feature requires another full training process.
35. Retraining changes correlated-feature compensation
The model may learn to use alternative features after one is removed, answering a different question from permutation importance.
36. Partial Dependence Plots show average model response
Vary one feature across a range while averaging predictions over the observed dataset.
37. PDP answers a model-based intervention-like question only superficially
It asks what the model would predict if the feature were set to different values while other features remained as observed.
38. PDP does not establish causal effects
Setting one feature while keeping correlated causes fixed can create impossible or confounded scenarios.
39. Correlated features make PDP dangerous
Changing one variable independently can move cases into unsupported combinations.
40. Individual Conditional Expectation curves preserve heterogeneity
Instead of averaging every case immediately, ICE plots show one response curve per observation.
41. ICE can reveal interactions
If different cases have different slopes, another variable is modifying the model’s response.
42. Centred ICE highlights shape differences
Subtract each case’s baseline prediction to focus on how the feature changes the prediction rather than absolute level.
43. Accumulated Local Effects reduce extrapolation under correlation
ALE estimates local prediction changes within regions where data actually exist and accumulates them across the feature range.
44. ALE is often safer than PDP for correlated predictors
It avoids evaluating the model far from the observed joint distribution as frequently.
45. ALE still explains the model, not the causal world
It reports predictive response under the fitted function.
46. Two-dimensional PDP or ALE can reveal interactions
They show how predictions vary across pairs of features.
47. Higher-order interpretation becomes difficult quickly
Human visualisation struggles beyond two or three dimensions.
48. Interaction strength can be measured
Friedman-like H-statistics quantify how much joint model behaviour departs from additive main effects.
49. Interaction strength depends on the fitted model
A model unable to represent interactions will report none, regardless of the real system.
50. SHAP comes from cooperative game theory
It assigns a model prediction among features using Shapley-value principles under a defined coalition and background distribution.
51. SHAP values are local additive attributions
The prediction is expressed as a baseline plus feature contributions under the chosen explanation setup.
52. Positive SHAP values push the prediction above baseline
Negative values push it below baseline.
53. The baseline matters
Change the reference population and the attribution changes.
54. SHAP values do not automatically mean causal contribution
They allocate predictive credit under a mathematical game, not physical causation.
55. Feature dependence complicates SHAP
Should missing features be treated as statistically independent, conditionally sampled, or integrated using another assumption?
56. Interventional SHAP and conditional SHAP answer different questions
One breaks dependencies more aggressively; the other tries to respect observed dependence.
57. Both require interpretation discipline
Neither automatically reconstructs a causal intervention.
58. TreeSHAP computes attributions efficiently for tree models
Algorithmic shortcuts make exact or near-exact Shapley-style values practical for large ensembles under specific assumptions.
59. KernelSHAP approximates SHAP for general models
It fits a weighted local surrogate over feature coalitions and can be computationally expensive.
60. Sampling noise can make approximate SHAP unstable
Repeated runs may produce different values unless enough samples are used.
61. LIME fits a local surrogate
Perturb cases around one prediction and fit a simple model to approximate the black box locally.
62. LIME explanations depend on neighbourhood definition
Kernel width, perturbation distribution and sampling strategy strongly affect results.
63. LIME can create unrealistic perturbations
Independent feature changes may produce samples outside the data manifold.
64. Local surrogate fidelity should be reported
If the simple surrogate poorly approximates the black box around the case, its explanation is weak.
65. Counterfactual explanations ask what minimal change would alter the prediction
“If income were X higher and debt Y lower, the decision would change.”
66. Counterfactual explanations are action-oriented
They can be easier for users to understand than dozens of feature-attribution values.
67. Predictive counterfactual is not causal advice
Changing a feature in the model does not guarantee changing it in reality will produce the predicted outcome.
68. Actionability matters
A counterfactual that asks a person to change age or birthplace is not useful as practical guidance.
69. Plausibility matters
Feature combinations should respect physical, temporal and social constraints.
70. Sparsity matters
Users often prefer a small number of meaningful changes over dozens of tiny changes.
71. Diversity matters
Several alternative counterfactuals can reveal multiple possible pathways to a different decision.
72. Causal counterfactuals require causal models
They account for how changing one variable affects downstream variables through structural relationships.
73. Feature attribution and counterfactual explanation answer different questions
Attribution asks what contributed to the current prediction; counterfactual asks what would need to change for the prediction to change.
74. Example-based explanations use similar cases
Show prototypes, nearest neighbours or influential training examples.
75. Similarity depends on representation
Two cases can be close in embedding space yet differ in scientifically critical variables.
76. Prototypes summarise typical cases
They can help humans understand regions of the model’s input space.
77. Criticisms or boundary cases show where prototypes fail
They reveal unusual observations poorly represented by the typical examples.
78. Influence functions estimate training-example impact
They approximate how model parameters or a prediction would change if one training case were upweighted or removed.
79. Influence methods depend on smoothness and approximation assumptions
They can be unstable in large non-convex neural networks.
80. Data attribution links outputs to training examples
It can help debug memorisation, contamination or harmful examples.
81. Attribution to training data is not provenance by itself
A high influence score does not prove the output literally copied that example.
82. Saliency maps explain image-model sensitivity
Gradients or perturbations highlight pixels associated with a prediction.
83. Saliency can look convincing while being unstable
Small input perturbations or implementation choices can change the map dramatically.
84. Sanity checks are essential
Randomise model parameters or labels and see whether the explanation meaningfully changes.
85. An explanation method that survives a randomised model is suspicious
It may be showing generic image structure rather than learned reasoning.
86. Attention weights are not automatically explanations
A transformer attending to a token does not prove that token causally determined the output.
87. Attention can be useful diagnostic information
But it should be combined with interventions, ablations and other evidence.
88. Ablation tests remove or alter information
If prediction changes strongly after one feature or component is removed, the model depended on it.
89. Ablations can break the input distribution
Removing a feature unrealistically may create a case the model was never designed to handle.
90. Mechanistic interpretability examines internal computation
For neural networks, researchers study neurons, circuits, representations and causal interventions inside the model.
91. Internal feature discovery is difficult
One neuron can be polysemantic, responding to several unrelated patterns.
92. Distributed representations spread concepts across many units
No single neuron may correspond neatly to one human concept.
93. Sparse autoencoders can extract latent features
They attempt to decompose dense activations into more interpretable sparse directions.
94. Extracted features remain model-dependent constructs
Naming a latent direction “honesty” or “syntax” requires rigorous behavioural validation.
95. Causal interventions inside a model are stronger than observation alone
Activate, suppress or patch an internal feature and observe whether model behaviour changes as predicted.
96. Internal causality is not world causality
Showing that a hidden feature causes a model output does not show the corresponding real-world concept causes the target event.
97. Model debugging is a major interpretability use
Find shortcuts, data leakage, unstable features or unintended dependencies.
98. Shortcut learning often appears through explanation
An image classifier may focus on a hospital watermark rather than pathology.
99. Shortcut explanation should trigger external validation
Hold out the site, remove the watermark and test whether performance survives.
100. Interpretability helps diagnose distribution shift
If a high-importance feature drifts, the model’s deployment risk may rise.
101. Feature-importance drift can be monitored
A model may start relying on different variables after retraining or environmental change.
102. Explanation drift can reveal behaviour change before headline metrics collapse
The model remains accurate overall but begins using fragile proxies.
103. Explanation stability is itself measurable
Repeat explanations across bootstraps, seeds, model versions and small input perturbations.
104. Unstable explanations deserve caution
If the explanation changes dramatically while the prediction barely changes, the attribution may not be robust enough for scientific claims.
105. Rank stability is one simple check
Do the top important features remain similar across resamples?
106. Attribution magnitude stability is stronger
Compare actual contribution values, not only rank order.
107. Direction stability matters
Does a feature consistently push predictions up, or does its sign flip across samples?
108. Local explanation stability matters most near decision boundaries
Small perturbations can flip the prediction and its explanation simultaneously.
109. Explanations can be faithful or plausible
A faithful explanation accurately reflects model behaviour. A plausible explanation merely sounds reasonable to humans.
110. Plausibility without fidelity is dangerous
A neat story can build trust in a model for the wrong reason.
111. Fidelity should be tested quantitatively
For surrogate explanations, measure how closely the explanation approximates the model in the intended region.
112. Completeness is another goal
Does the explanation account for the entire prediction or only selected parts?
113. Simulatability is another goal
Can a human use the explanation to predict what the model would do on a new case?
114. Decomposability is another goal
Can each component—feature, parameter, rule—be understood individually?
115. Algorithmic transparency is another goal
Can we understand the training procedure and convergence behaviour?
116. Different stakeholders need different explanations
A scientist, regulator, engineer, teacher and end user may require different levels of detail.
117. Technical explanations can overwhelm users
Hundreds of SHAP values may be accurate but unusable.
118. Simplification can omit important caveats
A one-line reason code may hide interactions or uncertainty.
119. Explanation interfaces should show uncertainty
Feature contributions estimated from finite data or approximate methods are not exact.
120. Explanation uncertainty can come from data
Refit the model on bootstrap samples and observe how explanations change.
121. Explanation uncertainty can come from the method
Sampling-based SHAP or LIME introduces Monte Carlo variation.
122. Explanation uncertainty can come from model multiplicity
Several equally accurate models can rely on different features.
123. Rashomon sets describe this multiplicity
Many models fit the data nearly equally well yet tell different explanatory stories.
124. Predictive equivalence does not imply explanatory equivalence
One model uses temperature; another uses humidity. Both predict similarly because those variables are correlated.
125. Model multiplicity weakens feature-level certainty
If many near-optimal models disagree about importance, do not report one explanation as uniquely determined.
126. Variable importance should be compared across plausible models
Stable conclusions across the Rashomon set are stronger.
127. Interpretability and feature selection overlap but differ
Feature selection chooses variables. Interpretability explains how the fitted model uses them.
128. Interpretability and regularisation overlap but differ
Sparse regularisation may make models easier to inspect, but interpretability is not the same as complexity control.
129. Interpretability and causality differ fundamentally
Predictive dependence can arise from confounding, mediation, selection bias or proxy variables.
130. A causal claim needs interventions or defensible causal assumptions
Feature importance alone does not identify causes.
131. Interpretability and mechanisms differ too
A model can rely on a feature that correlates with a mechanism without representing the mechanism internally.
132. Mechanistic validation requires intermediate predictions
Test whether the model predicts the internal process, not only the final outcome.
133. Interpretability and calibration are different
A transparent model can be poorly calibrated, and a black box can be well calibrated.
134. Interpretability and fairness are connected
Explanations can reveal reliance on protected attributes or proxies.
135. Removing a protected feature does not remove proxy dependence
Postcode, school, language or income may encode similar information.
136. Fairness audits need outcome metrics too
An explanation showing one feature matters does not quantify subgroup error or harm by itself.
137. Worked case: medical risk model
A gradient-boosted model predicts complications. SHAP says age, prior admissions and a laboratory marker are influential.
138. The first question is predictive fidelity
Do perturbation and ablation tests confirm that predictions truly depend on those features?
139. The second question is causal boundary
Age importance does not mean changing age would change outcome; it is a risk marker.
140. The third question is deployment stability
Does the laboratory marker retain the same measurement meaning across hospitals?
141. Worked case: machine failure
A model relies on vibration frequency, temperature and maintenance age.
142. PDP suggests sharply rising risk above a vibration level
But temperature and vibration are correlated, so PDP may evaluate unrealistic combinations.
143. ALE provides a more local alternative
Then a controlled engineering test can determine whether the apparent threshold reflects a real mechanism.
144. Worked case: learner diagnostic model
A model predicts failure on unseen Science questions from retrieval score, reading accuracy, explanation structure and response time.
145. Feature importance says explanation structure dominates
That tells the tutor what the model relies on, not automatically what caused the learner’s difficulty.
146. A targeted intervention provides stronger causal evidence
Improve explanation construction while holding other supports stable, then test unseen transfer.
147. Worked case: image classifier
Saliency highlights the corner of the image rather than the object.
148. A watermark shortcut is suspected
Remove or randomise the watermark and measure the performance drop.
149. Counterfactual image editing can test the shortcut
Change the corner while preserving the object. If the label flips, the shortcut has causal influence inside the model.
150. This still does not make the watermark causally related to the real-world class
It causes the model’s decision, not the phenomenon.
151. Primary Science can learn interpretability through rule inspection
Ask a child to explain which observation their rule used and whether that observation really belongs to the scientific concept.
152. Primary 3 can compare correct answer and correct reason
A learner can guess the right outcome for the wrong reason. Interpretability makes the reason visible.
153. Primary 4 can inspect a simple decision tree
Follow branches and identify which variable changed the final classification.
154. Primary 5 can spot proxy rules
A rule says “red objects conduct electricity” because every training example happened to align. Test a new red insulator.
155. Primary 6 can distinguish prediction from explanation
A feature helps predict plant growth but may not be the cause of growth.
156. Secondary Science can formalise permutation importance
Shuffle one variable in held-out data and quantify performance loss.
157. Secondary Science can compare PDP and ALE
Use correlated predictors to show why support and extrapolation matter.
158. Secondary Science can use local explanations
Decompose one prediction into a baseline and feature contributions.
159. Secondary Science can audit explanation stability
Refit the model across bootstrap samples and track whether importance rankings persist.
160. AI assistants need interpretability too
We may want to know which retrieved documents, tools, instructions or intermediate states influenced an answer.
161. Citation is a form of provenance, not full explanation
A cited source shows where evidence came from but not exactly how every token of the answer was generated.
162. Tool traces improve operational interpretability
They show which calculator, search result or database entry fed the system.
163. Reasoning summaries can improve human understanding
But concise explanations should be evaluated for fidelity rather than assumed to reveal internal computation perfectly.
164. Retrieval attribution is important in RAG systems
Was the answer based on the most relevant document, a distractor or unsupported prior knowledge?
165. Removing a retrieved document is an ablation
If the answer changes substantially, that source had behavioural influence.
166. Prompt sensitivity is interpretability evidence
Small wording changes that produce large answer changes reveal brittle decision boundaries.
167. System-prompt and tool-policy changes can alter explanations
Interpretation must be version-specific.
168. AI can help learners practise interpretability
Useful prompts include: “Create correlated features that fool PDP,” “Compare permutation importance and SHAP,” “Give me a counterfactual that is predictive but not causal,” and “Design an explanation-stability bootstrap.”
169. AI can fabricate explanatory certainty
A fluent reason for a prediction is not evidence that the model actually used that reason internally.
170. Parents can use interpretability in learning diagnosis
Do not ask only whether the answer is correct. Ask what clue the learner used and whether that clue transfers to a new context.
171. Tutors can distinguish right answer from right model
A child may arrive at the correct choice using a brittle shortcut. An oral explanation reveals the internal rule more directly.
172. Transfer questions are interpretability tests
Change surface features while preserving mechanism. If performance collapses, the learner may have relied on the wrong cue.
173. Explanation stability can be tested across contexts
Ask the learner to explain the same mechanism in several settings. A stable causal structure is stronger evidence than repeated vocabulary.
174. Independent-attempt task 1: coefficient interpretation
Given a standardised linear model, identify the largest coefficients and explain why magnitude does not prove causality.
175. Independent-attempt task 2: correlated permutation
Create two correlated predictors and predict how ordinary permutation importance can undervalue each one.
176. Independent-attempt task 3: PDP support
Sketch correlated temperature and humidity data, then identify impossible combinations a PDP might evaluate.
177. Independent-attempt task 4: local explanation
Take one prediction, define a baseline and assign feature contributions that sum to the prediction.
178. Independent-attempt task 5: counterfactual actionability
Compare a counterfactual changing age with one changing a modifiable sensor setting. Which is more actionable and why?
179. Independent-attempt task 6: causal boundary
Find a predictive feature that is a proxy. Explain how it can be important to the model without causing the outcome.
180. Independent-attempt task 7: explanation stability
Refit a thought-experiment model across five bootstrap samples and track how top features change.
181. Diagnostic error: feature importance equals causality
Repair with causal design or interventions.
182. Diagnostic error: one explanation method treated as ground truth
Repair by triangulating methods and testing fidelity.
183. Diagnostic error: correlated features ignored
Repair with conditional methods, grouped interpretation or domain knowledge.
184. Diagnostic error: local explanation generalised globally
Repair by checking population-wide behaviour.
185. Diagnostic error: global importance applied to one case
Repair by using local contributions for individual predictions.
186. Diagnostic error: PDP interpreted causally
Repair by acknowledging unsupported interventions and feature dependence.
187. Diagnostic error: SHAP baseline ignored
Repair by stating the reference distribution used.
188. Diagnostic error: counterfactual prediction treated as real-world advice
Repair by checking actionability and causal structure.
189. Diagnostic error: explanation stability not tested
Repair across seeds, resamples and model versions.
190. Diagnostic error: simple model assumed correct
Repair with validation, residual checks and external evidence.
191. Diagnostic error: explanation interface hides uncertainty
Repair by reporting ranges, instability or alternative plausible explanations.
192. Diagnostic error: explanation used to excuse poor performance
Repair by remembering that interpretability does not compensate for invalid predictions.
193. The independence test
Give a learner a black-box model, correlated features and several explanation tools. Can they separate global from local behaviour, predictive dependence from causality, and stable evidence from explanation artefact? That is transferable interpretability reasoning.
194. The evidence boundary
Interpretability methods explain a fitted model under specific assumptions, reference distributions and perturbation schemes. They do not automatically reveal real-world mechanism, causal effect or human-like internal reasoning. Scientific claims must stop where the explanation method stops.
195. A compact model-interpretability checklist
- Why is interpretation needed?
- Is the model intrinsically interpretable or explained post hoc?
- Is the question global or local?
- What reference population or baseline is used?
- Are predictors strongly correlated?
- Does the method create unrealistic perturbations?
- What does permutation importance say?
- What do PDP or ALE curves say?
- Are interactions present?
- What do local attributions say?
- Are SHAP assumptions appropriate?
- Is a counterfactual plausible and actionable?
- Does the explanation have measured fidelity?
- How stable is the explanation across resamples?
- Do near-equivalent models tell different stories?
- Could the model be using a shortcut?
- Has explanation behaviour changed under distribution shift?
- Is the claim predictive or causal?
- What external evidence supports the interpretation?
- Would an intervention test the proposed mechanism?
196. Frequently asked questions
What is model interpretability?
Model interpretability is the ability to understand and communicate how a predictive model uses inputs, forms predictions and responds to changes under defined assumptions.
What is permutation importance?
It measures how much held-out model performance decreases when one feature is shuffled.
What is a partial dependence plot?
It shows the average model prediction as one or more features are varied, usually while averaging over the observed values of other features.
What is ALE?
Accumulated Local Effects estimates local prediction changes in regions supported by the data and is often more suitable than PDP when predictors are correlated.
What is SHAP?
SHAP is a family of feature-attribution methods based on Shapley-value ideas that decompose a prediction relative to a reference baseline.
Does SHAP show causality?
No. It attributes model predictions under assumptions; causal effects require causal structure and evidence beyond predictive attribution.
What is a counterfactual explanation?
It describes how a prediction would change if selected input features were different, ideally under plausible and actionable constraints.
How does interpretability help PSLE Science?
The formal tools are advanced, but the core habit is familiar: a correct answer is not enough; we need to know which rule or evidence produced it.
How does it deepen in Secondary Science?
Students can distinguish model explanation from causal explanation and analyse coefficients, feature importance, response curves, local attributions and explanation stability more formally.
197. Continue the Science Education Systems series
- How Scientific Feature Selection Works
- How Scientific Distribution Shift Works
- How Scientific Causality Works
Conclusion: Explaining the model is the beginning of scientific interpretation, not the end
Maya asks what feature mattered.
Jia Jun asks whether another explanation method agrees.
Hana asks whether the feature is a proxy, shortcut or mechanism.
Ethan asks whether changing that feature would actually change the real-world outcome.
Science needs all four.
Inspect the model.
test the explanation.
respect feature dependence.
measure stability.
separate prediction from causation.
Then use interpretability to reveal how the model behaves without pretending that the model’s behaviour is automatically the mechanism of the world.
