Science Education Systems · Article 109. Maya, Jia Jun, Hana and Ethan are fictional learners used to make scientific reasoning visible. This article owns one distinct scientific job: conformal prediction—building prediction sets or intervals whose long-run coverage can be measured and, under stated assumptions, guaranteed at a chosen level. It does not replace probability calibration, confidence intervals or ordinary prediction. Its job is to wrap uncertainty around a prediction while keeping the guarantee tied to exchangeability, the calibration sample and the exact prediction task.
The 50-second parent route
A point prediction says, “I think the answer is 72.” Conformal prediction asks a different question: how wide must the prediction set be so that the true answer is included with a chosen long-run frequency?
The route is:
base predictor → held-out calibration data → nonconformity score → calibration-score distribution → coverage target → quantile → prediction set or interval → empirical coverage → efficiency → subgroup check → shift check → update
The fastest diagnostic is to ask: what exactly is guaranteed, under which exchangeability or related assumption, and is the guarantee marginal or conditional?
This article extends How Scientific Probability Calibration Works, How Scientific Uncertainty Works, How Scientific Cross-Validation Works and How Scientific Distribution Shift Works.
1. Prediction needs more than one number
A forecast of 72 looks precise. But if comparable outcomes sometimes land at 55 and sometimes at 89, the point estimate hides decision-relevant uncertainty.
2. Conformal prediction wraps uncertainty around any base predictor
The base model can be linear regression, random forest, neural network or another method. Conformal logic uses held-out calibration residuals or scores to determine how much uncertainty to add.
3. Maya’s first error is calling a 90% conformal interval a 90% probability that this one outcome lies inside
Her repair is to distinguish a repeated-sampling coverage guarantee from an individual posterior probability statement.
4. Jia Jun’s first error is forgetting exchangeability
He quotes the guarantee after the data-generating process has shifted. His repair is to state the assumption connecting calibration and future cases.
5. Hana’s first error is checking only coverage
An interval from minus infinity to plus infinity covers everything. Her repair is to evaluate efficiency—how narrow or informative the sets are while retaining valid coverage.
6. Ethan’s first error is claiming conditional coverage from marginal coverage
The method can cover 90% overall while covering only 70% in one subgroup. His repair is to audit subgroups and local regions separately.
7. The base predictor can be ordinary
Suppose a regression model predicts y-hat from inputs x. Conformal prediction does not require that the base predictor itself produce an uncertainty interval.
8. Calibration data must be kept separate from model fitting in split conformal
Train the base model on one subset, then use a different calibration subset to measure typical prediction errors.
9. Nonconformity scores measure how unusual an outcome is relative to the prediction
For simple regression, a common score is the absolute residual |y − y-hat|.
10. Large nonconformity means poor fit
A calibration case with a large residual is less conforming to the model’s typical prediction behaviour.
11. Calibration scores create an empirical uncertainty distribution
Sort the residuals and choose a high quantile corresponding to the desired coverage target.
12. The selected quantile becomes the interval radius in basic split conformal
If the 90th-percentile calibration residual is 8, a new prediction of 72 can produce approximately [64, 80], with the exact finite-sample quantile adjusted according to conformal rules.
13. The finite-sample adjustment matters
Naively taking the empirical 90th percentile can miss the exact rank correction that gives the stated finite-sample marginal coverage.
14. Rank logic drives the guarantee
Under exchangeability, the future nonconformity score has a rank among calibration scores that is symmetric enough to control miscoverage.
15. Exchangeability is weaker than assuming a specific probability distribution
The method does not need residuals to be Gaussian in the basic formulation.
16. Exchangeability is still a real assumption
If future cases are systematically different from calibration cases, the symmetry behind the guarantee can fail.
17. Independent and identically distributed data imply exchangeability
But exchangeability can also arise in broader settings where order does not change the joint distribution.
18. Time series usually violate naive exchangeability
Tomorrow is not interchangeable with a randomly chosen historical day when trend, seasonality or drift matters.
19. Spatial data can violate naive exchangeability
Nearby sites share environmental structure, so random calibration splits can be overoptimistic.
20. Grouped data can violate naive exchangeability
Repeated measurements from the same patient or student are more similar than measurements from new people.
21. Split conformal is simple and modular
Train once, calibrate once, then construct sets for future cases.
22. Split conformal sacrifices some training data
The calibration subset cannot be used to fit the base predictor under the simplest workflow.
23. Data efficiency matters when samples are small
A weak base model creates wide intervals; a tiny calibration set creates noisy quantiles.
24. Full conformal methods reuse data more aggressively
They conceptually refit or evaluate conformity with each candidate outcome, increasing computational cost.
25. Full conformal is often expensive
For large modern models, repeated retraining can be impractical.
26. Jackknife-like and cross-conformal methods improve data reuse
They combine resampling ideas with conformal calibration so more observations contribute to model fitting and uncertainty estimation.
27. Cross-validation-plus is one important regression approach
Predictions from cross-validation folds are combined with residual information to form intervals with rigorous coverage properties under stated conditions.
28. Jackknife-plus uses leave-one-out style fits
It can provide strong finite-sample coverage guarantees while reducing the data-splitting penalty.
29. Computational budget shapes method choice
A method requiring thousands of large-model refits may be scientifically elegant but operationally impossible.
30. Regression conformal prediction can use absolute residuals
That creates constant-width intervals around the base prediction.
31. Constant width is often inefficient
Some inputs are easy to predict and others are difficult. One global residual quantile treats them alike.
32. Heteroscedasticity motivates locally scaled scores
Divide residuals by a predicted scale so easy cases can receive narrower intervals and noisy cases wider intervals.
33. The scale model becomes part of the pipeline
If local uncertainty estimates are poor, interval efficiency and local coverage can suffer.
34. Conformalized quantile regression uses lower and upper quantile models
The base model predicts an asymmetric interval, then conformal calibration expands or adjusts it to restore target marginal coverage.
35. Asymmetric intervals matter for skewed outcomes
Future demand, waiting time or biological concentrations may have longer upper tails than lower tails.
36. Quantile regression can adapt to x-dependent noise
Conformal correction then protects coverage against finite-sample imperfections in the estimated quantiles.
37. Conformal prediction does not make a bad base model good
If the predictor misses structure, intervals become wide to compensate.
38. Better base prediction often improves efficiency
Coverage can remain valid across model families, but sharper models can produce narrower sets.
39. Classification conformal prediction outputs sets of labels
Instead of predicting one class, the method can return {cat}, {cat, fox}, or a larger candidate set.
40. Singleton sets are decisive
A one-label prediction set says the conformal rule found only one label sufficiently compatible with the case.
41. Multi-label sets express ambiguity
If several labels are plausible, the set can include all of them rather than forcing one answer.
42. Empty sets can occur in some conformal constructions
An empty set can signal that no candidate label looks compatible at the chosen threshold or that the scoring rule is poorly designed.
43. Set size is the classification efficiency metric
Coverage alone is not enough; a method returning every possible class always is useless.
44. Probability-based nonconformity scores are common
A true label receiving low model probability is treated as highly nonconforming.
45. Adaptive Prediction Sets accumulate high-probability labels
They can better adapt set size to difficult versus easy examples, especially in multi-class tasks.
46. Ranking quality affects set efficiency
If the base classifier ranks the true label poorly, conformal sets must grow to preserve coverage.
47. Marginal coverage is the default guarantee to understand
Across future cases drawn under the required assumptions, the true outcome is included with at least the target long-run frequency up to the method’s exact finite-sample statement.
48. Marginal does not mean every subgroup
A 90% overall method can under-cover a rare or difficult subgroup while over-covering an easy subgroup.
49. Conditional coverage asks for reliability at each x or subgroup
That is a much stronger requirement and is generally impossible to obtain distribution-free in a fully general finite-sample way without paying a severe price or adding assumptions.
50. Local reliability should still be audited
Even when the formal guarantee is marginal, practitioners can test coverage by subgroup, risk level, site, time and feature region.
51. Conditional undercoverage is a safety issue
A model may be valid overall while systematically missing outcomes in exactly the population that most needs protection.
52. Conditional overcoverage is an efficiency issue
Some groups may receive huge uninformative sets even though overall coverage looks excellent.
53. Coverage and efficiency form a trade-off
Higher coverage targets usually produce wider intervals or larger classification sets.
54. A 99% set is usually less informative than a 90% set
The extra protection costs width or ambiguity.
55. The “right” coverage target depends on the decision
A low-cost recommendation may tolerate more miscoverage than a safety-critical intervention.
56. Coverage target is a policy choice
Conformal machinery enforces the chosen target under assumptions; it does not decide which target society should choose.
57. Adaptive coverage policies can vary target levels across contexts
But the policy itself needs justification so protection is not weakened precisely where uncertainty is greatest.
58. Coverage should be reported with set size
Otherwise a method can look excellent by being maximally conservative.
59. Calibration-set size controls quantile resolution
With only nine calibration cases, attainable empirical ranks are coarse.
60. Small calibration sets produce unstable widths
One extreme residual can shift the chosen quantile substantially.
61. Bootstrap diagnostics can assess width stability
Resample calibration cases and see how the conformal quantile changes.
62. Bootstrap does not replace the conformal guarantee
It is a diagnostic for instability, not the source of formal coverage.
63. Calibration contamination widens or distorts intervals
Outliers, label errors or mixed regimes become part of the residual distribution.
64. Removing “bad” calibration cases can invalidate coverage if done opportunistically
Cleaning decisions should be prespecified and scientifically justified.
65. Distribution shift is the central deployment threat
Exchangeability can fail when the future differs systematically from calibration data.
66. Covariate shift can change local error rates
New input regions may have larger residuals than the calibration quantile anticipates.
67. Concept drift can destroy coverage more deeply
The same x no longer predicts y in the same way, so old nonconformity scores are stale.
68. Measurement shift can create artificial miscoverage
A sensor change modifies residual scale even if the physical process stays stable.
69. Weighted conformal methods address some non-exchangeable settings
They reweight calibration scores to better represent a target distribution under additional assumptions.
70. Weight estimation becomes another source of error
Poor density ratios or weak overlap can make weighted procedures unstable.
71. Time-adaptive conformal methods update calibration emphasis
Recent errors can receive greater weight when the process drifts gradually.
72. Adaptation trades classical symmetry for extra modelling assumptions
The guarantee changes and must be stated precisely rather than borrowed from ordinary split conformal.
73. Online conformal prediction updates sequentially
New outcomes can be used to refresh conformity scores or monitor coverage over time.
74. Delayed labels complicate online adaptation
If outcomes arrive months later, the system cannot immediately know whether its intervals are still covering correctly.
75. Prequential evaluation tracks predictions before outcomes arrive
Make the prediction, freeze it, then score coverage once the outcome becomes available.
76. Coverage monitoring should use rolling windows cautiously
Short windows react quickly but are noisy; long windows are stable but slow to reveal drift.
77. Conformal prediction is not probability calibration
Calibration asks whether 0.7 means seven in ten. Conformal prediction asks whether prediction sets achieve a target inclusion frequency.
78. A model can be poorly probability-calibrated yet have valid conformal coverage
If the nonconformity scores and calibration process work under the assumptions, set coverage can be valid despite distorted probabilities.
79. A calibrated model can still have poor conformal efficiency
Good probabilities do not guarantee narrow regression intervals or small classification sets.
80. Conformal prediction is not a confidence interval for a parameter
A confidence interval might estimate an unknown regression coefficient. A conformal prediction interval targets a future outcome.
81. Prediction intervals are wider than parameter intervals for a reason
They include both uncertainty about the model and irreducible future outcome variability in different ways depending on the method.
82. Conformal sets are not Bayesian credible intervals
Bayesian intervals derive from a posterior distribution under a prior and likelihood. Conformal coverage comes from rank symmetry under exchangeability or related assumptions.
83. Conformal prediction and selective prediction are connected
A large conformal set can tell a system that the case is ambiguous enough to defer.
84. Set size can become an abstention signal
If a classification set contains five labels, automatic action may be inappropriate.
85. Empty or huge sets can trigger review
The prediction set itself can become an operational uncertainty interface.
86. Selective prediction owns the reject decision
Article 110 focuses on when the model should abstain rather than force an answer.
87. Conformal prediction and decision thresholds are connected
A decision-maker can act only when an interval stays entirely above or below an operational threshold.
88. Interval-threshold interaction can be safer than point-threshold interaction
If the point estimate exceeds a safety cut-off but the interval spans both sides, defer or gather more evidence.
89. Uncertainty-aware action reduces brittle decisions near boundaries
The cost is more abstention or additional testing.
90. Worked case: house-energy prediction
A model predicts tomorrow’s electricity use at 24 kWh. Calibration residuals produce a 90% conformal interval of 19–29 kWh.
91. The interval is a future-outcome statement
Across exchangeable future households or days defined by the calibration setup, intervals should include outcomes at about the target frequency.
92. It is not a 90% posterior probability about this one day
The guarantee is repeated-frequency marginal coverage.
93. Worked case: medical length of stay
A base model predicts 4 days. Quantile regression predicts an asymmetric base interval of 2–8 days.
94. Conformalized quantile regression adjusts the interval
Calibration data reveal that upper-tail misses are larger than expected, so the final interval expands asymmetrically.
95. Efficiency reflects clinical usefulness
An interval of 1–30 days may cover well but be too broad for discharge planning.
96. Worked case: image classification
A model predicts bird species probabilities. A conformal set returns {sparrow, finch} for a difficult image.
97. The set preserves ambiguity instead of forcing one label
A human ornithologist can inspect the two plausible candidates.
98. A singleton set on easy images reduces review burden
Efficiency varies naturally with case difficulty when the score supports adaptive set size.
99. Worked case: learner mastery
A tutoring model predicts whether a learner can solve one of several unseen transfer tasks.
100. A prediction set can contain several likely error categories
For example, {mechanism explanation, question interpretation} rather than forcing one diagnostic label.
101. Wide diagnostic sets should trigger more observation
Ask another unseen task or oral explanation before deciding the intervention.
102. Educational use needs task-level definitions
Coverage of a diagnostic set should refer to a defined outcome on comparable tasks, not a permanent property of the learner.
103. Primary Science can learn conformal intuition through prediction bands
Predict how far a paper aeroplane will travel, then use past prediction errors to build a range for the next flight.
104. Primary 3 can compare point and range predictions
“It will travel 4 metres” versus “it will probably fall within a range built from our past errors.”
105. Primary 4 can count coverage
Across ten new flights, how often did the actual distance land inside the range?
106. Primary 5 can compare width
Two methods both cover nine of ten flights, but one uses much narrower ranges.
107. Primary 6 can study changing conditions
Calibrate indoors, then test outdoors in wind. Why might the old coverage stop working?
108. Secondary Science can formalise residual quantiles
Students can compute absolute residuals, sort them and construct split-conformal intervals.
109. Secondary Science can formalise exchangeability
They can identify data structures where random order is plausible and where time, groups or space break that assumption.
110. Secondary Science can compare marginal and subgroup coverage
A model may achieve 90% overall but 75% for one region and 97% for another.
111. Calibration should be performed after the base model is fixed
If the model is repeatedly changed after inspecting calibration residuals, the nominal split-conformal argument can be compromised.
112. Hyperparameter tuning belongs before the final conformal calibration split
Use training and internal validation to choose the model, then freeze it before estimating final nonconformity quantiles.
113. Alternatively, nested pipelines can preserve honesty
Every tuning and calibration decision must respect data separation.
114. Model selection creates another uncertainty layer
If the model family was selected from hundreds of options, the final conformal interval conditions on that search process unless the pipeline is designed carefully.
115. Conformal validity is not a magic eraser for selection bias
If calibration data influence feature selection or hyperparameters improperly, leakage can still undermine guarantees.
116. Group conformal methods target structured populations
Calibration can be stratified by predefined groups when sufficient data exist and the desired guarantee is group-specific.
117. Small groups create quantile instability
Protecting every subgroup separately can require much larger samples.
118. Mondrian conformal methods partition calibration cases
They calibrate within categories or groups to improve group-conditional behaviour under the chosen partition.
119. The partition must be chosen responsibly
Too many tiny groups make sets unstable; too few broad groups can hide local undercoverage.
120. Local conformal methods seek neighbourhood-specific uncertainty
They weight or select calibration cases similar to the target case.
121. Similarity becomes another modelling choice
Distance in raw variables, learned embeddings or domain-specific metrics can yield different local sets.
122. Locality can improve efficiency
Easy regions receive narrower sets while difficult regions widen appropriately.
123. Local methods usually weaken the simplicity of the classical guarantee
State clearly what remains guaranteed and under which assumptions.
124. Conformal risk control generalises the idea beyond simple coverage
Instead of controlling only whether the true label is in a set, methods can control other losses under suitable formulations.
125. Risk control connects prediction sets to decisions
The target might be false-negative rate, set loss or another operational error.
126. The chosen loss must reflect the real scientific job
A guarantee on the wrong metric can still produce unsafe decisions.
127. Conformal prediction under class imbalance needs care
Marginal coverage can be dominated by the majority class.
128. Class-conditional calibration can protect rare labels
But sufficient examples of the rare class are required to estimate reliable quantiles.
129. Rare labels create large sets
Limited calibration evidence forces conservative protection if coverage is demanded.
130. Regression tails are difficult too
Extreme outcomes may be underrepresented in calibration data.
131. Heavy-tailed residuals widen high-coverage intervals sharply
This is not necessarily a flaw; it may reflect genuine unpredictability.
132. Transformations can improve efficiency
Log-transforming positive skewed outcomes can make residual structure more stable, provided predictions are transformed back correctly.
133. Conformal prediction and uncertainty propagation differ
Propagation follows uncertainty through a known calculation. Conformal prediction empirically calibrates predictive error around a fitted model.
134. Conformal prediction and confidence intervals differ
Confidence intervals target parameters; conformal intervals target future outcomes.
135. Conformal prediction and Bayesian prediction differ
Bayesian prediction integrates a posterior predictive distribution; conformal prediction wraps empirical ranks around arbitrary predictors.
136. Conformal prediction and tolerance intervals are related but not identical
Both address future population coverage, but their constructions and assumptions differ.
137. AI systems can use conformal prediction for classification sets
An image model can return several plausible labels rather than one overconfident answer.
138. Retrieval systems can conformalise candidate sets
Instead of always returning top-k, a calibrated rule can vary the number of retrieved items to meet an empirical coverage target for relevant documents.
139. Language models can use conformal-style uncertainty wrappers
But defining the outcome, score and exchangeability unit is harder than in ordinary classification.
140. Free-form generation lacks a simple finite label space
Coverage of “the correct answer” may be ill-defined unless the task is structured.
141. Structured QA is easier
If answers come from a finite candidate set, conformal prediction can produce a set of plausible answers under a well-defined score.
142. Tool-use systems can conformalise routing choices
A set might include {search, calculator} when evidence does not justify one unique tool.
143. Large sets can trigger human review
This connects conformal prediction directly to selective prediction.
144. AI can help learners practise conformal reasoning
Useful prompts include: “Create twenty calibration residuals and form a 90% split-conformal interval,” “Show overall 90% coverage with subgroup undercoverage,” “Compare constant-width and locally scaled intervals,” and “Create classification prediction sets of different sizes.”
145. AI can overstate conformal guarantees
It may say “90% confidence for this person” when the correct statement is a marginal long-run coverage guarantee under stated assumptions.
146. Parents can use conformal thinking without statistics software
When predicting examination performance, do not give one guaranteed mark. Build a range from comparable unseen-paper errors and state the conditions clearly.
147. A wider range may be more honest
If performance varies widely across unseen tasks, pretending to know one exact score creates false confidence.
148. A narrowing range can show increasing stability
If unseen performance becomes consistently accurate, predictive intervals can shrink.
149. Range width is itself a learning diagnostic
A learner with high average performance but huge variance may need stability work rather than more content.
150. Small-group tuition can preserve calibration tasks
Use genuinely unseen questions to estimate how far actual performance deviates from tutor predictions.
151. Tutoring should not “train on the calibration set”
If the teacher coaches the exact holdout questions, they no longer measure future generalisation honestly.
152. Independent-attempt task 1: split conformal regression
Fit a simple predictor, compute ten calibration residuals, choose the conformal quantile and build an interval for one new prediction.
153. Independent-attempt task 2: marginal versus subgroup coverage
Create two groups whose combined coverage is 90% but whose group coverages differ strongly. Explain why the overall guarantee can hide local failure.
154. Independent-attempt task 3: efficiency
Compare two 90%-coverage methods with median interval widths of 5 and 20 units. What extra evidence is needed before preferring the narrower method?
155. Independent-attempt task 4: shift
Calibrate residuals in a low-noise environment, then double measurement noise at deployment. Predict what happens to coverage.
156. Independent-attempt task 5: classification sets
Create a five-class probability vector and design a nonconformity score that could produce a prediction set.
157. Independent-attempt task 6: action boundary
Use a conformal interval around a risk estimate and decide when the system should act, abstain or request more data.
158. Diagnostic error: marginal coverage described as individual certainty
Repair by stating the repeated-case coverage interpretation precisely.
159. Diagnostic error: exchangeability omitted
Repair by describing the population and conditions connecting calibration and future cases.
160. Diagnostic error: coverage reported without width
Repair by reporting efficiency alongside validity.
161. Diagnostic error: subgroup undercoverage ignored
Repair with prespecified group and local coverage audits.
162. Diagnostic error: calibration set reused for tuning
Repair by freezing model and scoring choices before final calibration.
163. Diagnostic error: time-series data randomly conformalised
Repair with time-aware methods or state that the ordinary exchangeability guarantee is not justified.
164. Diagnostic error: huge sets treated as success
Repair by evaluating efficiency and operational usefulness.
165. Diagnostic error: probability calibration confused with conformal coverage
Repair by separating probability meaning from set inclusion frequency.
166. Diagnostic error: distribution shift ignored after deployment
Repair with rolling coverage monitoring and revalidation.
167. Diagnostic error: conditional guarantee implied without proof
Repair by distinguishing marginal, group-conditional and local claims.
168. The independence test
Give a learner a base predictor, calibration residuals, a 90% coverage target, a rare subgroup and a shifted deployment period. Can they construct the set, state the guarantee correctly, identify the subgroup risk and know when the guarantee should no longer be trusted? That is transferable conformal reasoning.
169. The evidence boundary
Conformal prediction provides a rigorous way to control predictive coverage under explicit assumptions, most classically exchangeability. The standard guarantee is typically marginal rather than universal conditional coverage. Distribution shift, dependence, adaptive reuse of calibration data and unsupported subgroups can weaken or invalidate what may be claimed.
170. A compact conformal-prediction checklist
- What future outcome is being predicted?
- What base predictor is used?
- Which data fit the base predictor?
- Which data calibrate nonconformity?
- What nonconformity score is used?
- What coverage target is chosen?
- What finite-sample quantile rule is used?
- Is exchangeability plausible?
- Are there groups, time dependence or spatial dependence?
- Is the stated guarantee marginal or conditional?
- What is the empirical coverage on held-out data?
- What is interval width or set size?
- Does coverage hold across relevant subgroups?
- Does width adapt to case difficulty?
- Could calibration data have leaked into model tuning?
- Has the deployment distribution shifted?
- How will coverage be monitored over time?
- When should large sets trigger abstention or human review?
171. Frequently asked questions
What is conformal prediction?
It is a family of methods that wraps prediction algorithms with data-driven prediction sets or intervals designed to achieve a target coverage level under stated assumptions.
What is split conformal prediction?
It trains a base predictor on one subset and uses a separate calibration subset to estimate a quantile of prediction error or nonconformity.
What is a nonconformity score?
It measures how unusual an observed outcome is relative to a model prediction. Absolute residual is a common regression example.
What assumption matters most?
Classical finite-sample validity relies centrally on exchangeability of calibration and future examples under the chosen setup.
Does 90% coverage mean a 90% probability for one person?
Not in the ordinary frequentist conformal interpretation. It is a repeated-case coverage guarantee across comparable future examples.
Is the guarantee conditional on every subgroup?
Usually not. Classical conformal prediction primarily guarantees marginal coverage; subgroup and local behaviour need separate auditing or specialised methods.
How does conformal prediction help PSLE Science?
The formal mathematics is advanced, but the habit is accessible: predictions should include a range built from past errors, and the range must be checked on new cases.
How does it deepen in Secondary Science?
Students can connect residuals, ranks, quantiles, exchangeability, prediction intervals, classification sets and coverage auditing more formally.
172. Continue the Science Education Systems series
- How Scientific Probability Calibration Works
- How Scientific Selective Prediction Works
- How Scientific Distribution Shift Works
Conclusion: A useful uncertainty statement must say what it covers—and why
Maya sees the interval.
Jia Jun checks the calibration ranks.
Hana asks whether coverage survives across subgroups.
Ethan asks whether tomorrow is still exchangeable with yesterday.
Science needs all four.
Build the predictor.
reserve calibration evidence.
measure nonconformity.
choose coverage honestly.
report efficiency.
audit local failures.
Then let the prediction set say exactly what the evidence supports—no more and no less.
