Training Validity Is an Argument, Not a Sticker

It is tempting to treat validity as a property that a worksheet either possesses or lacks. Modern measurement thinking is more careful. Validity concerns the strength of the argument connecting an observed performance to a proposed interpretation and use.

The 2026 fifth edition of Educational Measurement places validity at the centre of testing, from defining the intended construct to evaluating consequences of score interpretations and uses. Its validity chapter describes validation as constructing and evaluating coherent arguments for and against proposed interpretations and uses. See Lane and Marion, 2026.

That framing is useful in tuition because the same score can support different claims of very different strength. A learner who gets 18/20 on a guided algebra worksheet has shown strong performance under that guidance. The score does not automatically establish independent method selection, transfer to a mixed paper, long-term retention or examination readiness.

The Validity Chain

1. Observation: What did the learner actually do?
2. Scoring: How was that response converted into a mark or judgment?
3. Interpretation: What capability do we infer from the observation?
4. Generalisation: How far beyond this particular task do we expect the inference to travel?
5. Use: What decision are we making because of the result?
6. Consequence: What happens if the interpretation is wrong?

Each link can fail independently. The answer can be scored correctly while the interpretation is too broad. The interpretation can be sensible while the decision is too consequential for the amount of evidence available. Good training validity asks where the weakest link in this chain sits.

Start With the Claim Before You Start With the Test

A weak assessment workflow starts by finding a worksheet and then inventing a meaning for the score. A stronger workflow begins with a claim.

• Mira can choose an appropriate method for a mixed set of quadratic equations without a method label.
• Jonas can make an evidence-bounded inference from a narrative passage without overstating certainty.
• Nadia can distinguish manipulated, responding and controlled variables in a fresh experimental context.
• A Primary 6 learner can select the correct percentage base when the surface story changes.
• A Secondary 4 learner can sustain accurate algebraic execution during the final third of a timed paper.

Once the claim is explicit, the tutor can design evidence that would genuinely support or weaken it.

The Claim Must Be Smaller Than the Evidence Can Support

Validity problems often begin with language that expands beyond the sample. “Weak in Mathematics” may come from four poor questions on algebraic translation. “Cannot do comprehension” may come from one unfamiliar passage. “Does not know Science” may come from one data-response task.

Use narrow claims first. If repeated evidence across different forms, contexts and times converges, the claim can gradually widen. Evidence should earn generalisation rather than being asked to carry it automatically.

The Five Main Sources of Validity Evidence

The 2025 Oxford chapter on current validity standards organises validity evidence around test content, response processes, internal structure, relations to other variables and consequences of testing. See Balkin and Juhnke, 2025.

Small-group tuition does not need formal psychometric studies for every diagnostic. But these categories provide an excellent discipline for deciding whether a classroom inference is credible.

1. Evidence Based on Content

Does the task represent the territory implied by the claim? If the claim is “mixed quadratic method selection,” the test should sample more than one method. If the claim is “English inference,” the items should not all be literal retrieval. If the claim is “scientific explanation,” the task should require explanation rather than only factual recall.

Content validity is not about having many questions. It is about sampling the right content and decision space.

2. Evidence Based on Response Processes

Did the learner use the reasoning process the task was intended to elicit? A correct answer reached by guessing does not support the same claim as a correct answer reached by valid reasoning. A memorised model answer does not necessarily demonstrate causal explanation. A multiple-choice answer chosen by test-wiseness may overstate the intended knowledge.

Think-alouds, short explanations, visible workings, confidence ratings and error traces can expose response process without turning every exercise into an interview.

3. Evidence Based on Internal Structure

In formal testing, internal structure examines whether patterns among items behave as the measurement model expects. Classroom tuition usually works at a smaller scale, but the principle remains useful.

If a supposed “method selection” form is dominated by computation speed, the structure of performance may reveal that the score is actually driven by arithmetic fluency. If every difficult item is also the wordiest item, reading load may be controlling the score.

Look for whether item patterns align with the intended capability rather than an accidental secondary demand.

4. Evidence Based on Relations to Other Variables

Does the new measure relate to other performances in sensible ways? If a learner shows strong algebraic method selection on a diagnostic, does that strength also appear in homework, classwork and mixed papers? If a reading inference score rises, does explanation quality improve in ordinary comprehension too?

Convergence across different sources strengthens the interpretation. Disagreement is not a nuisance to hide; it is a diagnostic signal asking what changed between conditions.

5. Evidence About Consequences

A score is rarely the end of the process. It may lead to more tuition, a harder class, a subject combination change, reduced support, examination acceleration or a conversation about capability.

The consequence determines how much evidence is enough. A low-stakes decision such as “practise one more set of pronoun-reference items” can be made from modest evidence. A high-stakes conclusion such as “this student should abandon Additional Mathematics” deserves much stronger and broader evidence.

Validity Is Fit for a Particular Interpretation and Use

A task can be excellent for one use and poor for another. A five-question micro-diagnostic may be valid for locating a likely weak link. It is not necessarily valid for ranking students across a cohort. A full examination may be valid for broad examination performance but inefficient for deciding whether one sign-control error has been repaired.

Ask, “Valid for what?” before asking whether a test is valid.

The Argument-Based Approach: Make the Assumptions Visible

A 2025 commentary on argument-based validity notes that the approach starts by stating the intended interpretation or use, identifying assumptions that must hold, and evaluating evidence for and against those assumptions. See Cook, 2025.

For training, this can be translated into a simple structure:

1. Claim: Mira can independently choose a quadratic method.
2. Assumption: the items actually require method choice rather than announce it.
3. Assumption: language does not create unrelated difficulty.
4. Assumption: scoring distinguishes method choice from execution.
5. Assumption: the sample contains enough variation to support the claim.
6. Evidence: performance across several fresh mixed items.
7. Counterevidence: failures when coefficients or representations change.
8. Decision: whether to progress, repair or test again.

Validity Is Plausibility, Not Proof

Educational measurement rarely provides mathematical proof that one interpretation is the only possible explanation. Validation strengthens or weakens the plausibility of an interpretation.

That means training language should remain calibrated. “The evidence suggests Mira’s main bottleneck is method recognition under mixed conditions” is more defensible than “We have proved Mira cannot recognise methods.” The first statement leaves room for new evidence. The second closes the model too early.

Alternative Explanations Are Part of the Validity Work

Every weak performance has competing explanations. A low algebra mark could arise from algebra, reading, careless copying, time pressure, unfamiliar notation, working-memory overload or a mixture.

Do not ask which explanation feels most intuitive. Design small contrasts that separate them.

• Remove dense wording but preserve the algebra.
• Keep wording stable but remove time pressure.
• Provide the equation but not the method.
• Ask the learner to explain the first decision before calculating.
• Repeat with a fresh parallel form.

Each contrast narrows the explanation space.

A Validity Contrast Is Often More Useful Than Another Full Test

If the question is whether reading load is blocking Mathematics, compare two tasks with the same mathematics and different language load. If the question is whether support is masking weakness, compare guided and independent attempts. If the question is whether memory is carrying the answer, use a fresh parallel form.

Good diagnosis changes one important condition at a time.

The Counterfactual Validity Question

Ask: if the learner truly possessed the target capability, what pattern should we expect across tasks? Then ask the reverse: if the learner lacked that capability, what pattern would we expect?

For example, if Jonas genuinely understands evidence-bounded inference, he should perform reasonably across different passages even when surface topics change. If he succeeds only when the passage resembles a practised text, memorised structure becomes a stronger alternative explanation.

The Negative-Control Idea for Training

Research uses controls to test whether an apparent relationship might arise for another reason. Tuition can borrow the logic informally.

If a student supposedly improved because of a new algebra strategy, include one task where that strategy should not apply. If the learner tries to use it everywhere, the training may have produced pattern overextension rather than discrimination.

The Positive-Control Idea for Training

Include at least one task the learner should be able to do if the intended mechanism is functioning. If that positive control also fails, the result may reflect attention, misunderstanding of instructions or a broader state problem rather than the narrow target alone.

Triangulation: Three Imperfect Views Can Beat One Perfect-Looking Score

A worksheet, a verbal explanation and a fresh application task each contain limitations. If all three point to the same bottleneck, the interpretation becomes stronger.

Triangulation does not mean collecting everything. It means using meaningfully different evidence sources that fail in different ways.

Convergent Evidence

Convergent evidence appears when several measures that should reflect the same capability show a similar pattern. Mira chooses methods well on a micro-diagnostic, mixed homework and a fresh mini-paper. That convergence supports the method-selection claim.

Discriminant Evidence

Discriminant evidence asks whether the measure is distinct from things it should not simply duplicate. If a supposed algebra diagnostic almost perfectly tracks reading-comprehension difficulty, the diagnostic may contain too much language load.

Incremental Evidence

Does the new measure add useful information beyond what is already known? If a five-minute diagnostic predicts exactly what the tutor can already see from ordinary work, it may not be worth administering. If it isolates the first weak link faster, it has practical value.

Validity and Effectiveness Are Related but Not Identical

A classroom assessment can support a valid interpretation yet still be inefficient, intrusive or useless for the decision at hand. Work on classroom assessment effectiveness argues that score use should be evaluated in relation to the problem the user is trying to solve. See Ferrara and Lai, 2021.

For tuition, ask two questions: “Does the evidence support the interpretation?” and “Does using this evidence help us make a better next decision?”

Worked Validity Contrasts: The Same Mark Can Mean Different Things

A score becomes diagnostically useful when the tutor can separate plausible explanations. The examples below are not miniature psychometric studies. They are disciplined contrasts that change one important condition and observe whether the learner’s performance changes with it.

Case 1 — Algebra or Reading?

Mira misses a word problem involving simultaneous equations. On the first attempt she cannot decide what to write. The tutor then gives only the two equations, without explaining how to solve them. Mira solves both accurately.

The original low mark remains real. But the strongest interpretation changes from “cannot solve simultaneous equations” to “has difficulty translating this verbal relationship into equations.” The repair should target representation before execution.

Case 2 — Algebra or Sign Control?

A learner performs well on equations until negative brackets appear. The tutor creates two near-parallel sets: one with identical equation structure but positive brackets, another with negative brackets. Performance collapses only in the second.

The valid claim is not “algebra is weak.” The bottleneck is sign control under distribution.

Case 3 — Method Knowledge or Method Selection?

Mira completes a full page of factorisation correctly. On a mixed page containing factorisation, completing the square and quadratic-formula opportunities, she chooses the wrong route repeatedly.

Blocked practice validly shows execution when the method is already implied. It does not establish independent method selection. A mixed form samples a different and more examination-relevant capability.

Case 4 — Calculation or Representation?

A learner can calculate gradient when two coordinates are supplied numerically but struggles when the same information is embedded in a graph. The computation is stable. The representation transition is not.

The training target becomes graph-to-coordinate extraction rather than relearning the gradient formula.

Case 5 — Percentage Knowledge or Base Selection?

A Primary 6 learner handles “20% of 80” easily but fails percentage increase and reverse-percentage questions. The arithmetic operation is not the main problem. The learner does not reliably identify which quantity is the base.

A validity-aware diagnostic varies the story while preserving the base-selection decision.

Case 6 — Geometry or Diagram Literacy?

A student knows angle facts when diagrams are clean and conventionally oriented but misses the same relationships when shapes are rotated or visually cluttered.

The concept may be known. Visual parsing and invariant recognition may be the weak link. A fair diagnostic uses several orientations before concluding that the angle relationship itself is absent.

Case 7 — English Inference or Vocabulary?

Jonas writes a weak inference after misunderstanding the word “reluctant.” The tutor defines the word without discussing the question. Jonas immediately identifies the character’s hesitation and supplies relevant evidence.

The initial response still shows a comprehension failure in authentic conditions. But the diagnostic interpretation narrows: vocabulary access blocked the inference process.

Case 8 — Inference or Claim Calibration?

Jonas consistently finds relevant evidence but writes conclusions stronger than the passage warrants. He is not failing to notice evidence. He is failing to control how far the conclusion travels beyond that evidence.

The training target becomes claim strength, not generic comprehension.

Case 9 — Grammar or Editing Attention?

A learner corrects subject–verb agreement accurately in isolated sentences but misses the same errors in their own composition. The rule is available when attention is directed toward it. It is not yet reliably monitored during broader writing.

Teaching the rule again may add little. Training self-monitoring and editing routines is more validly targeted.

Case 10 — Composition Quality or Topic Familiarity?

A student writes fluently about friendship but stalls on an unfamiliar public-transport prompt. Sentence control remains stable. Idea generation collapses.

One prompt cannot justify “weak composition.” Compare several prompts with different familiarity before deciding whether the bottleneck is planning, background knowledge, narrative structure or language.

Case 11 — Writing or Handwriting Speed?

A learner’s ideas and oral sentences are strong, but timed writing is incomplete. When allowed to type, the argument becomes longer and better organised.

The examination still requires handwritten performance if that is the official condition, so handwriting speed is not irrelevant to final performance. But it should not be confused with absence of ideas or argument structure.

Case 12 — Science Recall or Mechanism?

Nadia can recite a model explanation about heat transfer word for word. When the materials and arrangement change, she cannot predict the direction of energy transfer.

The original task measured recall successfully. It did not establish portable mechanism understanding. A fresh context is needed for the broader claim.

Case 13 — Scientific Reasoning or Apparatus Familiarity?

Nadia identifies variables correctly in familiar plant experiments but becomes confused by an unfamiliar cooling apparatus. When the diagram is simplified while preserving the same variable structure, performance recovers.

Representational novelty was contributing construct-irrelevant difficulty.

Case 14 — Data Interpretation or Arithmetic?

A learner can describe the pattern in a table verbally but makes arithmetic errors when calculating percentage change. If the target is evidence interpretation, those are different layers.

A diagnostic can temporarily provide the calculation to see whether the interpretation itself survives.

Case 15 — Experimental Design or Command Words?

A student knows that variables must be controlled but answers “state one variable kept constant” with a causal explanation. The science concept is present; answer-form control is not.

The valid training target may be command-word interpretation and response form rather than experimental design from scratch.

Case 16 — Vocabulary Recognition or Productive Use?

A learner scores 95% on multiple-choice meanings but cannot use the same words naturally in original sentences. Recognition evidence cannot support a broad claim of productive vocabulary mastery.

The response mode is part of the construct.

Case 17 — Knowledge or Retrieval Access?

A learner says “I know this” but cannot produce the definition unaided. After seeing the first letter, the full answer arrives immediately.

The cue reveals partial availability but also cue dependence. The correct interpretation is not “does not know” or “fully knows.” It is “knowledge is present but not yet independently retrievable under this condition.”

Case 18 — Knowledge or Recognition Familiarity?

A student consistently chooses the correct multiple-choice option but cannot explain why it is correct or why alternatives are wrong. The score may reflect genuine recognition knowledge, but it provides weaker evidence of generative understanding.

Case 19 — Exam Knowledge or Time Allocation?

A learner leaves twelve marks blank at the end of a paper but answers similar questions accurately when given separately. The examination score validly records total performance under time. It should not be interpreted as twelve marks of missing knowledge.

The weak link may be pacing, question selection or spending too long on early low-yield items.

Case 20 — Exam Knowledge or Reading Time?

A learner’s accuracy is strong after they understand a question, but they spend excessive time decoding dense wording. Under examination conditions, reading latency becomes part of performance.

For diagnosis, separate comprehension time from execution time. For final preparation, reintegrate them because both consume the same examination clock.

Case 21 — Performance or Fatigue?

A Secondary 4 learner scores 85% on the first half of a paper and 55% on the second. The later questions are not systematically harder. Repeated forms show the same late decline.

A global statement such as “content knowledge is 70%” hides a time-dependent performance state. Endurance or attention may be the relevant training target.

Case 22 — Performance or Anxiety?

A learner performs accurately in ordinary lessons but makes unusual first-step errors in formal timed tests. Anxiety may be one explanation, but it should not be assumed from the contrast alone.

Check whether the change is actually time pressure, unfamiliar environment, sleep, stakes, question order or another condition. “Anxiety” is a hypothesis, not a default explanation for every performance gap.

Case 23 — Performance or Calculator Fluency?

A learner selects the correct statistical method but loses marks through calculator entry and mode errors. If calculator use is part of the examination, this matters to examination performance. It is still diagnostically distinct from statistical reasoning.

Case 24 — No-Calculator Mathematics or Arithmetic Load?

Two no-calculator algebra items share the same structure, but one uses friendly factors and the other produces awkward fractions. If the target is algebraic structure, number friendliness can create construct-irrelevant variation.

If mental arithmetic is part of the intended examination demand, then the interpretation must include it rather than pretending the forms are structurally identical.

Case 25 — Optional Questions or Unequal Opportunity?

Two students earn the same mark after choosing different optional questions. One selected familiar topics; the other selected more difficult topics. The total score remains useful, but individual skill comparisons are weaker because the evidence samples differ.

Choice changes the interpretation.

Case 26 — Homework Performance or Support?

A student submits perfect homework. In class, they cannot begin similar items independently. The homework may validly show successful supported completion, but not independent mastery.

Ask what support was available: parent, tutor, answer key, AI, peer discussion or worked example. Provenance matters.

Case 27 — AI-Assisted Success or Learner Capability?

A learner uses an AI system to explain a difficult paragraph and then answers correctly. This can be valid evidence of tool-supported performance. It is weaker evidence of unaided comprehension.

The tool condition should be part of the interpretation, not hidden.

Case 28 — AI Agreement or Valid Scoring?

An automated marker gives almost the same grades as a human marker. Agreement alone does not establish that either is measuring the intended construct well. Both may reward surface features that correlate with quality without representing the full construct.

Validity requires examining what the scoring process attends to, not only whether two scorers agree.

Case 29 — Group Work or Individual Capability?

A project receives an excellent grade. One learner contributed the analysis; another designed the slides; a third presented. The group grade is valid evidence of group output. It is weaker evidence of each individual’s full capability.

Case 30 — Peer Assessment or Teacher-Level Judgment?

Peer feedback can reveal how classmates perceive clarity and persuasiveness. It should not automatically be treated as equivalent to expert scoring against a formal rubric.

Different evidence sources can be useful without being interchangeable.

Case 31 — One Bad Day or Stable Weakness?

A learner performs far below their ordinary range once. Before restructuring weeks of teaching, compare with recent work and obtain another fresh sample.

Single observations can be real and still be unrepresentative.

Case 32 — One Good Day or Stable Mastery?

The reverse problem is equally important. A perfect score after targeted revision may be a peak, not a stable state. Use delayed and fresh evidence before retiring the skill completely.

Case 33 — Improvement or Easier Retest?

Form A contains mixed, unfamiliar problems. Form B contains friendlier numbers, shorter wording and familiar contexts. The score rises sharply.

The learner may have improved, but form difficulty is a competing explanation. This is where Training Parallel Forms and comparability become part of validity evidence.

Case 34 — Improvement or Prompt Fading?

Accuracy remains at 70%, but prompts fall from four per task to zero. A raw-score interpretation says “no improvement.” A broader performance interpretation recognises increased independence.

Validity depends on whether independence was part of the intended construct.

Case 35 — Lower Score or Harder Transfer?

After successful repair on near-parallel forms, a learner scores lower in a novel representation. The lower mark may not indicate regression. It may reveal the boundary of transfer.

That is useful information if the interpretation names it correctly.

Case 36 — Mastery or Cue Leakage?

A worksheet is organised into sections labelled “Use the quadratic formula,” “Use factorisation” and “Complete the square.” Students score highly.

The worksheet validly measures execution under explicit method cues. It does not validly measure independent strategy selection.

Case 37 — Concept Knowledge or Answer-Pattern Recognition?

A multiple-choice learner notices that the longest option is often correct in a teacher-made set. Score rises without corresponding explanation quality.

Item-writing artefacts can become construct-irrelevant cues.

Case 38 — Scientific Explanation or Keyword Matching?

A student learns that certain key words are rewarded and inserts them without causal connection. Marks may rise if the rubric is keyword-heavy.

If the intended construct is mechanistic explanation, scoring must reward relationships, not isolated vocabulary alone.

Case 39 — Essay Knowledge or Memorised Template?

A student produces a polished paragraph whenever the topic matches a memorised template but cannot adapt the argument when the question changes direction.

Template use may be a useful support. It should not be mistaken for flexible argument construction until adaptation is demonstrated.

Case 40 — Oral Fluency or Topic Familiarity?

A learner speaks fluently about school life but hesitates on an unfamiliar social issue. Language fluency, idea availability and background knowledge interact.

Use several topics before deciding whether the weakness is oral language generally.

Case 41 — Learning or Test Familiarity?

A student repeats the exact diagnostic three times and improves each time. Some improvement may reflect genuine retrieval strengthening; some may reflect memory for item wording and route.

Use fresh forms to determine how much of the performance travels beyond the original items.

Case 42 — Poor Retention or Poor Initial Learning?

A learner forgets a method a week later. The immediate post-lesson test had been completed with heavy prompts. The issue may not be unusually rapid forgetting; the capability may never have been independently established in the first place.

Case 43 — Weak Transfer or Different Construct?

A tutor calls a radically different problem a “transfer test,” but the new task introduces unfamiliar concepts as well as a new context. Failure cannot be attributed cleanly to transfer distance.

Change enough to test portability, but not so much that a new construct becomes the main obstacle.

Case 44 — Slow Performance or Careful Performance?

A learner becomes slower immediately after learning a verification routine. Accuracy improves substantially. Calling the learner “less fluent” may be premature because deliberate checking temporarily increases time while reducing errors.

Case 45 — Fast Performance or Shallow Performance?

A learner answers quickly but with high-confidence mistakes. Speed alone is not evidence of mastery. Response time must be interpreted alongside accuracy, method and confidence.

The Pattern Across Cases

Validity improves when broad labels are replaced by bounded claims, competing explanations are tested rather than assumed, and the training target follows the evidence instead of the first impression.

A score tells you what happened on a task. Validity tells you how carefully you may travel from that event to a claim about the learner.

How to Design a Valid Training Measure

A valid training measure begins before the first question is written. The tutor needs a construct definition, a use case, a sampling plan and an interpretation rule. Without those, item writing can become a sequence of attractive questions with no clear argument connecting performance to the claim.

Step 1 — Name the Construct Precisely

“Mathematics” is not a useful construct for a five-minute diagnostic. “Select the correct algebraic method when several plausible methods compete” is. “English” is too broad. “Resolve pronoun reference across two competing antecedents” is testable.

A precise construct tells the tutor what must appear in the task and what must be controlled as irrelevant variation.

Step 2 — State the Intended Use

Will the result decide the next ten minutes of a lesson, the next month of training, or a major subject pathway? The same evidence can be sufficient for one decision and insufficient for another.

Make low-stakes decisions reversible. Require broader evidence for high-stakes decisions.

Step 3 — Define What Success Should Look Like

Do not wait until after the learner answers to decide what counts as competence. Define the performance indicators in advance.

• correct method choice;
• accurate execution;
• independent start;
• bounded explanation;
• appropriate use of evidence;
• reasonable completion time;
• successful verification.

The success definition should correspond to the intended construct rather than whatever is easiest to mark.

Step 4 — Define What Would Falsify the Claim

A strong diagnostic is not built only to confirm what the tutor expects. Ask what evidence would make the preferred interpretation less plausible.

If the claim is that Mira understands the quadratic structure, repeated failure when coefficients change should weaken that claim. If the claim is that Jonas understands inference, success only on one rehearsed genre should limit the claim.

Step 5 — Build a Content Map

List the important parts of the construct and decide which must be sampled. For method selection, the map may include recognising structure, comparing methods, choosing, executing and verifying. For comprehension inference, it may include evidence location, relation building, claim strength and explanation.

The content map prevents underrepresentation.

Step 6 — Identify Construct-Irrelevant Threats

Before writing the task, list things that could affect performance without being part of the intended construct.

• unnecessary reading complexity;
• unfamiliar diagrams;
• cultural assumptions;
• awkward arithmetic;
• poor formatting;
• hidden calculator demands;
• ambiguous pronouns;
• excessive memory load;
• unclear instructions;
• teacher cues.

Some of these may legitimately belong in final examination performance. The validity question is whether they belong in this diagnostic claim.

Step 7 — Choose the Response Mode Deliberately

Multiple choice, short answer, oral explanation, extended response and worked solution expose different evidence. The response mode should match the construct.

If the target is independent production, recognition items underrepresent it. If the target is discrimination among close concepts, carefully written multiple choice may be efficient. If reasoning process matters, visible working or explanation adds evidence.

Step 8 — Choose the Support Condition

Guided, prompted and independent performance are all educationally meaningful. They are not the same construct state.

Condition	What it can support	What it cannot establish alone
Fully guided	Can follow the process with explicit support	Independent initiation or selection
Method prompt	Can execute once route is identified	Independent method choice
General cue	Can recover with limited support	Fully independent access
Independent	Can perform under stated conditions unaided	Automatic transfer to different conditions

Step 9 — Set the Time Condition

Untimed work isolates capability differently from timed work. If the target is conceptual understanding, heavy time pressure may introduce speed as an irrelevant influence. If the target is examination readiness, speed is part of the performance system.

Use untimed diagnosis to locate the mechanism; timed retesting to examine performance under the real constraint.

Step 10 — Decide the Sampling Breadth

One item can locate a hypothesis. It cannot usually support a stable generalisation. Increase sample breadth when the claim or consequence grows.

A narrow first-weak-link check may need three to five items. A claim of durable examination readiness needs repeated evidence across larger and more varied samples.

Step 11 — Build Difficulty Around the Learner’s Range

A measure that is too easy cannot distinguish strong from very strong performance. A measure that is too hard collapses several weak states into the same floor score.

Use Training Responsiveness to make the measure sensitive around the capability range being trained.

Step 12 — Write the Scoring Rule Before Seeing the Answer

Post-hoc scoring invites bias. Define whether marks reward final answer, method, evidence, causal connection, precision, language or some combination.

If the construct is explanation, a rubric that rewards isolated keywords may produce weak validity even if marking is perfectly consistent.

Step 13 — Separate Accuracy From Process

A learner can be right for the wrong reason or wrong after using a strong method. Where diagnosis matters, record both outcome and process.

• final answer;
• method chosen;
• first wrong step;
• cue level;
• verification behaviour;
• confidence.

Step 14 — Use Fresh Evidence After Teaching

If the exact diagnostic becomes part of instruction, it loses independence as evidence of transfer. After repair, use a fresh approximately parallel task that preserves the capability demand while changing enough surface detail to require regeneration.

Step 15 — Check for Convergence

Compare the diagnostic with ordinary work, delayed performance and mixed tasks. When evidence agrees across meaningfully different sources, the interpretation strengthens.

Step 16 — Write the Interpretation in Bounded Language

Use “under these conditions,” “on this class of tasks,” “currently,” “with no prompts,” or “across three fresh samples” when those details matter.

Bounded language is not weakness. It makes the claim honest and testable.

Step 17 — Match the Decision to the Evidence

A narrow diagnostic should trigger a narrow response. If sign control is the likely weak link, repair sign control. Do not rebuild the entire algebra curriculum unless broader evidence supports that move.

Step 18 — Revalidate When Conditions Change

An interpretation that was valid under guided untimed conditions does not automatically remain valid under timed independent examination conditions. As the performance environment changes, obtain evidence again.

The Blueprint Table

Design question	Example answer
Construct	Independent quadratic method selection
Use	Decide whether to move from blocked to mixed practice
Content sample	Factorisation, completing square, formula
Response mode	Worked solution plus method label
Support	None
Time	Generous but bounded
Irrelevant threats	Dense language, extreme arithmetic
Scoring	Method choice separated from execution
Freshness	New coefficients and order
Interpretation	Performance on fresh mixed quadratics
Decision	Progress, repair or sample again

Validity and Reliability: Different Questions

Reliability asks whether measurement is sufficiently stable or consistent for its purpose. Validity asks whether the interpretation and use are supported. A measure can be consistently wrong about the intended construct.

Imagine a vocabulary test that always ranks students by reading speed because every item contains a long passage. Scores might be stable. If the intended claim is vocabulary knowledge independent of reading speed, stability does not rescue the interpretation.

Validity and Comparability: Meaning Versus Change

Validity asks what the score means. Comparability asks whether scores from different forms or occasions can be compared fairly enough for the intended conclusion.

To claim improvement, both matter. Form A and Form B must represent the intended construct and remain comparable enough that the difference is not mainly a form effect.

Validity and Responsiveness: Meaning Versus Detectability

A measure can target the correct construct yet be too blunt to detect improvement. If all items are far above or below the learner’s current range, score movement may not reflect the underlying change well.

Validity and responsiveness work together when training needs to detect small, meaningful repair.

Validity and Measurement Noise

Noise weakens the inference by increasing uncertainty. Attention, item luck, marking variation and ordinary performance fluctuation can all move a score.

Do not demand zero noise. Use repeated samples, multiple signals and appropriate caution so the conclusion remains stronger than the random variation around it.

Validity and Baselines

A baseline is only useful if it measures the capability relevant to the future comparison. A poorly targeted baseline can make every later gain calculation look precise while answering the wrong question.

Define the construct before collecting the baseline.

Validity and Parallel Forms

Parallel forms help defend against exact-item memory. But parallelism does not automatically guarantee validity. Two forms can be equally comparable while both underrepresent the construct or include the same irrelevant difficulty.

Use parallel forms as one piece of a validity argument, not as a substitute for it.

Validity and Anchor Tasks

Anchor tasks preserve a stable reference over time. They are useful for observing change on a known task, but repeated familiarity can alter what performance means.

Combine anchors with fresh confirmation when independent capability matters.

Validity and Transfer

Near-parallel performance supports a narrower claim than far transfer. Do not infer broad transfer from local success. Increase distance deliberately and describe exactly what changed.

Validity and Retention

Immediate success supports immediate accessibility. It does not establish retention over weeks. If durability matters, delayed evidence belongs in the validity argument.

Validity and Independence

Supported performance can be valuable evidence of learning potential or scaffolded capability. Independent performance is a different claim. Always record whether prompts, examples, hints or tools were present.

Validity and Examination Readiness

Examination readiness is a compound construct. It can include content knowledge, method selection, interpretation, pacing, endurance, checking, answer-form control and emotional regulation under stakes.

A short topic worksheet can validly measure one component. It cannot alone support the full examination-readiness claim.

Validity and Accommodations

An accommodation changes access conditions. The validity question is whether it reduces irrelevant barriers while preserving the intended construct.

For example, simplifying decorative layout may remove visual clutter without changing mathematical reasoning. Reading a question aloud might be appropriate if reading is irrelevant to the intended construct, but inappropriate if reading comprehension is precisely what is being assessed.

Fairness Is Part of Valid Interpretation

A task can disadvantage some learners for reasons unrelated to the intended capability. Unnecessary cultural references, inaccessible language, ambiguous formatting or prior exposure can distort performance.

Fairness does not mean making every task easy. It means preventing irrelevant barriers from masquerading as the construct.

A 2025 Learning-Study Example: Validation Must Be Discussed, Not Assumed

A 2025 paper on knowledge tests in learning studies examines test development, pilot testing and interpretation in relation to the intended object of learning, emphasising the consistency and fragility of assessment instruments and the need for explicit limitations. See Validity of interpretation and use of knowledge tests in a learning study.

The classroom lesson is straightforward: a locally designed test can be useful, but its limits belong inside the conclusion rather than outside it.

Thirty Validity Failure Modes in Training

Validity failures are often ordinary design mistakes rather than dramatic misconduct. A worksheet can be well intentioned, neatly marked and still support an interpretation that is too broad, too confident or pointed at the wrong mechanism.

Failure Mode 1 — The Construct Is Never Defined

The tutor writes “test algebra” and then mixes manipulation, graph reading, word problems and arithmetic. A low total score cannot tell which capability failed.

Repair: name the capability and build the sample around it.

Failure Mode 2 — The Claim Is Broader Than the Sample

Five literal comprehension items become “English comprehension mastery.” One chapter test becomes “strong at Mathematics.”

Repair: keep the conclusion no broader than the sampled territory.

Failure Mode 3 — Difficulty Is Mistaken for Depth

The task is made harder by longer wording, uglier arithmetic or more visual clutter. The tutor then assumes the harder item measures deeper thinking.

Repair: make difficulty serve the construct, not noise around it.

Failure Mode 4 — Easy Performance Is Mistaken for Validity

Students score highly and everyone feels the worksheet worked. But the method was cued, distractors were weak or the questions were near copies of practice.

Repair: ask what process generated the success.

Failure Mode 5 — One Item Carries the Entire Diagnosis

A single wrong answer becomes a stable learner label.

Repair: treat one item as a hypothesis generator; sample again before generalising.

Failure Mode 6 — One Perfect Score Carries the Entire Mastery Claim

A strong post-lesson result is treated as proof of durable independent mastery.

Repair: confirm with fresh, delayed and mixed evidence when the claim requires it.

Failure Mode 7 — The Response Process Is Invisible

The tutor records only right or wrong and cannot distinguish guessing, memorisation, valid reasoning or copied procedure.

Repair: capture workings or brief explanations when process matters.

Failure Mode 8 — Scoring Rewards the Wrong Evidence

A science explanation rubric gives marks for isolated keywords even when causal relationships are missing. A writing rubric rewards length more than coherence.

Repair: make scoring represent the construct, not easily counted proxies.

Failure Mode 9 — Support Is Hidden

Homework is recorded as 100%, but answers were completed with examples, hints or AI assistance.

Repair: record support condition and interpret the performance accordingly.

Failure Mode 10 — Timing Changes but Interpretation Does Not

An untimed baseline is compared with a timed retest as if score change reflected learning alone.

Repair: preserve time conditions for comparison or explicitly include timing in the new construct.

Failure Mode 11 — Form Difficulty Changes Silently

The second worksheet has shorter wording and friendlier numbers. Improvement is attributed entirely to training.

Repair: use approximately parallel forms and audit difficulty levers.

Failure Mode 12 — Exact Item Repetition Inflates Evidence

The student remembers the route, wording or answer.

Repair: use fresh forms when independent regeneration is the claim.

Failure Mode 13 — The Measure Has a Ceiling

The learner starts at 95%; later improvement in speed, explanation or independence cannot appear in the score.

Repair: raise resolution around the learner’s current range.

Failure Mode 14 — The Measure Has a Floor

The learner begins at 5%. Early progress from “cannot start” to “can choose the method” remains invisible in total marks.

Repair: score partial capability or use a more accessible diagnostic.

Failure Mode 15 — The Tutor Confuses Outcome With Mechanism

A low score leads to more of the same practice without identifying where the process breaks.

Repair: locate the first weak link before prescribing volume.

Failure Mode 16 — The Tutor Confuses Correlation With Cause

Students who take more practice papers also score higher, so the tutor concludes more papers caused the improvement. Stronger students may simply choose to practise more.

Repair: be cautious about causal claims from ordinary observational patterns.

Failure Mode 17 — The Tutor Confuses Reliability With Validity

A worksheet gives consistent scores, so it is assumed to measure the intended construct.

Repair: stability is necessary for many uses but does not establish meaning.

Failure Mode 18 — The Tutor Confuses Authenticity With Validity

A full paper looks like the real exam, so it is used to diagnose one small mechanism.

Repair: authentic tasks for integrated performance; controlled tasks for mechanism diagnosis.

Failure Mode 19 — The Tutor Confuses Familiarity With Fluency

The learner becomes fast on one repeated worksheet. Performance is called automatic.

Repair: check speed on fresh tasks from the same family.

Failure Mode 20 — The Tutor Confuses Confidence With Competence

A learner sounds certain and answers quickly, so the tutor assumes strong knowledge.

Repair: compare confidence with evidence, especially high-confidence errors.

Failure Mode 21 — The Tutor Confuses Low Confidence With Weak Knowledge

A cautious learner produces correct, well-reasoned answers but rates confidence low.

Repair: train calibration rather than reteaching content automatically.

Failure Mode 22 — The Wrong Comparison Group Is Used

A learner is judged weak because they are compared with a much more advanced class rather than the relevant curriculum expectation or their own growth trajectory.

Repair: choose the reference appropriate to the decision.

Failure Mode 23 — Growth Scores Are Treated as Error-Free

Pretest 61, post-test 67 becomes “six marks of learning.” Both scores contain measurement noise and form effects.

Repair: interpret gain alongside comparability, repeatability and other signals.

Failure Mode 24 — Averages Erase Conditions

Average score is 72%, but performance is 88% untimed and 56% under realistic pacing.

Repair: preserve the condition dimension when it matters to the claim.

Failure Mode 25 — A Composite Score Hides Compensating Weaknesses

Strong calculation compensates for poor interpretation. Total score looks average and both mechanisms disappear.

Repair: inspect component profiles where the construct is multidimensional.

Failure Mode 26 — The Measure Becomes the Curriculum

Everything taught is narrowed to what the diagnostic samples. Students become excellent at the gauge but weak beyond it.

Repair: teach broadly; sample strategically; test transfer.

Failure Mode 27 — Parent Language Outruns Evidence

“Your child is careless,” “has no foundation,” or “cannot cope” is said after a narrow sample.

Repair: report observed patterns, conditions and uncertainty before labels.

Failure Mode 28 — Improvement Is Attributed to the Latest Intervention

The learner improves after tuition, school revision, extra sleep and several practice papers. The newest technique receives all the credit.

Repair: distinguish “improvement occurred during this period” from stronger causal attribution.

Failure Mode 29 — AI Output Is Treated as Direct Evidence of Learner Knowledge

A polished answer produced through AI assistance is scored as if the learner generated the reasoning independently.

Repair: distinguish tool-supported performance from independent performance.

Failure Mode 30 — The Decision Is More Certain Than the Evidence

A small, noisy, narrow sample triggers a major irreversible decision.

Repair: increase evidence in proportion to consequence and prefer reversible actions under uncertainty.

Decision Validity: The Cost of Being Wrong

Training decisions have asymmetric costs. Keeping a learner on one extra week of sign-control practice may cost time. Misdiagnosing the learner as globally weak in Mathematics can change confidence, subject choices and expectations.

Decision validity therefore asks not only “Is the interpretation plausible?” but “Is it plausible enough for this action?”

A Reversibility Ladder for Educational Decisions

Decision	Reversibility	Evidence needed
Try one extra practice set	Very high	Small local signal may be enough
Spend two lessons on a repair	High	Repeated local evidence
Change revision plan for a month	Moderate	Several converging samples
Move class level	Lower	Broad repeated evidence
Abandon a subject pathway	Low	Strong multi-source evidence and careful review

Uncertainty is easier to manage when the first response is reversible.

Use Decision Thresholds, Not Labels

Instead of “weak” and “strong,” define what evidence changes the next action. For example: if Mira selects the correct method on at least four of five fresh mixed items independently across two sessions, move to timed mixed practice. If not, keep method-discrimination training active.

The threshold turns assessment into a decision rule rather than a personality judgment.

Do Not Confuse a Threshold With Truth

A threshold is a practical boundary for action, not a magical point where the learner changes identity. A score of four out of five and three out of five may not represent fundamentally different ability states.

Use thresholds to organise action while retaining uncertainty around the boundary.

False Positives and False Negatives in Training

A false positive occurs when we conclude a skill is mastered when it is not. A false negative occurs when we conclude a weakness is present when the learner actually possesses the skill but the task or condition hid it.

The preferred balance depends on consequence. Before retiring a crucial prerequisite, avoid false mastery. Before imposing months of remediation, avoid falsely diagnosing a weakness.

Validity and Opportunity to Learn

A low score can be a poor basis for judgment if the learner was never adequately taught the sampled content. Assessment evidence should be interpreted in relation to opportunity to learn.

This does not erase the performance gap. It changes what the gap says about the learner versus the learning history.

Validity and Language Background

When subject knowledge is assessed through complex language, linguistic access can influence performance. Sometimes that language is legitimately part of the subject demand. Sometimes it is avoidable load.

The diagnostic should decide which role language is playing before interpreting the score.

Validity and Cultural Familiarity

Examples and contexts can privilege learners familiar with particular experiences. If background knowledge is not the construct, choose contexts that do not make cultural familiarity the hidden gate.

Validity and Test Security

If items circulate beforehand, the meaning of performance changes. The score may remain accurate as a count of correct answers, but it provides weaker evidence of fresh capability.

Retire compromised items from independent measurement and move them into practice.

Validity and Marking Consistency

If the rubric is valid but marking changes substantially between tutors or days, interpretation becomes unstable. Calibration supports validity by reducing scoring noise, but agreement alone is not enough. The rubric must still represent the intended construct.

Validity and Holistic Versus Analytic Scoring

A holistic composition mark can represent overall quality efficiently. An analytic rubric can expose component strengths such as organisation, language, evidence and task fulfilment. Neither is universally superior.

Choose the scoring mode that supports the intended decision. Use analytic evidence for repair; holistic evidence for integrated performance where appropriate.

Validity and Automated Scoring

Automated scoring can be useful for speed and scale. Validity still requires evidence that the features used by the scoring system support the intended interpretation. High human agreement can coexist with shared blind spots.

For high-consequence use, audit what the system rewards, where it fails and whether students can game surface patterns.

Training Validity Across the Learning Journey

The same validity principles apply from early Primary to Junior College, but the form of the evidence changes. Younger learners need smaller tasks and clearer observation. Older learners can handle more integrated tasks, more explicit self-monitoring and more realistic examination conditions.

Primary 1–2: Observe the Relationship Before the Score

For young learners, a total mark can hide the most useful information. Watch how the child begins, what they attend to, whether they understand the instruction and where the first breakdown appears.

If a child fails a subtraction task because the wording is misunderstood, reteaching subtraction alone misses the access problem. If the child can solve with counters but not symbols, representation transition may be the next target.

Validity at this stage often comes from close observation plus a small number of carefully varied examples.

Primary 3–4: Separate Rule Knowledge From Application

Learners can increasingly explain what rule they know and where they used it. A child may recall a multiplication fact yet fail a word problem because the operation is not recognised.

Use pairs: one direct item, one contextual item. The contrast reveals whether the bottleneck is fact retrieval, operation selection or language interpretation.

Primary 5–6: Build Evidence That Survives PSLE Variation

Upper-primary training needs stronger variation. A learner should demonstrate a relationship across different wording, diagrams and contexts. Success on one familiar template should not be inflated into broad mastery.

Use near-parallel forms first, then mixed transfer. For PSLE English, separate reading access from inference and answer-form control. For Mathematics, separate representation, method choice and execution. For Science, separate factual recall, mechanism, evidence interpretation and command-word response.

Secondary 1–2: Test Method Selection

As the curriculum grows, the learner faces competing procedures. A blocked worksheet can validly measure execution but may not measure selection. Mixed forms become increasingly important.

Validity claims should distinguish “can perform method X when prompted” from “can identify when method X is appropriate among alternatives.”

Secondary 3–4: Integrate Examination Conditions Carefully

Upper-Secondary performance is shaped by timing, endurance, topic switching, calculator use, answer formats and checking. Those conditions should gradually enter the assessment argument.

Use controlled diagnostics to repair mechanisms, then full or half-paper conditions to test whether the repair survives the integrated performance environment.

Junior College: Validate Models, Assumptions and Evidence Use

JC tasks often require more than selecting a formula. Learners may need to define assumptions, compare models, interpret uncertainty, construct arguments and evaluate limitations.

A routine calculation cannot support a broad claim of model-based reasoning. The assessment needs to sample the reasoning process itself.

The Three-Student Small-Group Advantage

A three-student class creates a useful validity advantage: the tutor can watch process, not only collect outcomes. The same task may expose three different bottlenecks.

• Student A identifies the method but makes execution errors.
• Student B executes perfectly after a cue but cannot select independently.
• Student C selects and executes but loses control under time.

The shared worksheet produces different valid interpretations because the response process differs.

Do Not Force Three Students Into One Diagnosis

Small-group teaching becomes weak when one class average drives the next lesson. Use the common content as a shared surface, then track the first weak link individually.

The Tutor Observation Grid

Signal	What to watch
Task access	Understands instruction and representation
Recognition	Identifies what kind of problem this is
Selection	Chooses a valid route
Execution	Carries out the route accurately
Monitoring	Notices uncertainty and checks appropriately
Support	Level of cue needed
Speed	Latency to first meaningful step and completion
Transfer	Performance when surface changes

The One-Lesson Validity Loop

1. Read: inspect ordinary student work.
2. Hypothesise: name the likely first weak link.
3. Contrast: change one condition to separate alternatives.
4. Repair: teach the narrow mechanism.
5. Practise: vary the surface while preserving the relationship.
6. Fresh retest: sample with a new task.
7. Interpret: bound the claim to the evidence.
8. Decide: continue, progress or test again.

The One-Week Validity Loop

A weekly repair cycle can separate immediate learning from durability.

• Early week: baseline or ordinary-work signal.
• Middle: targeted repair and varied practice.
• Late week: fresh near-parallel check.
• Following week: delayed sample or mixed transfer.

The exact schedule is flexible. The validity principle is that different tests answer different questions.

The Monthly Validity Review

Once a month, review whether active measures are still doing useful work.

• Which measures have reached ceiling?
• Which forms are now overfamiliar?
• Which constructs were defined too broadly?
• Which diagnostics never changed teaching?
• Which measures predict ordinary performance well?
• Which learner states now need a harder or more authentic condition?

The Term Review: Stop Measuring What Is Already Stable

A diagnostic should have an exit condition. When a capability is repeatedly independent, durable and transferable, stop measuring it so frequently. Redirect attention to the current weak links.

Measurement is valuable because it guides action, not because more measurement is always better.

The Validity Ledger

For important training targets, keep a compact record rather than relying on memory.

Field	Example
Target claim	Independent percentage-base selection
Evidence source	Fresh 5-item mixed micro-form
Condition	Independent, untimed
Result	4/5
Main error	Reverse percentage
Alternative explanation	Language load low
Confidence	Moderate
Next action	One reverse-percentage contrast, then delayed check

Evidence Confidence: Green, Amber, Red

A simple confidence code can help tutors and parents avoid overclaiming.

• Green: repeated fresh evidence, conditions known, alternative explanations weak, decision low risk or well supported.
• Amber: promising pattern but limited sample, mixed evidence or one unresolved alternative explanation.
• Red: single noisy observation, unknown support, unclear construct or high-stakes decision unsupported by current evidence.

The colour describes confidence in the interpretation, not the child.

A Parent Conversation: Report the Evidence Before the Label

Instead of saying “She is careless in Mathematics,” say: “Across two fresh mixed sets, she chose the correct method on most questions but lost marks through sign errors after negative brackets. When the brackets were positive, accuracy returned. We are treating sign control as the current weak link and will retest it next week.”

This is more informative, more testable and less likely to turn one pattern into a permanent identity.

A Parent Conversation: Describe Support Honestly

“He can complete the task with a general prompt but is not yet starting independently” is better than either “he can do it” or “he cannot do it.” Support level is part of the state.

A Parent Conversation: Separate Progress From Prediction

A learner can show genuine improvement without the evidence yet being strong enough to predict an examination grade. Say what improved, then say what still needs to be tested under examination conditions.

A Student Conversation: Teach the Learner to Interpret Their Own Evidence

Older students can learn to say: “I know the formula, but I choose it too slowly when several methods compete,” or “I can find evidence, but I overstate my conclusion,” or “I am accurate for forty minutes and then sign errors rise.”

This is metacognitive validity: the student’s internal model begins to match the observed evidence.

The Independence Test

After a repair, reduce prompts. Can the learner identify, select, execute and verify without intervention? If not, support may be carrying part of the performance.

The Durability Test

Wait long enough for immediate familiarity to fade. Then use a fresh sample. Durable learning should remain accessible beyond the lesson that created it.

The Transfer Test

Change the surface, representation or context while preserving the underlying principle. Record what changed so a failure can be interpreted rather than merely observed.

The Stress Test

Once a capability is stable in controlled conditions, add realistic difficulty: time pressure, mixed topics, longer duration or competing methods. A stress test is not a fair first diagnostic; it is a later test of performance robustness.

The Four-Proof Rule

1. Freshness: succeeds on a new task.
2. Independence: succeeds with reduced support.
3. Durability: succeeds after delay.
4. Transfer: succeeds when surface changes meaningfully.

Not every small skill needs all four proofs before moving on. But the broader the mastery claim, the more of this evidence should exist.

When to Stop a Repair

Stop active repair when repeated evidence shows the original weak link is no longer the limiting factor. Continuing the same intervention after the bottleneck moves wastes time and can create boredom or overpractice.

When to Escalate a Repair

Escalate when the same narrow weakness persists across fresh forms, different explanations have been ruled out, support does not fade and the weakness blocks higher-level work.

Escalation may mean more explicit instruction, prerequisite rebuilding, increased practice or a different representation—not automatically more of the same worksheet.

When to Abandon the Original Hypothesis

If the predicted contrast does not appear, update the model. A tutor who believed reading load was the problem should change course if simplifying the language produces no improvement.

Good validity practice makes hypotheses disposable.

Do Not Make Students Carry Old Diagnoses

A label that was once accurate can become stale. “Weak at fractions,” “careless,” “slow reader” or “needs prompting” should be re-evaluated when new evidence changes the state.

The learner model should update when the learner does.

Validity Is Dynamic Because Learning Is Dynamic

A diagnostic interpretation is a snapshot under conditions. Training changes the learner, and changing the learner changes which tasks remain informative.

Measures should evolve from foundational access to independent performance to transfer and stress conditions as capability grows.

Advanced Validity: Tools, Drift, Causality and the Limits of a Score

Training evidence becomes harder to interpret as the learning environment becomes richer. Students use calculators, formula sheets, search, worked examples, peers, teachers and increasingly AI tools. Curriculum changes. Learners grow. Tasks drift. The same score can therefore mean different things under different conditions.

The solution is not to remove every support. It is to define the performance condition honestly and make the claim match it.

Unaided Capability and Tool-Supported Capability Are Different Constructs

A student who can solve a problem with a formula sheet has demonstrated something valuable. A student who can retrieve and apply the formula without the sheet has demonstrated something different. A student who can use an AI explanation to recover the method has demonstrated tool-supported problem solving. None of these states should be dismissed, and none should be silently substituted for another.

Before interpreting a result, name the condition: unaided, calculator-supported, formula-supported, teacher-prompted, peer-supported, search-supported or AI-supported.

The Tool-Condition Matrix

Condition	Strongest defensible claim	What still needs separate evidence
Unaided	Can perform independently under stated conditions	Transfer, durability, performance under other constraints
Calculator-supported	Can solve with calculator access	Mental or no-calculator execution
Formula-sheet supported	Can select and use supplied formulas	Independent retrieval
Teacher-prompted	Can recover with instructional support	Independent initiation and selection
AI-supported	Can use an AI tool to reach a valid solution under observed conditions	Unaided knowledge, source evaluation, independent verification

AI Assistance Changes the Evidence Path

If an AI system supplies the interpretation, method or wording, the final answer is no longer direct evidence of the learner generating all of those components independently. The learner may still be demonstrating valuable skills: asking a good question, selecting useful output, checking a proposed route, integrating information or identifying an error.

Those skills should be named rather than hidden inside an unaided mastery claim.

The AI Provenance Rule

When AI materially contributes to an answer used for assessment, record what the tool contributed.

• Did it define a term?
• Did it suggest the method?
• Did it generate the final prose?
• Did it check an answer?
• Did it provide a worked example?
• Did the learner reject or correct any AI output?

Provenance turns a vague “used AI” condition into interpretable evidence.

AI Verification Can Be a Construct in Its Own Right

One useful modern capability is not merely producing an answer but evaluating whether a generated answer is trustworthy. A learner who detects a false assumption, checks a calculation against first principles and finds the original source is demonstrating verification skill.

If verification is the intended construct, the assessment should deliberately include plausible but imperfect outputs and score the learner’s checking process.

Human-Like AI Scoring Is Not Automatically Valid Scoring

An automated marker can agree closely with human marks while still rewarding the wrong features. Humans can also share the same blind spots. Agreement is evidence about consistency between scorers, not a complete validity argument.

For consequential uses, inspect which response features drive the score, whether atypical but strong answers are treated fairly, whether superficial patterns can game the model and whether the scoring rule represents the intended capability.

The Same Task Under Different Tool Conditions Is Not the Same Measurement

A learner completes Form A without a calculator and Form B with one. Another completes Form A independently and Form B with an AI hint. Even if the questions are parallel, the administrations are not fully comparable.

When tool conditions change, interpret the change in performance jointly with the change in support.

Construct Drift: When the Thing Being Measured Quietly Changes

A training programme may begin by measuring arithmetic accuracy and later begin valuing method selection, explanation and checking without updating the score. The construct has drifted while the label stayed the same.

Review construct definitions periodically. If the educational goal changes, change the measure and version the interpretation.

Curriculum Drift

A diagnostic built for one syllabus can become less representative after curriculum emphasis changes. Even within one school year, teaching may move from routine execution toward mixed application and evaluation.

A historical score remains a valid record of what was measured then. It may no longer be a good basis for today’s broader claim.

Form Drift

Tutors often revise worksheets gradually. One difficult item is removed, wording is simplified, a hint is added, then the same form name is reused. Months later, scores are compared as if the measure never changed.

Version forms when substantive changes occur. Keep enough provenance that a future comparison can tell which form was actually administered.

Support Drift

A tutor who knows a learner well may begin giving subtle cues unconsciously: a pause, a gesture, a leading question or a familiar sequence of prompts. Performance can improve partly because the support environment becomes better calibrated.

For important independence checks, standardise the support condition and deliberately reduce cueing.

Learner Drift Is Usually Growth

The learner also changes. A form that was diagnostic in January may become trivial by September. The item has not failed; its job may be finished.

Raise the measurement range when the old form reaches ceiling. Do not keep using an obsolete gauge simply because it creates a comforting high score.

Baseline Invalidation

If the target construct changes substantially, the old baseline may no longer anchor the new claim. A baseline for routine calculation cannot be treated as the baseline for mixed-topic method selection merely because both sit under the Mathematics label.

Create a new baseline when the measurement question changes materially.

Causal Attribution: Improvement Does Not Prove What Caused It

A learner improves after six weeks of tuition. During the same six weeks, school revision intensified, the learner completed more homework, sleep improved and examination familiarity increased. The improvement is real. Attributing all of it to one intervention is a stronger claim.

Use restrained language unless the design genuinely isolates cause.

Better Causal Language for Tuition

• “Performance improved during the repair period.”
• “The improvement is consistent with the targeted intervention helping.”
• “The repeated error declined after focused sign-control practice.”
• “We cannot attribute the entire score gain to one factor because several conditions changed.”

This language remains useful without claiming more than the evidence warrants.

Regression to the Mean

An unusually poor result is often followed by a less extreme result even without a powerful intervention. Likewise, an unusually excellent score may fall toward the learner’s ordinary range.

When training is triggered by one extreme score, obtain another sample before attributing all rebound to the intervention.

Maturation and Ordinary Development

Children and adolescents change over time. Reading fluency, working knowledge, confidence and general school experience can improve independently of one targeted programme.

Long-term growth claims should acknowledge that multiple developmental processes may be contributing.

Concurrent Interventions

When school, home and tuition all change simultaneously, isolate what can be isolated and avoid pretending the rest is known. One practical approach is to track the specific error mechanism each intervention is expected to affect.

If sign errors fall specifically after sign-control work while unrelated error types remain stable, that pattern strengthens the targeted explanation without proving exclusive causation.

Evidence Sufficiency: How Much Is Enough?

There is no universal number of questions or administrations that makes an interpretation valid. Sufficiency depends on claim breadth, task variability, measurement noise and decision consequence.

A useful practical rule is: stop collecting when additional comparable evidence is unlikely to change the next decision. If each fresh sample keeps changing the diagnosis, the evidence is not yet saturated.

Evidence Saturation

Suppose five different fresh items, two lessons and one mixed-paper example all point to the same weakness: the learner chooses the wrong percentage base. Another identical micro-test may add little. It is time to repair.

After repair, evidence saturation resets because the learner state has changed.

The “What Would Change My Mind?” Test

Before declaring a diagnosis, ask what evidence would make you revise it. If the answer is “nothing,” the process is no longer diagnostic.

For a reading-load hypothesis, strong performance should improve when language is simplified while the target mathematics stays constant. If it does not, the hypothesis should weaken.

Discrepancy Analysis: When Measures Disagree

Disagreement between evidence sources is often the most informative event in the system. A student may score strongly on homework and weakly in class, strongly untimed and weakly timed, strongly on recognition and weakly on production.

Do not average the disagreement away. Ask what condition differs and whether that difference points to the hidden construct.

A Discrepancy Table

Pattern	Possible interpretation to test
Homework high, independent classwork low	Support dependence
Untimed high, timed low	Pacing, retrieval speed or endurance
Recognition high, production low	Retrieval depth or generative use
Blocked high, mixed low	Method selection
Immediate high, delayed low	Durability
Near transfer high, far transfer low	Representation or generalisation boundary
Oral explanation high, written answer low	Writing or answer-form constraint

Validity Debt

When a programme repeatedly uses vague measures, unclear support conditions and unversioned forms, interpretation debt accumulates. Later, nobody can tell whether improvement reflects learning, easier forms, altered marking or increased familiarity.

Validity debt is reduced by simple habits: name the construct, version forms, record conditions, separate outcome from process and write bounded claims.

The Consequence Audit

Before acting on a score, ask:

1. What action will this result trigger?
2. How reversible is that action?
3. What is the cost of a false positive?
4. What is the cost of a false negative?
5. Is the evidence broad enough for that consequence?
6. Would one more sample plausibly change the decision?

The Claim Ladder

Claims can be arranged from narrow to broad. Move upward only when evidence earns the step.

1. Performed correctly on this item.
2. Performed correctly on several similar fresh items.
3. Performed independently across an item family.
4. Performed after delay.
5. Performed across varied representations.
6. Performed inside mixed work.
7. Performed under realistic examination conditions.
8. Shows a stable general capability across contexts.

A learner may legitimately sit at different levels for different skills.

The Final Validity Protocol for Tutors

1. Define the exact claim.
2. State the intended use.
3. Identify what evidence should look like if the claim is true.
4. Identify plausible alternative explanations.
5. Build a task that samples the construct.
6. Control irrelevant difficulty where appropriate.
7. Record support, time and tool conditions.
8. Observe process as well as outcome.
9. Score according to a predefined rule.
10. Use a fresh form after instruction.
11. Check whether other evidence converges.
12. Use delayed evidence for retention claims.
13. Use transfer evidence for generalisation claims.
14. Match decision consequence to evidence strength.
15. State the conclusion in bounded language.
16. Write down what evidence would change your mind.
17. Update the learner model when new evidence arrives.
18. Retire the measure when it no longer changes decisions.

Validity FAQ

Can a reliable test be invalid?

Yes. A measure can produce stable scores while consistently measuring something other than the intended construct.

Can an unreliable test be valid?

Severe instability weakens most interpretations because the observed score becomes too noisy. Reliability is not identical to validity, but it often limits how strong a validity claim can be.

Is a full examination always more valid than a short diagnostic?

No. A full paper may be better for integrated examination performance and worse for isolating one small mechanism.

Can one wrong answer matter?

Yes, as a clue. It usually should not support a broad stable diagnosis by itself.

Can one perfect answer prove mastery?

No. It proves successful performance on that task under those conditions. Broader mastery needs broader evidence.

Does a harder question provide better evidence?

Only if the extra difficulty represents the intended construct. Irrelevant difficulty can weaken validity.

Does high agreement between two markers prove validity?

No. It supports consistency between markers. Both could still reward the wrong features.

Does improvement prove the intervention worked?

It shows improvement occurred. Strong causal attribution needs stronger design because many other influences may change at the same time.

Is AI-assisted performance invalid?

No. It can be valid evidence of tool-supported performance. It should not be silently interpreted as unaided performance.

Should every diagnostic be standardised?

No. Local teacher-made diagnostics can be valuable for local decisions. Their claims should remain proportionate to their design and evidence.

How many items are enough?

There is no universal number. Narrow reversible decisions need less evidence than broad consequential claims. Sample until the next decision is unlikely to change with one more comparable observation.

Should support be removed immediately?

No. Support is part of teaching. Remove it when the question changes from “Can the learner perform with support?” to “Can the learner perform independently?”

Should a low score always trigger remediation?

No. First determine whether the low score represents the target capability, an access problem, noise, support difference or another condition.

Can a high score hide a weakness?

Yes. Strong execution can hide weak method selection, strong recognition can hide weak production, and familiar items can hide poor transfer.

Can a low score hide improvement?

Yes. The learner may need fewer prompts, make errors later in the process or succeed on harder transfer even while the raw percentage remains modest.

What is the best validity question a tutor can ask?

“What else could explain this result, and what small contrast would separate those explanations?”

What is the best validity question a parent can ask?

“What exactly does this result tell us, and what does it not tell us yet?”

What is the best validity question a student can ask?

“What part of the task did I actually fail: understanding, choosing, doing, checking, timing or transferring?”

Research Boundaries

Teacher-made diagnostics are local decision tools. They are not substitutes for formally validated standardised assessments when a high-stakes psychometric claim is required. Their strength is speed, proximity to instruction and the ability to test specific hypotheses about learning.

The 2026 fifth edition of Educational Measurement and the broader Standards tradition make a central point that travels well into tuition: validity belongs to the interpretation and use supported by evidence, not to a test label alone. That principle should make classroom assessment more useful, not more bureaucratic.

The Final Formula

Define the capability → build evidence that actually requires it → expose competing explanations → record the conditions → interpret no further than the evidence travels → make the smallest useful decision → test again when the learner or the conditions change.

That is training validity in practice.