The two questions

▲CONCEPT

Take the claim from the opening page: "Students in our school are healthier than the district average." Read cold, it sounds solid. The figure beside this paragraph opens it up, bracket by bracket, at every point where someone made a choice.

Bracket one asks what was actually measured. Bracket two asks who was in the sample, and what it was compared against. Neither question changes a single number in the claim. The data did not change — the claim changed, because the two questions were asked. The right-hand panel rewrites the same claim honestly: still a good result, now precise about what it can and cannot say.

walk one claim from stated to honest

1

The claim, as stated — 'our school is healthier than the district' — looks solid

2

Open bracket ① — what was MEASURED? (water ml/day, self-reported)

3

Open bracket ② — who was in the SAMPLE? (42 students, one class, voluntary)

4

Open the BASELINE — compared to what? (last year, a different school type, a different method)

5

Rewrite it honestly — still a good result, now precise about its limits

the data did not change — the claim changed, because the two questions were asked

ai-data-literacy.claim-brackets-walk

/50-system/standards/book-look/cell1-a4/adv-gautama.png

Gautama

sarus crane

A claim offered without its grounds asks you to obey, not to understand — so ask it plainly: what was measured, and compared with whom?

▲CONCEPT

The first formal question is: what was the camera pointed at? Its name is the operational definition — the exact specification of what is measured and how.

It names four things. First, the thing you are measuring — for example, student health. Second, the exact indicator that stands in for it — daily water intake in millilitres. Third, the method used to collect it — a self-report survey. Fourth, the time period it covers. Every measurement was a choice — a choice about what to look at, not a neutral fact handed down by the number itself.

Two studies can use the same word, say "health" or "stress," and still measure two different things. Without a stated operational definition, a reader has to guess what the number actually stands for. The person making the claim can then wave away any reading that turns out wrong.

question ① — what was the camera pointed at?

COMPOSED OF AN OPERATIONAL DEFINITION

THING — what is being measured? (e.g. 'student health')

INDICATOR — the exact proxy used (e.g. 'water intake, ml/day')

METHOD — how it was collected (e.g. a self-report survey)

TIME — over what period (e.g. one week in March)

two studies can use the same word and measure entirely different things

Meera's two-column check on "student stress"

Weaker. Two papers both claim to measure "student stress," so Meera's first glance treats the word as one finding and starts to line the two numbers up side by side. One paper reports a 10-point self-report anxiety scale — what a student is willing to say about how they feel. The other reports cortisol level in saliva — a hormone marker of the body's stress response. Same word on both covers. Two completely different indicators underneath it.

Stronger. Meera writes the two studies as two columns on her sheet before comparing anything: one headed "self-report anxiety scale," the other "cortisol level." Filled in side by side, the columns will not merge into one number — the self-report column is a report of felt stress, the cortisol column a chemical marker of physiological stress. Neither measurement is wrong, and neither is "the real" student stress; they are two different operational definitions wearing the same label. Her verdict is not which one is right — it is that the two studies cannot share one column until she knows which one each number actually is.

"Numbers are objective — they don't lie. Only graphs lie."

Every number came from a measurement, and every measurement was a choice about what to point the camera at. "42 students scored high on a self-report stress survey" and "42 students had a cortisol level under 15 nmol/L" can both get called evidence of student stress, and they measure two different things entirely. A number does not lie on its own — but an unstated operational definition lets its meaning drift away from what the measurement can actually support. Trusting a number more than a graph, simply because it is a number, is exactly the mistake.

▲CONCEPT

The second formal question is: what is outside the frame? Its name is sample and baseline, and the two always travel together.

The sample is who is actually inside the frame — its limits are size, how people were selected, and what it covers. The baseline is what the sample gets compared against. A claim with no named baseline, like "our school does better," is not comparable to anything yet. A claim with a mismatched baseline, a voluntary survey set against a mandatory census, misleads even when both numbers are perfectly accurate.

Get either one wrong and a true number misleads. The sample is the frame's inside edge. The baseline is its outside edge. Both have to hold for the comparison to mean what it claims to mean.

question ② — what is outside the frame? (three ways it breaks)

SIZE — too few to trust ('4 out of 5 dentists' = four people)

SELECTION — self-filtered ('students who use our app report higher grades')

COVERAGE — reaches only part of what it claims ('citywide air quality' from monitors in three neighbourhoods)

sample = who's inside the frame; baseline = what it's compared against; get either wrong and a true number misleads

Three ways a true number turns into a misleading claim

Check the size. "Four out of five dentists recommend this toothpaste" sounds like a survey. If only five dentists were ever asked, that is four people, not a trend. A sample this small cannot carry a claim this broad.
Check the selection. "Students who use our revision app report higher grades." Only app-users were ever asked. Students who never installed it were never in the sample — its coverage stops at app-users, so a self-filtered sample answers a narrower question than the claim states out loud.
Check the coverage. "Air quality across our city improved this year." The monitoring stations all sit in three neighbourhoods near the river. Neighbourhoods with no station nearby — including the ones with the most traffic — were never inside the sample's reach, so a citywide claim rests on data collected from part of the city.

▲CONCEPT

Accurate and fair are two different tests, and a claim can pass one and fail the other every single day.

Accurate is not the same as fair. Accurate means the data was recorded correctly — the numbers themselves are right. Fair means the claim stays inside what the data can actually support. The sample, the baseline, and the operational definition all have to match what the claim implies.

A self-selected sample, a mismatched baseline, or a too-narrow operational definition — any one of these fails the fairness test. Every number can still stay perfectly accurate. "The numbers are right" is where an honest audit begins. It is never where the audit ends.

"If the numbers are accurate, the claim must be fair."

Accuracy and fairness are separate tests, and passing one says nothing about the other. Accuracy asks whether the data was recorded correctly. Fairness asks whether the claim stays inside what that data supports. A self-selected sample, a baseline collected a different way, or an operational definition narrower than the claim implies — each of these fails fairness while every number stays completely accurate. Accuracy is where the audit begins. It says nothing yet about where the claim is allowed to end.