Guides
Statistics Verification Guide: A Reading Framework
This statistics verification guide offers a framework for tracing figures, judging uncertainty, reviewing surveys, and checking what charts support.
Using this statistics verification guide
- A figure is only meaningful with its definition, population, unit, period and revision status attached.
- Pull the table or microdata, not the chart in the press release, and recompute the headline number yourself.
- Sampling error is one uncertainty out of several. Coverage, measurement, processing and model error sit alongside it.
- "Significant" answers a question about a null hypothesis, not about size, cause or consequence.
- Rebuild every chart from its underlying values before you repeat what the title says.
"Crime rose 20 percent" can mean five extra reports in a small jurisdiction, a change in a rate, a model estimate, or an administrative reclassification. The sentence is identical in all four cases.
Build the data identity card
Data identity card
- Producer and exact table or variable
- Collection purpose and legal basis
- Target population, sampling frame, geography
- Unit of observation
- Inclusion, suppression, missing-data rules
- Weighting and seasonal status
- Release version with download time
Two tables can share a label and measure different things. "Employment" covers people, jobs, payroll positions, respondents and modeled estimates. Read the data dictionary and the technical notes before you describe a trend.
Trace and reproduce the number
Go to the closest available table or file, not the chart pasted into a press release. Keep the raw values and the formulas. Recompute totals, percentages, rates, index changes and per-capita figures.
Name the parts of a number
Percentage
- What to record
- Numerator and denominator
- Common error
- Missing denominator
- Check before use
- Recompute the share
Change
- What to record
- Start and end values
- Common error
- Absolute vs percentage
- Check before use
- Say which change
Rate
- What to record
- Population and time basis
- Common error
- Wrong time basis
- Check before use
- Keep the basis
Index
- What to record
- Base and what it measures
- Common error
- Unstated base
- Check before use
- State the base
For a percentage, name the numerator and the denominator. For a change, record the start and end values, and say whether the claim means absolute change, percentage change or percentage points. For a rate, keep the population and the time basis. For an index, state the base and what it measures.
Rounding before calculation can move a result. Overlapping categories cannot be added. Seasonally adjusted components cannot be summed unless the producer says so. Nominal and inflation-adjusted money stay in separate columns.
Understand how the data were made
Ask what kind of source this is:
Survey inspection checklist
- Recruitment and sampling
- Response rate
- Weighting
- Question wording and order
- Field dates
- Mode
- Subgroup sizes you plan to quote
For a survey, inspect these elements:
- recruitment and sampling
- response rate
- weighting
- question wording and order
- field dates
- mode
- the size of each subgroup you plan to quote A margin of sampling error does not capture every error in the estimate.
For administrative data, learn what action creates a record, who can file one, how duplicates are removed, and whether a policy change altered the count. A rise in reports can mean more events, easier reporting, a broader definition, or all three.
For a series over time, know what has already been removed. In seasonal adjustment, the UK Office for National Statistics (ONS) splits a series into three components: calendar-related effects, irregular fluctuations, and trend. Its calendar bucket holds weather patterns, the school year, cultural events, and months with different numbers of trading days.
Test comparisons and uncertainty
Two point estimates can differ without supporting a claim that the populations differ. Use the producer's published standard errors, margins of error, confidence intervals, replicate weights, or recommended test. Where the design is complex, account for it.
The U.S. Census Bureau’s Statistical Quality Standard E2: Reporting Results of Statistical Studies requires sample-based estimates and comparisons to carry appropriate uncertainty, with methods, assumptions and data limits stated where readers need them. That standard governs Census products. Its separation of estimate, uncertainty and method travels well.
Significance is not size, and it is not cause. Report how large the difference is, which way it runs, how uncertain it is, and what a reader needs to place it.
Audit the chart as a claim
Before repeating a chart's claim, check:
- Whether the underlying values support the title.
- Whether color, area, or perspective exaggerates the magnitude.
- Whether bars start at zero, or a nonzero line-chart range is marked and justified.
- Whether dual axes have a clear reason.
Chart audit checklist
- Chart type
- Title
- Source line
- Units
- Denominator
- Period
- Categories
- Axis origin, scale, ordering, missing values, annotations
Bars start at zero because length encodes the value. A line chart may use a nonzero range when it is marked and justified, but the title must not oversell a small movement. Dual axes need a reason the relationship cannot be shown more plainly.
Write what the data support
Say "the estimate rose" rather than "it rose" when uncertainty matters. Hold back on highest, lowest, leading, majority, average, typical, risk, rate and correlation until the definitions and the comparison earn them. Put the period and the population next to the claim, not three paragraphs later.
Keep the analysis file, the raw download, the query or code, the chart data, the review notes and the final copy. When a revision changes the story, update every chart and every summary already sent out.
Common questions
Is a large sample always representative?
No. Size governs precision. Coverage, selection, response and weighting govern who the sample actually represents, and a large biased sample stays biased.
Is a government dataset automatically accurate?
No. It can be authoritative for a defined administrative process and still carry omissions, revisions, measurement limits and model assumptions. Read the methodology note for that specific release.
Does statistical significance mean a difference matters?
No. It describes evidence against a statistical null under stated assumptions. It says nothing about practical importance or about cause.
Should every chart show error bars?
No. Show uncertainty when it changes how a reader interprets the figure, and publish the uncertainty data alongside the chart and the text.
In this guide
- How to audit a poll or survey story before you cite itPoll audit workflow for sponsor, population, sample, recruitment, fieldwork, questions, weighting, uncertainty, subgroup comparisons, headlines, and corrections.
- Counts, percentages, rates, indexes, and averages comparedStatistical measures compared across counts, proportions, percentages, rates, ratios, indexes, means, medians, change measures, and seasonal adjustment.
- Chart verification checklist: the reviewer's sign-offA chart verification checklist for the person who signs off: what the reviewer receives, reproduces and checks before a chart, caption and headline go out.
- Common statistics and chart reporting problemsStatistics and chart reporting problems explained, from missing denominators and uncertain estimates to distorted axes, weak poll questions, and false causal claims.
- Fictional case: correcting a false survey trend and chartSurvey correction case showing how a newsroom finds an unsupported trend, checks margins of error and wording, repairs a chart, and publishes a complete correction.
