Statistics, polls, and charts: a verification guide

What to take away

• Define the measure, population, unit, period, and source before interpreting a number.
• Reproduce material calculations from the closest available data.
• Separate sampling uncertainty from coverage, measurement, processing, and model error.
• Compare like definitions and test whether an apparent difference is supported.
• Make headlines and charts retain the limits shown in the methodology.

A number does not arrive alone. It carries a definition, collection process, unit, time period, population, revision status, and uncertainty. Removing any one can change its meaning. "Crime rose 20 percent" may describe five additional reports in a small area, a rate change, a model estimate, or an administrative reclassification.

Photo and credit

The image documents historical data processing. It does not describe any modern dataset, survey result, or statistical method in this guide.

Build the data identity card

Before analysis, record:

• producer and specific dataset, table, variable, and release
• collection purpose and legal or administrative basis
• target population, sampling frame, geography, and period
• unit of observation and unit of measure
• inclusion, exclusion, suppression, and missing-data rules
• weighting, adjustment, modeling, and seasonal status
• revision schedule, version, and download time

Do not assume two tables with the same label use the same concept. "Employment" can refer to people, jobs, payroll positions, respondents, or modeled estimates. Read the data dictionary and technical notes before writing a trend.

Trace and reproduce the number

Find the closest available table or file, not a chart copied into a press release. Preserve the raw values and formulas. Recompute totals, percentages, percentage changes, rates, index changes, and per-capita figures.

For a percentage, name the numerator and denominator. For a change, record the start and end values and whether the language means absolute change, percentage change, or percentage-point change. For a rate, keep the population and time basis. For an index, explain the base and what the index measures.

Check whether rounding before calculation changes the result. Do not add overlapping categories. Do not sum seasonally adjusted components unless the producer says that is valid. Keep nominal and inflation-adjusted money distinct.

Understand how the data were made

Ask whether the figures come from a census, probability sample, opt-in survey, administrative system, model, sensor, transaction log, or combined source. Each produces different coverage and error questions.

For surveys, inspect sampling and recruitment, response, weighting, question wording, order, field dates, mode, and subgroup sizes. A margin of sampling error does not capture every source of error. Opt-in and model-based estimates may use other uncertainty measures.

For administrative data, learn what action creates a record, who has access, how duplicates are handled, and whether policy changes alter counts. More reports may reflect more events, easier reporting, broader definitions, or all three.

Test comparisons and uncertainty

Point estimates can differ without supporting a conclusion that populations differ. Use the producer's published standard errors, margins of error, confidence intervals, replicate weights, or recommended testing method. Account for design and dependence where applicable.

The Census Bureau's standard for reporting statistical results requires appropriate uncertainty for sample-based estimates and comparisons, along with information about methods, assumptions, and data limits when readers need them. The standard applies to Census products, but its separation of estimate, uncertainty, and method is a sound reporting discipline elsewhere.

Statistical significance does not establish practical importance or cause. Report the size and direction of a difference, the uncertainty, and the context readers need.

Audit the chart as a claim

Check chart type, title, source, units, denominator, period, categories, axis origin, scale, ordering, missing values, and annotations. Rebuild the chart from the underlying values. Confirm that color, area, and perspective do not exaggerate magnitude.

Use bar charts from zero because length encodes value. A line chart may use a nonzero range when clearly marked and justified, but the title must not exaggerate a small movement. Avoid dual axes unless the relationship cannot be shown more clearly another way.

Write what the data support

Distinguish "rose" from "the estimate rose" when uncertainty matters. Avoid highest, lowest, leading, majority, average, typical, risk, rate, and correlation unless the definitions and comparison warrant them. Put the period and population near the claim.

Preserve the analysis file, raw download, query, code or formulas, chart data, review notes, and final copy. When a revision changes the story, update every chart and distributed summary.

Common questions

Is a large sample always representative?

No. Size affects precision, while coverage, selection, response, and weighting affect who the sample represents.

Is a government dataset automatically accurate?

No. It may be authoritative for a defined administrative process and still carry omissions, revisions, measurement limits, or model assumptions.

Does statistical significance mean a difference matters?

No. It addresses evidence against a statistical null under stated assumptions, not practical importance or cause.

Should every chart show error bars?

No. Show uncertainty when it affects interpretation and provide accessible uncertainty information with the data and text.