Common statistics and chart reporting problems

What to take away

• A correct number can support a false impression when its denominator, definition, period, or uncertainty is missing.
• Poll results cannot be interpreted safely without the exact question, choices, field dates, sample, and recruitment method.
• A chart must preserve the scale, gaps, and uncertainty present in the data.
• Correlation, sequence, and statistical significance do not establish cause or practical importance.
• Fix the headline, chart, caption, social copy, and data table together when an error crosses formats.

Statistical reporting fails in recognizable ways. Some errors begin in arithmetic. More begin when a writer shortens a qualified result into a clean but unsupported claim. The following problems are arranged by the point where they enter the reporting process, with a repair for each one.

1. The denominator disappears

"Complaints rose to 900" is not enough to show that service worsened. The system may have doubled its customers, added a reporting channel, or changed what counts as a complaint. A rate per customer or transaction might move in the opposite direction from the raw count.

Repair: Name the count, denominator, population, and period. If the denominator is unavailable, describe the result as a count and do not imply a rate.

2. Unlike units are compared

A story may compare people in one year with visits in another, survey estimates with administrative records, or monthly totals with annual totals. The values can look comparable because both are numbers in the same table.

Repair: Write a unit label beside every value before calculating. Check the source definition and time basis, not just the column heading.

3. Percentage change becomes percentage points

A rise from 20 percent to 25 percent is an increase of 5 percentage points and a 25 percent increase relative to the starting value. Swapping those descriptions changes the apparent size of the movement.

Repair: Publish the start and end values. Use "percentage points" for subtraction and "percent" for relative change.

4. A poll answer is separated from its question

The result depends on what respondents were asked, which options they saw, the order of questions, and whether an interviewer or screen presented them. "Do you support the transit plan?" does not measure the same attitude as a question that mentions its cost, route changes, or expected benefits.

Pew Research Center's guide to writing survey questions explains how wording, response choices, question order, and mode can affect answers. A reporter should quote or reproduce the exact item when the wording is material. A paraphrase should not broaden a narrow answer into support for an entire policy.

Repair: Obtain the questionnaire, response options, order, field dates, sponsor, sample method, and mode. If those details are withheld, state the limitation and narrow the claim.

5. The sample is treated as the population

A voluntary website poll measures its participants, not automatically all readers or residents. A probability sample can still miss parts of its target population or produce unstable subgroup estimates. A large sample does not cure a recruitment method that systematically excludes relevant people.

Repair: Identify the target population and how participants entered the sample. Report weighted and unweighted sample sizes when available. Do not assign a conventional margin of sampling error to a method that does not support it.

6. A point estimate is reported as exact

An estimated 52 percent is not proof that precisely 52 percent of the population holds a view. Sampling error, nonresponse, measurement choices, weighting, and other sources of error affect the result. A narrow majority claim may disappear once uncertainty is considered.

Repair: Use estimate language. Give the uncertainty measure supplied by the producer and explain whether it applies to the reported comparison.

7. Overlapping intervals are used as a complete significance test

Writers sometimes decide that two estimates differ only by looking at whether separate confidence intervals overlap. That shortcut can be conservative or wrong because the correct test depends on the estimates, standard errors, confidence level, and whether the observations are independent or paired.

Repair: Follow the producer's comparison method or calculate the difference with the appropriate covariance assumptions. Record the formula, inputs, and confidence level.

8. Uncertainty vanishes from the chart

A graph may show two crisp points even when each value is an estimate with a wide interval. Readers then see a definite rise, fall, or ranking that the evidence cannot distinguish. Forecasts create a related risk when their central line is displayed without a prediction range.

The UK Office for National Statistics provides specific guidance for showing uncertainty in charts, including confidence intervals, fan charts, and written annotation. The display method should fit the uncertainty and the task. At minimum, the text and graphic should not imply precision the source does not have.

Repair: Add intervals, bands, or a clear note. Avoid ranking estimates whose ranges do not support the order claimed.

9. The axis manufactures drama

A bar chart that starts far above zero magnifies small differences because bar length encodes magnitude. Unequal panel scales can make similar trends look different. A dual-axis chart can align unrelated movements through arbitrary scale choices.

Repair: Start bar and area charts at zero. Label nonzero line-chart axes plainly, keep comparable panels on the same scale, and test whether the conclusion survives a less dramatic display.

10. Missing values are drawn as zero

Unavailable, suppressed, and inapplicable data have different meanings. Turning any of them into zero invents observations. Connecting a line across a break can also suggest continuous measurement where none exists.

Repair: Preserve the source's missing-value codes. Use gaps, symbols, or notes, and explain why data are absent.

11. A recording change is called a real-world trend

Administrative totals depend on rules, software, staffing, access, and reporting behavior. A new form or legal definition can produce a jump without a comparable change in the underlying event.

Repair: Read technical notes and release histories. Mark breaks in series and seek a stable comparison measure.

12. Correlation is converted into cause

Two variables can move together because one affects the other, both respond to a third factor, the apparent relationship is accidental, or the measurement process links them. A before-and-after pattern also does not isolate the effect of an intervention.

Repair: Describe the observed association and the study design. Reserve causal language for evidence that addresses confounding, selection, time order, and plausible alternatives.

13. Statistical significance is sold as importance

A very small effect can be statistically distinguishable in a large sample while having little practical consequence. A meaningful-looking effect can also remain uncertain in a small sample.

Repair: Report the effect size and uncertainty, then explain the real-world scale. Do not use a significance threshold as a substitute for judgment.

14. A revision updates the table but not the story

Official datasets can be revised. If the spreadsheet changes but the headline, embedded chart, newsletter, and social post remain untouched, the publication preserves multiple versions of the claim.

Repair: Keep a distribution list for every data product. When a material number changes, update all surfaces and add a dated correction explaining the original and replacement values.

A compact diagnosis table

Symptom	Likely hidden issue	First document to request
Large count with no context	Missing denominator	Data dictionary
Sharp one-year jump	Definition or system change	Release notes
Narrow poll majority	Sampling uncertainty	Methodology report
Different subgroup rankings	Small samples or weighting	Crosstab notes
Dramatic bar difference	Truncated baseline	Chart data
Clean forecast line	Omitted model uncertainty	Technical appendix
Cause stated from observation	Confounding	Study design

Common questions

Is a mathematically correct percentage enough to publish?

No. Its denominator, population, period, definition, and relevance to the claim must also be correct.

Should every poll story print the full questionnaire?

The article should quote material wording and provide enough method detail for interpretation. A separate methodology document can carry the complete questionnaire.

Are truncated axes always misleading?

No. A clearly labeled nonzero axis can help examine small changes in a line chart. It is unsuitable for bars when length represents magnitude.

What deserves a correction rather than a clarification?

Correct a claim when the published wording or display is false or materially unsupported. Clarification is not a softer label for an error.