How to audit a poll or survey story before you cite it

What to take away

A poll result is not only a percentage. It is an answer produced by a question, among selected respondents, through a mode, during a defined period, then weighted and analyzed. Audit that chain before reporting what "Americans," "voters," "workers," or "customers" believe.

Photo and credit

The image supplies data-processing context. It does not depict the administration or analysis of any poll discussed here.

1. Obtain the full disclosure

Find the questionnaire or topline, methodology, tables, field dates, sample source, sponsor, conduct organization, weighting variables, design effect or uncertainty method, and sample sizes. A press release or chart is not enough for a consequential claim.

Record who selected the topic, paid for the work, wrote the questions, collected responses, analyzed data, and chose what to publish. Sponsorship does not invalidate a result, but it helps readers judge design and selection.

2. Define the target population

Write the population exactly: U.S. adults, registered voters, likely voters under a named model, customers who bought in a period, employees who answered an internal survey, or users of an opt-in panel. Compare that with the headline.

Check eligibility, geography, language, age, internet or phone access, panel requirements, exclusions, and coverage. A survey of app users cannot become a claim about all residents merely because the sample is large.

3. Inspect recruitment and response

Determine whether the sample was probability-based, address-based, random-digit dialed, recruited from a panel, intercepted online, open to volunteers, or drawn from a customer list. Note contact attempts, incentives, response or participation metrics, and replacement rules.

Do not apply a probability-sample margin-of-error formula mechanically to an unrestricted opt-in poll. Use the producer's stated uncertainty method and describe the design plainly.

A low response rate is not the whole story. Pew Research Center's study of what low response rates mean found telephone response rates settling near 9 percent after decades of decline.

The bias this introduced was limited in scope. The study also said response rate is an unreliable indicator of bias. Ask what was measured against benchmarks, not merely how many people answered.

4. Read every load-bearing question

Copy exact wording, response choices, preceding questions, randomization, and mode. Look for loaded premises, double questions, missing options, unclear time frames, forced choices, unfamiliar terms, and response orders that could affect answers.

Disclosure is the thing to insist on, but it is not endorsement. AAPOR's Transparency Initiative recognizes organizations that pledge to publish research methods behind released studies and make them open to inspection.

AAPOR states plainly that it makes no judgment about the approach, quality, or rigor of the methods disclosed. A pollster who shows the wording has met the first test, not the last one.

If the article paraphrases an answer, confirm that the new wording preserves the construct. "Supports the proposal" is not interchangeable with "believes the proposal will pass."

5. Audit weighting and estimates

Identify the variables and targets used for weighting, the range of weights if disclosed, and whether design effects affect precision. Confirm weighted and unweighted sample sizes. Heavy weighting can make a nominally large subgroup behave like a smaller one statistically.

Reproduce headline percentages from the table. Check excluded nonresponses, rounding, multiple-response questions, combined categories, and whether rows or columns should total 100 percent.

6. Test comparisons

For change over time, compare identical wording, response options, population, mode, weighting, and field timing. For subgroups, use their own sizes and uncertainty. Do not infer that two groups differ merely because one estimate is significant and the other is not.

Account for overlapping samples or repeated respondents where relevant. Ask the pollster for the recommended test and underlying values. Avoid "surge," "collapse," "lead," or "record" when the difference is not supported.

7. Review release and nonresponse context

Check whether the organization released all asked questions, selected only favorable items, changed its likely-voter model, or compared a current estimate with a convenient older point. Seek prior waves and independent polls using similar measures.

Nonresponse can bias results even when a reported sampling margin is small. Look for analyses of who did not respond and how weighting addresses known differences.

8. Rewrite and document

Name sponsor, field dates, population, sample size, mode, and key uncertainty near the first finding. Link to the specific methodology and questionnaire. Explain material limits without burying them after a strong headline.

Save the source tables, calculations, exact wording, emails, and final chart. If disclosure is missing, either narrow the claim, label the result as an unverified sponsor figure, or omit it.

Common questions

Is a poll of 1,000 people enough?

Sample size alone cannot answer that. Design, coverage, response, weighting, subgroup analysis, and intended population all matter.

Can an opt-in poll be useful?

Yes, with an appropriate design, adjustment, disclosure, and cautious inference. It should not be described as a simple random sample.

Does a close race mean a statistical tie?

Not automatically. Use the poll's design and uncertainty method, and avoid treating one poll as an election forecast.

Should undecided respondents be excluded?

Only if the question and analysis call for it, and the denominator is disclosed clearly.