How to audit a poll or survey story

What to take away

• Identify who sponsored, conducted, and published the survey.
• Match the sample to the population named in the headline.
• Read the exact question wording, order, response choices, and field dates.
• Check weighting, uncertainty, subgroup sizes, and comparison methods.
• Rewrite the claim when the available disclosure cannot support it.

A poll result is not only a percentage. It is an answer produced by a question, among selected respondents, through a mode, during a defined period, then weighted and analyzed. Audit that chain before reporting what "Americans," "voters," "workers," or "customers" believe.

Photo and credit

The image supplies data-processing context. It does not depict the administration or analysis of any poll discussed here.

1. Obtain the full disclosure

Find the questionnaire or topline, methodology, tables, field dates, sample source, sponsor, conduct organization, weighting variables, design effect or uncertainty method, and sample sizes. A press release or chart is not enough for a consequential claim.

Record who selected the topic, paid for the work, wrote the questions, collected responses, analyzed data, and chose what to publish. Sponsorship does not invalidate a result, but it helps readers judge design and selection.

2. Define the target population

Write the population exactly: U.S. adults, registered voters, likely voters under a named model, customers who bought in a period, employees who answered an internal survey, or users of an opt-in panel. Compare that with the headline.

Check eligibility, geography, language, age, internet or phone access, panel requirements, exclusions, and coverage. A survey of app users cannot become a claim about all residents merely because the sample is large.

3. Inspect recruitment and response

Determine whether the sample was probability-based, address-based, random-digit dialed, recruited from a panel, intercepted online, open to volunteers, or drawn from a customer list. Note contact attempts, incentives, response or participation metrics, and replacement rules.

Do not apply a probability-sample margin-of-error formula mechanically to an unrestricted opt-in poll. Use the producer's stated uncertainty method and describe the design plainly.

4. Read every load-bearing question

Copy exact wording, response choices, preceding questions, randomization, and mode. Look for loaded premises, double questions, missing options, unclear time frames, forced choices, unfamiliar terms, and response orders that could affect answers.

AAPOR's survey research best practices advise defining the population, using balanced questions, disclosing data collection, and taking special care when analyzing nonprobability surveys. The guidance also explains weighting and reporting sampling uncertainty for suitable designs.

If the article paraphrases an answer, confirm that the new wording preserves the construct. "Supports the proposal" is not interchangeable with "believes the proposal will pass."

5. Audit weighting and estimates

Identify the variables and targets used for weighting, the range of weights if disclosed, and whether design effects affect precision. Confirm weighted and unweighted sample sizes. Heavy weighting can make a nominally large subgroup behave like a smaller one statistically.

Reproduce headline percentages from the table. Check excluded nonresponses, rounding, multiple-response questions, combined categories, and whether rows or columns should total 100 percent.

6. Test comparisons

For change over time, compare identical wording, response options, population, mode, weighting, and field timing. For subgroups, use their own sizes and uncertainty. Do not infer that two groups differ merely because one estimate is significant and the other is not.

Account for overlapping samples or repeated respondents where relevant. Ask the pollster for the recommended test and underlying values. Avoid "surge," "collapse," "lead," or "record" when the difference is not supported.

7. Review release and nonresponse context

Check whether the organization released all asked questions, selected only favorable items, changed its likely-voter model, or compared a current estimate with a convenient older point. Seek prior waves and independent polls using similar measures.

Nonresponse can bias results even when a reported sampling margin is small. Look for analyses of who did not respond and how weighting addresses known differences.

8. Rewrite and document

Name sponsor, field dates, population, sample size, mode, and key uncertainty near the first finding. Link to the specific methodology and questionnaire. Explain material limits without burying them after a strong headline.

Save the source tables, calculations, exact wording, emails, and final chart. If disclosure is missing, either narrow the claim, label the result as an unverified sponsor figure, or omit it.

Common questions

Is a poll of 1,000 people enough?

Sample size alone cannot answer that. Design, coverage, response, weighting, subgroup analysis, and intended population all matter.

Can an opt-in poll be useful?

Yes, with an appropriate design, adjustment, disclosure, and cautious inference. It should not be described as a simple random sample.

Does a close race mean a statistical tie?

Not automatically. Use the poll's design and uncertainty method, and avoid treating one poll as an election forecast.

Should undecided respondents be excluded?

Only if the question and analysis call for it, and the denominator is disclosed clearly.