A summary of Jon Krosnicks presentation at AAPOR 2026 in Los Angeles, link to the session: The Past, Present, and Future of Probability and Non-Probability Samples for Surveys

“Don’t lie, don’t repeat history’s mistakes”

Abstract

This lecture delivers a rigorous empirical defense of probability sampling over non-probability (opt-in and river) methods in survey research. Synthesizing historical polling failures from 1948 and 2016 alongside benchmark validation data , the analysis demonstrates that true random samples maintain highly accurate baseline error rates (~3%) and remain resilient against declining response rates. Conversely, non-probability approaches yield severe, unpredictable errors that statistical weighting routinely worsens by inflating fundamentally unrepresentative respondents. Furthermore, a systematic audit of literature defending opt-in panel accuracy exposes pervasive scientific errors, invalid benchmarks, question mismatches, and a lack of transparency. The lecture concludes with an unyielding mandate to the survey research community to protect scientific integrity by refusing to substitute convenience for random probability designs.

This document provides a comprehensive summary of Jon Krosnick’s lecture on the historical context, methodological differences, and empirical performance of survey sampling methods.

Historical Context and the 1948 Election Mistake

• In the 1940s, prominent polling figures Elmo Roper, Archibald Crosby, and George Gallup were highly regarded leaders in American survey research.
• All three organizations famously miscalled the 1948 U.S. presidential election by predicting a victory for Thomas Dewey over Harry Truman because they relied on opt-in sampling.
• In contrast, a less famous group at the University of Michigan created the American National Election Study (ANES) and successfully predicted the correct outcome using probability sampling.
• Over the subsequent decades, the probability-based results of the ANES continued to align beautifully with official government election reports.

The Evolution of Probability Internet Panels

• Remote random sampling began in the 1980s when Willem Saris in the Netherlands pioneered the idea of placing Commodore 64 computers and modems into a random sample of households.
• Norman Nie and Doug Rivers brought Saris’s idea to America, utilizing random digit dial telephone calls to recruit a random sample that became the “Knowledge Panel” (now owned by Ipsos).
• This probability-based online panel methodology remains alive and well today across numerous countries and through various prominent U.S. panels, including NORC’s Amerispeak, USC’s Understanding America, SSRS, Gallup, the Pew Research Center, and Novus in Sweden.

Quantifying Accuracy and the Failure of Weighting

• A landmark 2011 study by David Yeager compared survey results against known government benchmarks and found that probability samples yielded a low average error of about three percentage points.
• While commercial opt-in panel companies claimed they could deliver equal or superior data quality for less money, Yeager’s research proved that non-probability samples yielded substantially higher, widespread errors reaching up to 39 percentage points.
• Statistical weighting slightly improves the accuracy of true probability data, but it overwhelmingly fails to fix or improve non-probability data.
• Applying weights to an opt-in sample fails because it merely inflates individuals who are fundamentally the wrong participants for the study.

Comparison Matrix: Sampling Methods

Feature / Metric	Probability Sampling	Non-Probability Methods (Opt-in / River)
Average Error Rate	~3 percentage points (Yeager, 2011)	Significantly higher, up to 39 percentage points
Impact of Weighting	Slightly improves accuracy	Overwhelmingly hurts accuracy (inflates the wrong respondents)
Impact of Declining Response Rates	Almost no negative impact if the sample remains random	Not applicable (inherently lacks a random baseline)
Historical Examples	ANES (1948 accurate prediction)	Roper, Crosby, Gallup polls (1948 failure); 2016 election polling failures

“River Sampling” and the 2016 Election Failure

• “River sampling” refers to the practice of scooping up random internet users as they browse rather than maintaining a structured panel, a method that consistently performs worse than traditional opt-in panels.
• The heavily flawed forecasts for the 2016 presidential election—which overwhelmingly favored Hillary Clinton—were driven by a reliance on non-probability river samples.
• During the final week before the 2016 election, polls utilizing probability sampling maintained a very small error margin, whereas non-probability polls missed the mark by as much as 17 to 18 percentage points.
• Attempting to average a massive volume of these non-probability polls together did nothing to cancel out their underlying errors.

Debunking the Response Rate Myth and Literature Audits

• Despite widespread concerns over falling response rates, empirical evidence demonstrates that survey accuracy declines almost not at all as response rates drop, provided the sample is drawn randomly.
• Historical data from pre-election polls between 1936 and 2004 shows that average polling errors actually decreased over time even as response rates were actively declining.
• Stanford researcher Lavie Sundar conducted a systematic audit of published literature that claimed non-probability samples could perform as well as probability methods.
  Sundar’s audit uncovered a massive volume of scientific errors, including the use of non-random surveys as baseline benchmarks, comparing surveys with completely mismatched questions, mathematical typos, undisclosed methodologies, and lost or restricted records from defunct companies.

Conclusion

Krosnick concludes the lecture with two straightforward requests for practitioners: do not lie about data quality and do not repeat history’s mistakes by abandoning random sampling.

About Jon A. Krosnick, Ph.D.

Frederic O. Glover Professor, Stanford University

Jon A. Krosnick is a leading social psychologist and a globally recognized authority on survey research methodology, political behavior, and the psychology of attitudes. At Stanford University, he holds professorships in Communication, Political Science, and Psychology, and directs the Political Psychology Research Group.

His foundational methodological research focuses on questionnaire design, maximizing survey data quality, and demonstrating the critical importance of scientific, probability-based sampling. Substantively, he is widely recognized for his decades-long tracking of American public opinion on climate change and his insights into how voter preferences are formed.

Dr. Krosnick previously served as the Co-Principal Investigator of the American National Election Studies (ANES) and has acted as an expert witness on survey design in over 160 legal cases. He holds a Ph.D. from the University of Michigan and an A.B. from Harvard University. His contributions earned him the AAPOR Lifetime Achievement Award in 2014, and he is an elected Fellow of the American Academy of Arts and Sciences.

Read more about Jon here: Jon Krosnick | Political Science