Notes on Pew Research / YouGov

References

• Read The Public Opinion Polling Basics on the PEW Research Site
• Watch this Playlist on Pew Research Center Methods
• Open Secrets Pew Charitable Trusts
• YouGov methodology

Pew Research

Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. We conduct public opinion polling, demographics research, content analysis, and other data-driven social science research. We do not take policy positions.

• Independence, objectivity, accuracy, humility, transparency, and innovation are indispensable to the mission and success of the center.
• The center generates a foundation of facts that enriches the public dialogue and supports sound decision making. Our public opinion surveys allow the voice of the people to be heard, and our demographic, economic, and political analyses provide context to understand how the world is changing. Their mission is to inform, not to prescribe.
• The center employees must avoid conflicts of interest.
• The center does not contribute, directly or indirectly, to political campaigns or to political parties or groups seeking to raise money for political campaigns or parties, and the center does not reimburse any employee fir political contributions. In addition, the Center prohibits partisan political activity by senior staff.
• The Center is committed to conducting research in a manner that is impartial, open-minded, and meets the highest standards of methodological integrity. They describe their findings and methods accurately and in sufficient detail to permit outsiders to evaluate the credibility of the results.
• They study a wide range of topics including politics and policy; news habits and media; the internet and technology; religion, race, and ethnicity; international affairs; social, demographic, and economic trends; science; research methodology and data science, and immigration and migration.
• They have a staff of more than 160 people and 11 research teams. Their experts combine the observational and storytelling skills of journalists with the analytical rigor of social scientists. Being inclusive, diverse and equitable is foundational to the Center’s mission and is integral to how we, at the Center, achieve excellence.
• Pew Research is a subsidiary of The Pew Charitable Trusts, their primary funder.

Public Opinion Polling Basics

For centuries, scholars have debated the question: “What is public opinion?” Or, put differently, “What is the will of the people?” And for nearly as long, they have debated how best to measure it. The great political scientist V.O. Key Jr. defined public opinion as “those opinions held by private persons which governments find it prudent to heed.”

The typical opinion poll consists of a set of interviews with people selected to be representative of a population of interest, such as all US adults. In survey interviews, a pollster will ask everyone the same set of questions in the same way, then tally up the answers and report the results. If the poll is designed properly, the responses collectively can describe the thinking and experiences of the much larger group of people the sample is chosen from.

Why do We Have Opinion Polls?

The word poll originally was a synonym for head. Polls counted heads at meetings or rallies to gauge popular sentiment. Elections can not tell you why people voted for a certain candidate, and citizen communications, through actions like protests, are hardly representative of the broader population. They tend to be more highly educated and more affluent than the average adult. Voters and those who contact public officials tend to be older. Also, a lot of polling is conducted about topics other than those directly addressed by the government.

The basic goal of a good public opinion poll is to give everyone in the population - regardless of their wealth, age, education, race, knowledge of politics, or experience with it - an equal voice about the issues of the day. Pew uses the technique called random sampling to try and accomplish this. If it works properly, the sample they come up with for a survey can be assumed to be representative of the larger population on any question Pew might be interested in. Random sampling makes sure that some voices aren't louder than others. Polls can help serve as a check of those in power who might try to claim that they know what the American people think or want.

Evidence for polling affecting the outcome of elections is mixed and complicated. Pollsters argue that polling data - whether in elections or just when issues are being debated - is just another source of information that people have a right to see and incorporate into their decision making. Having a realistic view of what other people think might even turn down the heat on partisan polarization.

How Does Polling Work

The general goal of a poll is to describe or explain something about a population of interest (for example, the general public or registered voters) by asking questions of a small number of people who represent that larger population. The poll requires three things:

• a set of questions to ask (a questionnaire)
• a set of people to interview (a sample)
• some way of connecting the two (sometimes called the mode of interview)

Once you've conducted the interviews, you have to tally up the data and summarize the results in a way that accurately reflects what the respondents told you. In-person interviews are very expensive and rarely happen today. Paper-and-pencil surveys remain a staple of the polling world today. Telephone interviewing is still used often. Today, web surveys account for the majority of all polling because they are faster and - because an interviewer is not needed - much less expensive. Typically, people are initially contacted by phone, mail, or email and then invited to take a survey on the web. Each method has strengths and weaknesses. Self-administered surveys, like those done on paper or online, may yield more honest and accurate answers and allow people to respond whenever it's convenient for them. They reduce social desirability bias, a tendency of people to answer in a way that leaves a favorable impression. The goal is for all respondents to have the same experience and answer freely and thoughtfully. One of the interesting trends in polling is the rise of multimode surveys - polls that use more than one way of reaching people or that give the option of responding in more than one way.

The best way to get diversity is through something called random sampling, a method that gives everybody in your population of interest an equal (or at least known) chance of being included. Typical random sampling approaches include calling random selections of telephone numbers or mailing a survey to a random sample of addresses. Telephone numbers can be generated given area codes and the US Postal Service maintains a list of all residential addresses in the US.

In a typical poll, most who get contacted don't get interviewed. There are some differences between those who are contacted and participate in a poll and those who do not. Pollsters can correct for this problem using a technique called weighting.

Weighting boosts the voices of people who belong to groups that are less likely to participate in polls and lowers the voices of people from groups that are more likely to take polls. Pollsters rely on knowledge about the population from the U.S. census or other sources to guide their decisions on weighting. Weighting is one of the more complex things pollsters do.

Opt-in sampling relies on various techniques to get people online to volunteer (opt-in) to take surveys, often in exchange for small rewards like gift cards. Opt-in polls make up a big share of the market research world and were used by a sizable share of organizations doing presidential polling in 2020.

As a group, opt-in surveys are less accurate than those that use random sampling. This is especially true for estimates for young adults, Hispanics and other minority groups.  

Opt-in surveys are also vulnerable to contamination from bogus respondents - people who are ineligible for the survey or who provide insincere responses.

It's important that poll questions are relatively simple, clear and understandable by people from a wide variety of backgrounds. It's also important to ask questions that people can answer.

Don’t be too suspicious about this [referring to leading questions], though. Most public pollsters are trying to get a fair reading of people’s opinions.

What are the Different Kinds of Polls

The term poll is usually applied to a survey designed to measure people's opinions. Some polls include questions about people's experiences, habits, or future intentions. The Pew Research Center uses US Census Bureau surveys as benchmarks to help ensure the accuracy of their own polls.

Issue polling – our bread and butter here at the Center – is focused on opinions and experiences: what people know about an issue, how it affects them, how important they think it is and what they think should be done about it.

Businesses often conduct surveys to help them better understand consumer sentiment and preferences. A great deal of market-related survey research is done to help businesses predict demand for new and existing products and services.

Other methods of getting public opinion include:

• Analyzing election results
• interviewing people
• scraping social media data
• analyzing Google searches
• in-depth interviews and focus groups

But as exciting as these new methods may be, a  variety of challenges  have hindered the development of such methods as a potential replacement for surveys. 

What is the deal with Election Polls?

An election poll is a poll conducted before an election that focuses on the election. You'll often hear about election polls when they're being cited to show who is ahead/behind in the race. Election polls are useful for understanding the meaning of the election. Polls help clarify what voters - and nonvoters - are saying with their actions. A list of how election polls are used:

• Help identify what kinds of people are voting for which candidate
• Help explain the issues and concerns that may be leading to a choice for a particular candidate
• Provide insight into why some people vote and others do not, including differences in engagement between supporters of each candidate
• Reveal how, when and where people cast their votes
• Describe how satisfied people are with their choice of candidates
• Tell us how the public judges the process and reacts to the outcome of the election

Election polls are conducted just like other polls. As seen in the chart below, opt-in sampling has grown rapidly. The number of pollsters has grown rapidly since 2008. The one thing that is unique about pre-election polls is the fact that they have to determine which people among those interviewed will actually turn out.

Election polls in the US and around the world have a good track record of accuracy. A comprehensive study found that polls taken shortly before an election had an average error of less than 2 percentage points in estimates of support for a given party or candidate. The errors in 2016 mostly affected state polls, but many of the errors were in critical battleground states that contributed to Donald Trump winning the presidency. National polling that year was quite accurate, but quantitative forecasts of the election - some of which gave Hilary Clinton a greater than 90% chance of winning - may have undermined public confidence in the polls by leading many observers to believe the outcome was far more certain than the polls actually indicated. Even larger errors affected the state and national polls in 2020.

According to the American Association for Public Opinion Research (or AAPOR, the professional association for the survey research field), national polls in 2020, on average, overstated Joe Biden’s margin over Trump by 3.9 percentage points, the largest such error since 1980.

The traditional margin of error doesn't capture all the sources of error affecting polls.

Is accurate polling becoming harder to do?

• It's getting harder to reach people and persuade them to participate in a poll
• • People have grown wary of answering calls from unknown numbers or cooperating with requests for information from a person or organization they are not familiar with.
  • In part because of this decline in telephone response rates, Pew Research Center decided a few years back to stop relying primarily on telephone surveys and switch to using rigorous online surveys.
  • Research has shown that the quality of answers people give in self-administered surveys is often higher than in surveys with an interviewer.

Pew Research Center Methods YouTube Videos

About Pew Research Funding

Founded in 1948, The Pew Charitable Trusts uses data to make a difference. Pew addresses the challenges of a changing world by illuminating issues, creating common ground, and advancing ambitious strategies that lead to tangible progress.

Pew's mission is to:

• Improve public policy by conducting rigorous analysis, linking diverse interests to pursue common cause, and insisting on tangible results
• Inform the public by providing useful data that illuminates the issues and trends shaping the world
• Invigorate civic life by encouraging democratic participation and strong communities

Their values include: equity, humility, impact, inclusion, integrity, and nonpartisanship. Pew Charitable Trusts is the sole beneficiary of seven individual trusts established between 1948 and 1979 by two sons and two daughters of Sun Oil Co. founder Joseph S. Pew and his wife, Mary Anderson Pew.

We know that when we include a diverse range of perspectives and backgrounds, we can get better at asking the right questions and crafting innovative solutions. Read more about our commitment to DEI.

YouGov

About

YouGov is an international online research data and analytics technology group.
Our mission is to offer unparalleled insight into what the world thinks.

• Their solutions help the world's most recognized brands, media owners and agencies to plan, activate, and track their marketing activities
• At the core of their platform is an ever-growing source of connected consumer data that has developed daily over 20 years of operation. They call it living data
• They are the innovators and pioneers of online market research, and they have a strong reputation as a trusted source of data and insights. Testament to this, YouGov is regularly referenced by the global press, and they are the most quoted market research source in the world
• Their purpose is to give our global community a voice by collecting, measuring, and analyzing their opinions and behaviors and reporting the findings accurately and free from bias

Methodology

• All of YouGov's surveys are run with consistent methods and values
• YouGov conducts surveys for a wide variety of clients. Their high methodological standards ensure the reliability and validity of their full range of surveys
• Their editorial staff carries out surveys themselves, with the goal of creating and sharing accessible and neutral public surveys that inform the readers about where Americans stand on issues of the day. They aim to be transparent.
• All their surveys are conducted online, and respondents can choose when and where they take the survey.
• While anyone can join their panel, YouGov chooses which panelists they invite to take each survey, and they have rigorous processes in place to ensure the quality of responses included in the final results.
• YouGov operates its own panel, providing them with direct management over the sampling process and ensuring a high standard of data quality. Their online panel uses what's known as nonprobability sampling, which allows them to collect data quickly and cost-effectively from specific groups of interest. This differs from probability sampling, in which all people have an equal chance of being selected into a panel. To ensure their findings are representative, they invite a representative set of panelists to take each survey and apply statistical weighting to adjust for differences between the sample and the target population.
• YouGov has a large panel of Americans who take part in our surveys, and anyone who is an adult living in the U.S. is eligible to sign up to participate. For completing a survey, respondents are awarded points that eventually can be exchanged for a small amount of money - or the equivalent - at various vendors, which are chosen based on feedback from the panelists.
• YouGov starts by deciding whose opinion, behavior, and characteristics they are trying to estimate with each survey. They then calculate basic demographic and political information about the group that they are studying, using government data and other sources. This includes characteristics such as age, gender, race, education, and voting behavior.
• • They determine the sample size for each survey based on the level of precision needed and whether they plan to analyze results for specific subgroups within the population being surveyed. For general population surveys, they aim for a sample size of 1,000 to 2,000 respondents, which provides a good balance between statistical reliability and resource efficiency.
  • Most surveys are designed to be completed in less than 20 minutes. The time it takes to field a survey is typically 1 to 5 days and depends on the target audience and complexity of the survey.
• After the surveys finish, they use a process called weighting to make the results more accurately reflect the population they intend to represent. This method gives more or less weight to each respondent based on their demographic characteristics - such as age, gender, race, and presidential vote. The share of respondents with these characteristics is compared to benchmarks from sources such as the US Census and election results. When a survey has a higher or lower share of people with a certain characteristics than the population they are studying, they adjust the respondents' weights - and their responses' influence on the results - accordingly.

To ensure that our data accurately reflects American opinion, we employ a strategy that includes monitoring, testing, and refinement. Our approach involves a dedicated team and a set of procedures to catch respondents who are misrepresenting themselves or providing otherwise unreliable data. It starts when a person joins our panel: We require all panelists to activate their account via email; we then run checks on their IP address and verify email addresses. Among the techniques we use is what we call a response quality survey, which gauges the reliability of responses by comparing them against known or highly predictable information about the panelists. Additionally, we use data about respondents' devices and locations to detect misrepresentation.

• They also check whether the panelists answer the survey too quickly, answer in inconsistent ways, or repeatedly give the same answer to similar, consecutive questions to detect and disqualify potentially fraudulent responses. Respondents failing these quality control checks are removed from the final sample, and those who repeatedly fail are excluded from the panel altogether. They randomize the order of questions as well.
• When they report findings, they are aggregated to a degree that protects any individual respondent from being identified.
• YouGov reports the margin of error on each survey to show the range that the shares of answers to the survey questions would likely fall between if they had talked to everyone in the country instead of just a sample.
• • This margin of error only applies to the whole sample. When examining subgroups with smaller sample sizes, the margin of error will be higher.
• For projecting votes, they rely upon a multilevel regression with post-stratification (MRP) model.
• Surveys on their website:
• • The Daily Questions: On most weekdays, they ask a large sample of panelists at least one set of three topical questions. Anyone can answer the Daily Questions and immediately view live, unweighted results from people who have answered thus far.
  • Trackers: The data displayed in trackers comes from regular tracking surveys conducted by YouGov. Each survey indicates a representation sample of respondents - typically at least 1,000.