At Pew Research Center, we mostly survey the U.S. public through the American Trends Panel (ATP) – a randomly selected, nationally representative group of more than 12,000 adults who agree to take our polls on a regular basis. We created the panel in 2014 and have made several improvements to it since then.
In this Q&A, we chat with Dorene Asare-Marfo, senior panel manager of the ATP, about the fundamentals of the panel, some recent changes we’ve made to it, and more. (Read more about the ATP’s methodology.)
Let’s start with the basics. What is a survey panel? How do you get people to join one? How do participants take the surveys?
A survey panel is made up of a group of people who have agreed to regularly take surveys.
At the Center, we recruit people to the American Trends Panel using printed mailers that we send to a random sample of U.S. addresses. This way, nearly every American has a chance to be included. We ask a person who lives at the sampled address to take one of our surveys, and at the end of that survey, we invite them to take additional ones in the future.
To help persuade people to join the panel, we include a small amount of money in the mailers we send out. And for those who end up joining, we provide compensation for every survey they complete.
We typically send out one or two surveys each month, but not every panelist is asked to take each one. Rather than invite every panelist to every survey, we’ll often send a survey to a subset of panelists (e.g., to 5,000 or 10,000 of them).
The vast majority of ATP participants take our surveys online, using a smartphone, laptop or any internet-enabled device they prefer. But we also now give people the option to take surveys over the phone with an interviewer. This is to account for the fact that not everyone is online or feels comfortable taking surveys online.
Why did you create the American Trends Panel in the first place?
It was very much a response to the challenges facing the survey research industry.
The model for most polling in the past few decades was “one-off” surveys. That is, each survey relied on a new sample of respondents who we interviewed once and never again. This was a workable model when response rates to surveys – particularly phone surveys – were a lot higher than they are now. But response rates have plummeted over the years, so the benefits of a panel approach became more apparent.
Panels are also cost-effective. While we devote a lot of resources to recruiting panelists and getting a good representative sample of the country, the people who ultimately join the panel tend to participate at high rates. Typically, more than 90% of the panelists we invite to take a particular survey end up doing so. Over time, this is a much more efficient and economical way to conduct high-quality surveys.
Having a panel also allows us to avoid asking some of the same questions every time we conduct a survey. For example, once we’ve collected our panelists’ basic demographic information – race, gender, age and so forth – we don’t need to ask those questions again in each survey.
One other important bonus is that a panel allows us to see how people’s attitudes and behaviors change – or don’t change – over time. For example, a few years ago, we published a study showing that most people who had voted for Donald Trump in 2016 continued to have very warm feelings about him in early 2018.
Are there any downsides to using a survey panel?
American Trends Panel participants take a lot of surveys. Like anything you do regularly, you become familiar with the experience, which can be good or bad.
The good part is that experienced panelists better understand what’s expected of them. They may gain trust in the survey research process, making them more honest and careful with their answers.
The downside is that they might become more interested in or knowledgeable about the topics we ask about, or they may change in other ways because of the experience of being part of the panel. In other words, panelists may no longer behave the way a fresh sample of people would. We call this “panel conditioning.”
We took a closer look at panel conditioning in 2021. The good news is we didn’t find any evidence that panel conditioning led to inaccurate estimates for news consumption, discussing politics, political partisanship or voting, though we did see that joining the panel led to a slight uptick in voter registration.
Another important thing to keep in mind is that having a survey panel is not necessarily a guarantee of quality. How the panel is constructed also matters. Some panels are recruited using “opt-in” methods, meaning that people can volunteer to take part. These panels may suffer from data quality problems because of the presence of people who are there only for the compensation and are providing insincere or sloppy responses to questions.
The ATP is much less likely to have this problem because we do not recruit via opt-in methods. Instead, we’re the ones selecting people for inclusion in our panel, and we use random sampling to do that. (Read more about some of the pitfalls of opt-in polling.)
When you recruit new people to the American Trends Panel, what kinds of people are you looking for?
Our main goal is to make sure the panel represents the general public – that is, people who live in the United States and are ages 18 and older. At the same time, we’re interested in understanding the views and experiences of many smaller groups in the population, so it’s essential for us to have enough people from those groups represented on the panel.
To do this, we’ve been using our annual recruitment efforts to target certain groups that the panel may be underrepresenting. For instance, we “oversampled” Hispanic, Black and Asian adults in 2019, 2022 and 2023, respectively. This year, we oversampled young adults, people who do not have a college degree, and certain religious minorities, including Muslim and Jewish Americans. Oversampling allows us to report on the opinions of small groups.
It’s important to note that any numbers we report about the whole population are statistically adjusted so that smaller subgroups are represented in their correct proportions.
What changes have you made to the American Trends Panel recently and why?
We’re always looking for ways to improve the panel, but this year brought an especially important change. We’ll now be giving our panelists the option to take our surveys over the phone with an interviewer, in addition to online. This is mostly aimed at people who don’t have internet access or who don’t want to take our surveys online.
For the last eight years, we included these people by providing them with an internet plan and a tablet that they could use to take our surveys. But over time, we found this was not as effective at capturing offline Americans as we’d hoped. So we felt that offering a phone option would be an improvement.
Another change is that a new vendor, SSRS, has taken over the data collection and panel maintenance responsibilities this year. They’ll handle various logistical tasks, such as taking the questionnaires we write and programming them for administration on the web and phone; handling panel relations and ensuring panelists get paid for each survey they take; collecting and cleaning the data we gather and sending it to us; and so much more.
