President Alf? The Intricacies of Sample Size and Response Rate in Short-Form Research


ADOTAS – Statistics are one of the most abused forms of communication that exists in the market today. This has always been true, but it really has been hitting home for me recently. Just the other morning, I was rummaging around in the fridge for breakfast when I noticed some yogurt that boldly proclaimed to be the “no. 1 selling 60-calorie yogurt.” Looking at that yogurt, I thought to myself, how many 60-calorie yogurts are on the market?

Face it: we’re impressed by numbers, by stats and rankings, so naturally people go out of their way to manipulate those stats to make you believe what they want you to believe. Politicians have been doing it for hundreds of years, marketers since the dawn of the advertising claim, and now even researchers get in on the act.

This week I visited a client who’d decided to purchase some short-form ad effectiveness research using a vendor that employed a one-question approach to measurement. Now if you’ve ever done online ad effectiveness research in the past or read some of our previous posts on the subject, you’ll know that you need to weigh the data you collect in these studies to ensure you have a matched test and control cell.

Without weighting the data, the results can (and often are) misleading and patently wrong. But wait (no pun intended) — apparently, weighting is not required because of a higher response rate, as this vendor has claimed. This assertion makes the researcher in me cringe. It is pure manipulation of information, particularly because it confuses the concept of responses with response rate.

The Sampling Rate Situation

Before I get into this, let’s start with a scenario. We are standing in front of a door to a room. I tell you that inside that room are 100 people and that each person has a favorite color. I’m planning a party for these folks and I’d prefer to choose the theme color that is the same as the favorite color of the majority of the people in the room. Here’s your job: go into the room and figure out the color I should use.

But there’s a catch. I want you to ask as few people as possible before leaving the room and telling me the most common favorite color. So the question you have to ask yourself is: what is the smallest number of people I can talk to and feel comfortable that I’ve determined the most common favorite color?

Hold on to that number for a second. Now figure out the same number for a room with 1,000 people or 10,000 people.

Whatever numbers you’ve come up with, those are the number of responses you’ll get (I’m making the assumption everyone will answer your question). The number of responses is the raw data; it’s what you add up to get your answer to my question. Now, if you take the numbers you came up with and divide them into the number of people in the room you’ll get your sampling rate.

So if you needed to talk to 25 people to figure out the favorite color of 100 people, that’s a 25% sampling rate. If you needed to talk to 200 people in the room with 1,000 people, that’s 20%, and 500 people for the room with 10,000 is 5%. Notice, though, that the percentage of the people I sampled in these three scenarios gets smaller. There’s a mental trick at play here which is that we’re much better at putting things into perspective with tangible numbers.

So 25 out of 100 might seem at a quick glance to be just as accurate as 500 out of 10,000. This is because we can’t imagine a room of 10,000 people. But logically thinking it through, if we wanted to have the same level of accuracy in both rooms we’d sample a uniform 25% (25 of 100 and 2,500 of 10,000). Makes sense, right?

The Response Rate Angle

Now thinking of that 100-person room, what was your number? Was it 10, 20, 50? I’m guessing it was somewhere between 25% and 50%. If you use that as your accuracy rule of thumb, let’s now transition to the idea of response rate. The response rate to any survey is the number of people who complete a survey divided by the number invited. In this scenario, the number completing is akin to the number of people asked about their favorite color, and the number invited is akin to the number of people in the room.

Your typical response rate for an online pop-up survey is less than 1%. So for argument’s sake let’s pretend it’s 1% — how does that compare to your number from my example? I’m assuming it’s significantly lower. How confident would you be walking into that room and asking 1 person in 100 for their favorite color and assuming that opinion represents everyone in the room?

There’s the big challenge facing survey research in general… response rates are low. Research has shown that by asking one question you can increase that response rate to 3% to 5%. But I’d still make the same point: is 3 or 5 people out of 100 enough to accurately answer your question? Nope, not by a long shot. What happens when you speak to fewer people? Well the chances that you’re wrong increase significantly.

How do you ensure accuracy? In ad effectiveness studies we make sure we’re comparing apples to apples and that requires — you guessed it — weighting. Unfortunately, you can’t weight a 1 question survey since you haven’t asked any questions to drive the weighting, and while you may have higher response rates, you’re still not close to being in the territory where you can avoid weighting.

Where the spin happens is in the confusion between response rates and responses. With any 1-question questionnaire I can get a lot of responses, but without the right response rate, my data can still be inaccurate. Ten thousand responses on 1 million impressions sounds like a lot of people until you put it into the context of my fictional room of 100 people — then it’s still just 1%.

This is sampling 101 stuff, but it’s a lesson we learn the hard way every time. Probably the best example of assuming more data equals accuracy is the 1936 presidential election where the Literary Digest surveyed 2.4 million people and predicted Alf Landon would beat Roosevelt. Since most of us only know of one Alf, and he’s a hairy TV sitcom monster from the 80s, it’s obvious that Literary Digest was wrong.

Two morals of this little story:

• All surveys have bias no matter how big the sample, so make sure you’re taking those biases into account and do what you can to eliminate them (i.e. weighting)

• Don’t be fooled by spin! Needless to say, in my household we’re now purchasing the no. 1 selling 58-calorie yogurt.

This piece is the latest in a series of articles by InsightExpress’ Marc Ryan about evaluating and buying online ad research. The first can be found here.



Please enter your comment!
Please enter your name here