Web polls.

Dear Professor Mean, I'm conducting a number of web-based polls and would like to include a margin of error in my results. How do I do this?

I've got bad news and more bad news. First, the traditional formula for the margin of error will not account for bias and will grossly underestimate the uncertainty of your estimates. Second, we can't quantify how much bias you are likely to have.

When you are conducting any poll, the first question you ought to ask is "what population am I trying to generalize these results to?" For example, if you ask the question "Would you vote for Rush Limbaugh for president of the United States in the year 2000?" the only logical population is all U.S. voters.

In just about every case that I can envision, the population that you might be interested in extrapolating to would be severely misrepresented by a web-based sample. First of all, your web-based sample is going to under-represent people who don't own computers. Second, your sample consists of volunteers, who usually hold much stronger viewpoints than the average person.

There are ways that you can estimate and adjust for bias. For example, in some polls you can characterize the demographics of those who didn't respond, and if those demographics are comparable to the demographics of the sample itself, then we might feel safer in claiming that there is no bias. Other times, we can get a rough feel for what non-respondents are like by using follow-ups and reminders, and seeing if those who responded only after the reminder differed from those who responded right away. Finally, if you know exactly how much certain groups are over or under-represented, sometimes you can weight the results by the amount of underrepresentation to get an unbiased estimate.

I can't imagine, though, how you would apply any of these techniques to your web based sample. My advice is to use the same disclaimer that you see on these psychic hot-line numbers: "This poll has no scientific value and is intended for entertainment purposes only."

Further reading

1. Methods for the design and administration of web-based surveys. Schleyer TK, Forrest JL. J Am Med Inform Assoc 2000: 7(4); 416-25. [Medline] [Abstract] [Full text] [PDF]
2. Comparative response to a survey executed by post, email, & web form
3. Mail and Internet Surveys: The Tailored Design Method. Dillman DA (2000) Canada: John Wiley & Sons, Inc.

Category: Ask Professor Mean, Category: Research designs