More on those evolution polls

[mmcirvin]

I don't know why I'm so fascinated by opinion polls and the defects and manipulation thereof. Anyway, having written a post on some seemingly disturbing polls about public opinions of evolution a while ago, I'd be remiss not to point you to this excellent Skeptical Inquirer article by Chris Mooney about these polls that segues into a more general explanation of how subtle details of the wording and method of a poll can affect the results, either on purpose or out of sheer cluelessness. He also says a little bit about how polling organizations make money from advocacy groups that are clearly trolling for preconceived results.

The biggest risk with these things is a sort of game of Telephone. Often when you see these surveys fully described in context, with the questions spelled out in full and the methods made clear, the results don't seem as remarkable as they do when the numbers get cited with vague descriptions in an op-ed by somebody else, let alone when that gets cited by some talking head on TV or turns into an e-mail outrage item.

By the way, my favorite passage in the article is near the bottom. In a discussion of results from a poll about biology education designed to be worded more favorably to evolution proponents funded by Intelligent Design advocates (sorry, misread that one):

Indeed, the most startling thing about the poll was the fact that a largish 20 percent minority of respondents actually seem to think that biology textbooks should contain factual errors. Who the hell are these people?

McIrvin's Law of Surveys: At least 20% of the respondents in any survey will give totally absurd responses.

Comments

The real message here

[swinehund.livejournal.com]

Being nothing about which of evolution or intelligent design should be taught in schools, but about the need for more about statistical interpretation and sampling error.

I would also LOVE to see some basic epistemology taught in schools. I guess I'll have to be satisfied with The Matrix for now, though.

Re: The real message here

[mmcirvin.livejournal.com]

The phrase "statistically significant" so often hides mischief of some sort that people would be well advised to be wary of it without further qualification.

Re: The real message here

[swinehund.livejournal.com]

Well, yeah, but I'd just like to see the actual numbers they're working with (P-values, at LEAST) and their sample group. These should be things that are reported along with "Muskrat Oil New Miracle Cancer Cure" kind of headlines, as a matter of course.

Re: The real message here

[mmcirvin.livejournal.com]

Yeah, that was pretty much what I was trying to say. When I read claims that something or other is statistically significant, I want to know how many sigma the signal is, and whether that includes some estimate of systematics. Otherwise you get travesties like that Scandinavian powerline/cancer study where they looked at ten kinds of cancer, found that two of them had elevated incidence near powerlines at the one-sigma level, and claimed they had shown some positive result. Gee, you'd expect to find about three, wouldn't you?

[jmkelly.livejournal.com]

For a while one of my kids worked for an outfit that did phone surveys for political parties. She was amused by her bosses' insistence that she ask the questions exactly as written so as not to skew the results--when the questions themselves were written to skew the results.

[mmcirvin.livejournal.com]

Many of those are "push polls", in which the real purpose isn't even to collect data, but to get in some advocacy that might otherwise be limited by campaign regulations, in the wording of the questions. I think this is supposed to be illegal, but of course it's almost impossible to enforce.