Damn Polls

The distinction is critical, however, because there is an intrinsic fluctuation caused by the way polls determine who is and isn’t a likely voter. “The problem is — and this is particularly true of Gallup — that they use an elaborate screening mechanism,” explains Ruy Teixeira, a fellow at the Center for American Progress who has been a vocal critic of the likely-voter model. “Gallup asks seven questions that try to gauge involvement and interest in the campaigns. This captures people who are most energized by the campaign at any given time. And that has party implications, because at different times, different people are energized.”

Again, the period after the RNC stands as an example: Bush supporters are excited, so more Republicans than Democrats turn up in the likely-voter samples. “This means,” says Teixeira, “that much of the movement in the Gallup Poll is due to who is counted as a likely voter rather than movement in public opinion.”

Gallup recognizes that their likely-voter model encourages wider swings — but they say that swing is what they want to measure. “Gallup looks for movement,” explains Frank Newport, the editor of the Gallup Poll. “Ours is more sensitive than other polls, so that we can see changes.” Jim Norman, the polling director for USA Today, which along with CNN has an exclusive arrangement with Gallup, also acknowledges that their system “at certain times exaggerates the turnout from one party or another.”

Kerry partisans note that the exaggeration this year has favored the Republicans. Some have been making the conspiratorial observation that Gallup’s CEO, Jim Clifton, is a Republican donor, and they suggest that the organization puts its thumb on the scales. Every polling professional I spoke with dismissed that theory out of hand. But in this election cycle there is no question that for whatever reason, Republicans have been overrepresented by Gallup and other likely voter polls.

Take the Gallup report with Bush’s 13-point advantage, for instance. Here was the party identification among likely voters, with 7 percent more Republicans in the sample:

GOP: 305 (40 percent)

Democrat: 253 (33 percent)

Independent: 208 (28 percent)

Actual voter turnout, however, has never looked like this. In 2000, the proportion was even: 34 percent Republican, 34 percent Democrat, and 33 percent Independent. For the 20 years previous, more Democrats than Republicans voted in national elections. Adjusting Gallup’s sample composition to reflect that kind of turnout in 2004 would slash Bush’s lead. That would bring it more in line with Pew’s poll, whose sample had slightly more Democrats than Republicans.

Gallup’s explanation for the GOP tilt in their sample is that they don’t weight based on party identification because they view that as a fluid attitude rather than a firm characteristic like gender. “We do carefully weight the sample for hard demographic data like age, region, ethnicity and so on,” says Newport. “But people change their minds about affiliation, so we treat that as a variable that moves.”

Michael Dimock, the research director for Pew, said the same about their process. “We don’t weight for party ID either, because it shifts from one poll to the next,” he said.

But if both the Pew and Gallup polls were conducted with similar methodologies over mostly overlapping time periods, I asked, why would they have such different samples and results? “Yes,” Dimock responded, “that’s a good question.”

Neither did Newport have an explanation. But the discrepancy was caused by something. It could have been the likely-voter screen, or some other bias in the questions or time frame. Or the culprit may be what professor Vavreck called “the reality of sampling,” which is that “You just don’t always get a great draw.”

THIS MAY HAVE BEEN the case for the Gallup Poll, which looks like what statisticians call an outlier — a technical term for sampling error. Defending his poll, Frank Newport said that Gallup’s numbers were not out of line with other polls, none of which tilted toward Kerry — which is true, but it is also true that no other poll showed Bush with a 13-point lead.

The true state of opinion, most likely, is somewhere in the middle. “You need to look at a preponderance of the evidence,” professor Vavreck advises about gauging the information contained in polls. “Look at several and get a general sense.” Newport himself suggested doing the same, and cited the Web site Real Clear Politics, which compiles an average of available polls. Their current estimate: Bush is up by 5 points.

This is not, unfortunately, how the media usually approach poll numbers. Not only do news stories never question — or even describe — methodology, the reporters rarely create context by noting that the poll they’re reporting is just one of many.

And herein lies yet another systemic polling problem: Each network or paper commissions its own survey and then acts like it got some kind of scoop. They want to get their numbers out first, and tout them as news, so they don’t mention that the other guys’ numbers are different. The competition between proprietary polls forces the media, when reporting on the extremely sensitive state of national elections, to be essentially un-journalistic: They use only a single source — their own poll — while ignoring other, potentially conflicting information.