On excluding polls

My goal for the polling analyses is to include all polls that legitimately provide insight into the race. For this reason, I want to comment on some polls that I have excluded from analysis.

As you know, I have predefined criteria on what polls can be included in my analyses. For example I exclude all on-line polls…period. And I exclude partisan polls that are selectively released. Typically this happens when a campaign releases a poll it commissioned. Even though the poll may be above reproach, including selectively released polls introduces bias to the meta-analysis of polls.

To see why, imagine a neck-in-neck race between Republican Bob and Democrat Steve. Both campaigns do polling and, quite sensibly, release those that favor their candidate. Furthermore, suppose Crossroads GPS is very interested in Bob race, so they do a bunch of polling. Steve’s campaign does a dozen polls across the campaign, and they release the six polls that show Steve with a slight lead. Bob’s campaign is a little more poll-obsessed, and they do two-dozen polls and release the 12 polls that favor their candidate. On top of that, Crossroads GPS does and additional dozen polls and releases six that show Bob in the lead.

We can assume that individual polls are above reproach. Even so, the selective release means that at a typical point in the campaign, there will be about three polls showing Bob in the lead for every poll that shows Steve in the lead. The meta-analysis will show Bob leading in a race that is, in fact, a tie.

I bring this up because I’ve had to exclude a couple of polls from my analyses.

Yesterday, Wenzel Strategies released a poll for Missouri that showed Romney leading Obama, 54.9 to 41.1. The poll appears to be legitimate and I have no reason to believe the poll is biased or improperly done. In fact, Wenzel does something I really appreciate: they publish the counts of responses in addition to the percentages. I use the counts directly whenever possible, but aside from Survey USA, most pollsters just provide rounded percentages.

The Wenzel poll was conducted on behalf of Citizens United that:

…exists to support true conservative candidates running for federal office through direct candidate advocacy and contributions, based on our in-depth candidate research and surveys.

So, it is obviously a right-leaning organization. But that is not enough to exclude an organization’s polls. For example, the Civitas Institute is a right leaning think tank in North Carolina. I include their polls, because they pre-announce the polls, and release the results regardless of the outcome.

I though the Wenzel polls were okay, because they were released on the firm’s web site. But, I wanted to be sure. I made a call to Fritz Wenzel to ask him about it. What I learned was the following:

1. The decision to publicly release the poll was made by his client, not him.
2. He only published the results on his web site after the results had been released by his client.
3. When I asked if he had done other polls for this client that had not been released, he deferred the question to his client.

From our discussion, I got the impression is that Mr. Wenzel is a serious pollster who takes pride in producing high quality polls for his clients. Even so, I’ll exclude this poll, since it appears to be a selective release by a highly partisan group.

Another poll I am excluding was brought to my attention by Sam Minter, for American Crossroads:

According to a new poll shared exclusively with POLITICO, Mitt Romney is leading President Obama in a congressional district in Maine — raising the possibility of the GOP nominee winning an electoral vote in a deep blue state.

The live-call poll, conducted by Glen Bolger of NMB Research and provided to POLITICO by American Crossroads communications director Jonathan Collegio, shows Obama winning statewide 48 percent to 44 percent.

That result already puts Romney in a surprisingly competitive position in a nominally blue state.

But in Maine’s second congressional district, Romney actually tops Obama 49 percent to 44 percent.

The ME-2 result is surprising but not a valid reason for exclusion. It is the selective nature of the poll’s release that results in exclusion. American Crossroads is engaging in good PR by releasing this information, but it isn’t likely to be representative of the polling overall. Granted, I did not call up the pollster, but the case for selective release seem pretty clear.

I point out these exclusions because I want to be as transparent about some decisions that may not, a first glance, be obvious.

Additionally, I’ll use this opportunity to ask for your help. If you see a poll I am including but shouldn’t, or a poll I’ve left out but should include, let me know. The best way is to make your case for inclusion or exclusion in a poll analysis comment thread.