Garbage In, Garbage Out: Are Republican “Pollsters” Gaming Poll Aggregators with Bullshit “Polls?”

There’s been a lot of bedwetting in recent days over a flood of new polls that threatened to undermine the foundation of Hillary Clinton’s once formidable electoral college firewall. Key swing states like Florida, Nevada, and North Carolina, which for weeks had shown small but consistent leads for Clinton, suddenly plunged into Trump territory, while once solid leads in New Hampshire, Virginia, and Pennsylvania appeared to melt into the margin of error. As a result, polling aggregation sites from Nate Silver’s widely-followed 538 to Darryl’s less well known but equally pedigreed analyses here on HA showed Trump’s chances of winning surge into the disturbing realm of the remotely possible.

But what if some of the polls that introduced all this late uncertainty weren’t just wrong, they were, well, intentionally wrong?

Ignore for a moment the reasons why, but if your goal was to game the polling aggregation sites you would probably want to wait until the final weeks of the election to release a flood of new polls with absurdly large sample sizes (and absurdly low margins of error) surveying a handful of key swing states while providing little or no prior track record on which to judge your credibility. In other words: Meet the Remington Research poll.

Coming out of virtually nowhere, the Remington Research Group in collaboration with Axiom Strategies suddenly emerged as one of the most prolific pollsters of the fin de cycle électoral, simultaneously surveying eight “battleground” states on October 23, October 30, and again on November 3. This sudden displaying of polling prowess is particularly remarkable given the incredibly large size of its voter sample—and by “incredibly,” I do mean that these numbers are not credible.

For example, Remington’s November 3 poll (surveyed Nov 1-2) includes and analyzes the responses of 19,640 likely voters across eight states, ranging from a robust 1,793-voter sample in Nevada to an inexplicable 3,076-voter sample in Virginia. Compare that to the ten most recent Virginia polls posted to 538:

The second largest sample size of the group is SurveyMonkey’s 2,109, and even that’s an outlier given that it’s an online poll that typically uses larger samples in order to compensate for selection bias and other errors. Remington claims to be conducting its surveys via IVR (interactive voice response) technology—good old fashioned robocalling—a method that normally would suffice with about a 1,000-voter sample in a state the size of Virginia. A similar pattern can be seen in survey after survey, state after state, with Remington standing out by far as the largest sample size in any state (and generally one of the most Trump-leaning).

Darryl continues to include Remington in his analysis because it fits his inclusion criteria, and he correctly does not want to change criteria mid-cycle. But those large sample sizes immediately jumped out at Darryl as odd. As Darryl explains, response rates for IVR are notoriously low—probably around 10 percent. That means Remington would have needed to dial about 200,000 voters over two days to achieve its 19,640-voter sample, and then analyze all these results. That may not be beyond the capabilities of a large, established pollster, but it’s not the sort of operation one builds overnight. Further catching Darryl’s eye is Remington’s unusually low reported margin of error: “+/- 1.77” in that Virginia poll, for example. That’s not only weirdly low, it’s weirdly precise; experienced pollsters don’t tend to present that number out to the second decimal place, says Darryl.

Remember: polling is expensive. A typical 600-800-voter IVR survey might cost a campaign or media outlet about $8,000 or more, and a live interviewer telephone survey would cost several times that. (A high-quality live-interview bilingual internal poll of the sort statewide campaigns use might cost $30,000 or more.) So why would Axiom/Remington put in the extra time and money to survey four times as many voters as the typical public state poll?

One obvious answer would be to game the average margins calculated by all these polling aggregators everyone is obsessing on. I’m way oversimplifying it, but the way these polling aggregators generally work is that they add up all the samples from various polls over a specified period of time, and then do some math to them. They essentially create one giant poll out of many in an effort to smooth out the variation and find (as Silver would say it) “the signal in the noise.” (Darryl could explain it better, but in many more words. If you’re curious, you can read his entire simulation FAQ here.)

And what that means is that Remington’s 3,076-voter survey that had Clinton at just +2 influences the average about 4.7 times more than Roanoke College’s 654-voter sample that had Clinton at a robust +7. (Well, not exactly. But again, ask Darryl.) In fact, under most models, Remington’s extremely low reported margin of error would overweight its impact even more.

Thus, flood the media with some very-large-sample Trump-leaning polls, and the state of the race in some of these battleground states starts to look very different. For example, as of last night, Darryl had Clinton with only a 27.9 percent probability of winning Florida—but remove the Remington polls and Clinton’s probability jumps to 68.2 percent! In other words, a single pollster can flip Florida. Likewise, Clinton’s probability of winning an Electoral College majority would’ve climbed from 80 percent with Remington to 94 percent without. For many nervous poll-watchers, that’s the difference between wet sheets and dry sheets.

But what’s in it for Remington? Well, to know for sure, you’d have to ask the Remington Group. Or maybe their partner, Axiom Strategies.  Or better yet, Ted Cruz’s former presidential campaign chief strategist Jeff Roe, who happens to own both the Remington Group and Axiom Strategies. But I have three theories:

1. Bump up turnout in support of down-ballot Republicans. As the Princeton Election Consortium’s Sam Wang recently explained, “increased confidence is associated with increased turnout.” Sure, Trump may be a lost cause, but as long as Trump supporters don’t know it, down-ballot Republicans won’t suffer quite as much.
2. Undermine public faith in the outcome of this election.  If Hillary Clinton substantially over-performs the polling averages it will look awfully suspicious to voters already primed by Trump’s relentless voter fraud allegations. I mean, Nate Silver has never been wrong before, so why would he miss a few key states this time around? Such suspicions will serve Republicans well as they continue their efforts to delegitimize Clinton through hearings, confirmation battles, and ultimately impeachment.
3. Fuck with Nate Silver. Silver has been the target of a lot of well-deserved criticism this cycle for all the click-baity volatility he built into his new (and opaque) models—but four years ago he was a goddamn hero for the way he championed data-driven journalism over the usual data-free kind. Republicans hate Silver for interfering with their election narrative. So maybe this is just payback?

Whatever the reason, the fact is that these Remington polls just don’t pass the smell test. (Clinton is winning only 65% of African-Americans in Virginia, and only 48% of Hispanics in Nevada? I don’t think so.) And Remington isn’t the only late-comer to flood the swing state polling swamp with suspiciously pro-Trump results. Republican consultancy shop Trafalgar Group (infamous robocaller and former Chris Christie advisor Robert Cahaly) has generated a wave of weirdly-worded surveys that purport to reveal a “shy Trump-voter” demographic that every other pollster has missed! Trafalgar too calculates its margin of error to the second decimal place, a level of precision that feels out-of-place amidst their outlier results. (Trump is up 2 points in Michigan, or something!)

Yeah, sure, all of the aggregators will ultimately call this election correctly, in that Clinton will win by some margin or another. And at this late stage there’s little if any argument over who is leading where. But I’ve no doubt that Republicans are attempting to game the system in a way that makes this whole poll aggregation endeavor far less useful than it was in the past two presidential cycles. So don’t be surprised on Tuesday night if the presidential election turns out to be less competitive than we’ve all been led to believe.