With a hand recount looming in our historically close gubernatorial election, there has been much debate over the relative accuracy of hand counts versus machine counts, and the error rate of vote counting technologies in general… most of it uninformed.
In my typically wonkish fashion I decided to dive into the most technical research I could find, and tediously share my gleanings with you. My primary source is the CalTech/MIT Voter Technology Project, and much of my data is drawn from the following reports:
Conflicting numbers regarding the error rate of voting machines have been tossed about in partisan blogs, the news media, and the John Carlson Show (which doesn’t really fall into either category.) On the one hand, the certification standard for voting machines in Washington state is indeed one failure in one million. But it is also true that the residual vote rate — the primary statistical measure of the performance and accuracy of voting technologies — is one to two votes in one hundred.
The residual vote rate is the difference between total ballots cast, and votes counted for a particular office, such as president or governor; these residual votes represent the “over” and “under” votes we hear about. And it is this measure that is significant in comparing competing technologies.
The fact that the counting machines themselves are virtually flawless is meaningless in the context of the larger discussion, because they are only flawless in counting flawlessly prepared ballots. For example, both optical scan and punch card machines will test to the same high certification standards, yet the average residual vote rate for punch card ballot systems is nearly double that for optical scan.
The large performance difference between the two systems is due, not to mechanical failure, but to a higher rate of “human error” by those voters using the punch card ballot system versus those using optical scan. This is a reflection of the way people relate to the particular technology… a concept that should be well familiar to user interface designers, or anyone who has ever manned the technical support line at a software company.
Indeed, Direct Recording Electronic devices (DREs) — including the touch-screen voting machines to which many counties are switching — have amongst the highest residual vote rates of all existing technologies, despite the fact they are programmed to make over-votes (voting for multiple candidates in the same race) impossible, and always tabulate ballots with 100% accuracy. (Or so Diebold tells us.) Meanwhile, hand-counted paper ballots, which provide no safeguard against over-votes, and risk introducing human error into the counting process, have one of the lowest residual vote rates.
Thus it is not the counting machine that introduces statistically significant error into the voting system, but rather the interface by which voters are asked to mark the ballot. It has become popular for bloggers and columnists to criticize voters for not following instructions, but when one voting technology produces error rates twice that of another, the technology deserves part of the blame. And clearly, voters in counties using punch card systems are being disenfranchised at rates nearly twice that of those in counties using optical scan… and at a rate well outside the margin of error in our gubernatorial election. (I’ll get to that in a moment.)
As to the relative accuracy of the various voting technologies, the CalTech/MIT studies found that voting systems fell into two clusters: paper ballots, lever machines and optical scanned ballots produced residual voting rates of one to two percent. Punch card and electronic voting methods produced rates of approximately three percent.
So, are hand counts accurate? According to the studies, hand counts are at least as accurate as lever machines and optical scanners, and significantly more accurate than punch card and electronic voting systems. In fact, when looking at counties that switched from one technology to another and comparing the resulting residual vote rates, the study found that while the overall results were consistent with other analyses…
Paper might even be an improvement over lever machines and scanners.
CalTech/MIT also explores a second measure of accuracy, tabulation validation rate — the agreement between initial counts and recounts of ballots in contested elections. This metric is less useful as a comparative tool, because it cannot be used to measure mechanical (lever) and electronic voting machines, as there are no ballots to recount. And the study in question only compared hand-counted paper ballots with optically scanned ballots.
The study found the tabulation invalidation rate was .83 percent for paper and .56 percent for optical scanning. Thus the discrepancy between the initial count and the recount was less for optical scan than for hand-counted paper.
To be honest, I’m not sure what, if anything, this says about the relative accuracy of hand recounting optical scan ballots, let alone punch cards. And here’s the part that might piss some people off… I’m not sure it even matters.
For what this study does tell us is that even the most accurate voting technology still is not accurate enough:
Considering these tabulation errors, how confident should we be in vote counts, and when should we have a recount? The tabulation invalidation rate was low, especially for optical scanning. However, it was not trivial. In a US House election with 250,000 votes, the invalidation rate of .005 for scanners amounts to 1250 votes. The tabulation errors may swing toward any of the contestants in a recount. Assuming a uniform distribution of tabulation errors, any race decided by less than .5 percent of the vote will have a non-trivial probability of being reversed in a recount.
A .5 percent invalidation rate in a gubernatorial election with over 2.8 million votes cast amounts to 14,000 erroneous votes! With only 42 votes separating the two candidates, no counting method can accurately tell us who really got the most votes.
Republicans scoff at Gregoire calling this election a tie, but statistically speaking, it is. This election is so far within the margin of error, that there is no practical way to accurately determine the winner.
Thus, the results of the third count — whether hand or machine — will be just as meaningless as the results of the first two.
Republicans argue that “winning” the first two counts gives Rossi legitimacy. It doesn’t. It’s like flipping a coin and having it land on heads two times in a row.
And Democrats argue that a hand recount will more accurately determine the winner. It can’t. This race is simply too close to call. We’ll never know who really got the most votes.
Fortunately, the law does prescribe an endgame. Gregoire will request a hand recount, and whoever “wins” that, will be governor.
And who do I think is gonna win? Flip a coin.
[Linking here from (un)Sound Politics? You can comment directly on Stefan’s critique, or join a discussion here: “Lies, damn lies, and statistics.”]