Guest Blogged by Mitch Trachtenberg of the Humboldt County Election Transparency Project
California Secretary of State Debra Bowen has released a remarkable 13-page report [PDF] of her office’s investigation into how the Diebold/Premier GEMS software silently dropped all votes contained on 197 ballots from the Humboldt County, California, November 2008 general election.
The report shows that this version of GEMS not only deleted a batch of ballots without any request by — or alert to — the elections staff, but also failed to note the deletion in the system’s audit log. The report also points to other startling deficiencies with Diebold/Premier’s software: “Key audit trail logs in GEMS version 1.18.19 do not record important operator interventions such as deletion of decks of ballots, assign inaccurate date and time stamps to events that are recorded, and can be deleted by the operator.”
That’s right. The Diebold/Premier vote tabulation system in question not only fails to record all events accurately, and sometimes at all, it also allows anyone with access to the system to completely delete audit logs, covering the tracks of any tampering that may have occurred, at any time, on the system.
Any of these flaws, the report concludes, “appears to violate the 1990 Voting System Standards to an extent that would have warranted failure of the GEMS version 1.18.19 system had they been detected and reported by the [federal] Independent Testing Authority [ITA] that tested the system.”…
Discovering Diebold’s Failure…
I’m proud to be a member of the Humboldt County Election Transparency Project, the volunteer group that Humboldt County Clerk and Registrar of Voters Carolyn Crnich invited into her office to conduct an independent scan of all ballots cast. The project, proposed by Humboldt commercial fisherman Kevin Collins, and founded by Collins and Crnich along with Tom Pinto, a tech staffer at the district attorney’s office, David Cobb, a former Green Party Presidential Candidate, and Parke Bostrom, a concerned citizen. I was there when we ended up scanning more ballots than were counted in the official results. I thought we’d made a mistake ourselves.
I’d developed an open-source vote counting program called Ballot Browser, and I ran it on our collection of scans the night we finished scanning, to try to hunt down our mistake. The transparency project had also kept careful logs of our scanning. With a few exceptions, those logs would also tell us how many ballots we’d scanned for each precinct. (The exceptions were a few mixed batches; most of the ballots were batched by precinct, but not all.)
When I got Ballot Browser’s results and compared them with the official count, precinct by precinct, I was able to quickly point to precinct 1E45 as having 197 more ballots in Ballot Browser’s count than in the official count. The discovery would lead to corrected, re-certified election results, the CA Secretary of State’s investigation and recent finding, the likely decertification of the Diebold software in question, and — hopefully — decertification in the approximately thirty other states which use the same software, or a version with the same bug, to count ballots cast in their elections.
When the apparent mistake in the Ballot Browser’s results orignally came to light, Carolyn Crnich was checking both the official Diebold scanning logs and our independent scanning logs and also narrowed in on precinct 1E45. It looked like, somehow, a batch of 197 vote-by-mail ballots from precinct 1E45 had been scanned but then deleted. The deletion of this batch, not-so-coincidentally the very first deck of vote-by-mail ballots that the elections staff had scanned into its Diebold/Premier equipment, seemed to have taken place after the election night results, but before the official numbers were certified.
It turned out that the problem was not that the Transparency Project had mistakenly double-scanned a batch of ballots, and not that the elections staff had somehow missed a batch of ballots. The problem was a bug in Diebold’s GEMS software: under fairly common circumstances, the first deck of ballots might be silently deleted, with no notice given to election officials overseeing the computerized tabulation.
Since this news broke last December, some have questioned why normal ballot reconciliation procedures didn’t flag the dropped ballots. The Secretary of State’s report makes this clear [emphasis in original]:
It was only later, after the GEMS Central Count Server was re-opened and new decks of vote-by-mail ballots that had been received on Election Day were tallied for the first time, that Deck 0 was deleted, without any warning or notification to the elections official, as a result of the software programming flaw. Because the deletion of the votes from the 197 ballots in Deck 0 occurred long after they were counted and after repeated reports showed them properly accounted for, nothing indicated any need to recheck the earlier reconciliation for a third time.
It would be reasonable to think that the Humboldt County Election Transparency Project had discovered a previously unknown bug in GEMS. Reasonable, but wrong. The Secretary of State’s report confirms what The BRAD BLOG reported in one of its follow-ups last December: the error was long known by Diebold.
As the Secretary of State’s report now confirms:
The email and attachment did not inform the elections officials that failure to follow these instructions would likely result in deletion of tallied votes by GEMS without any warning or notice to the system operator. The email and attachment also failed to inform counties that it was a programming flaw in the GEMS software that made the special instructions necessary. The chief elections official for Humboldt County, Registrar of Voters Carolyn Crnich, states that she never saw the email or the attached instructions. The former county elections officer apparently had received the email in 2004, but left Humboldt County in 2007 to work in another county’s elections office. That county elections officer did not, according to Registrar of Voters Carolyn Crnich, pass the information along to her or anyone else in her office. This helps to explain why the Deck 0 software error manifested itself in the November 2008 election.
Diebold Audit Logs Also Found Useless, Gameable…
It is stunning enough that the lost ballots flaw, known to Diebold/Premier since 2004, has been allowed to remain in versions of GEMS still in use. But there turn out to be more problems, arguably more serious than even dropping ballots, as noted in the SoS report:
The problems with the audit logs, apparently undetected during the federal certification of GEMS by EAC/NASED, call the entire certification process into question.
I don’t think it’s possible to overstate the importance of 100% confidence in a voting system’s audit logs. Without confidence that a voting system is providing what the Federal Election Commission’s 1990 voting system standards call “a concrete, indestructible archival record of all system activity related to the vote tally,” there can simply be no confidence in the reported results of an election.
That a “Clear” button was found in the software, capable of deleting an entire audit log, without even confirmation of the clear required by the system operative, is unimaginable in such a system, not to mention, in strict violation of the voting system standards under which this system had, several times, been certified by the federal so-called Independent Testing Authority (ITA).
When election integrity advocates have criticized touch-screen voting, the touch-screen vendors have pointed to the audit logs, and how any tampering will show up in them. And yet here is a system, federally certified, where the audit logs are not only undependable, but can be (and have been!) inadvertently deleted by an elections staffer.
Any of the problems that have turned up, according to the Secretary of State’s report, “would warrant a finding by an Independent Testing Authority (ITA) of ‘Total Failure'”. But that never happened despite multiple federal certifications of both this GEMS system, version 1.18.19, but also “subsequent GEMS versions (1.18.20, 1.18.21, 1.18.22 and 1.18.23) that contain the same software error,” as the report notes.
Although the Secretary of State’s report is written in calm language, it is a bombshell. In Humboldt County, nearly two hundred citizens were nearly disenfranchised by a four year old bug in a system that had passed federal certification.
Fortunately, Humboldt County still uses paper ballots, so a recount was possible.
Fortunately, Humboldt County Clerk and Registrar of Voters Carolyn Crnich was willing to stick her neck out and allow a group of volunteers to come in and conduct an independent recount.
Fortunately, once the Humboldt County Election Transparency Project turned up a four year old bug in GEMS, California’s Secretary of State Debra Bowen conducted a thorough investigation.
Fortunately, that investigation was able to discover what those who conducted federal certification did not: the lack of integrity of the GEMS audit logs.
Does this mean the system worked, or does it just mean that this time, we lucked out? In how many jurisdictions nationwide would the ballots have been recounted, even though no candidate questioned the results? In how many jurisdictions nationwide would there even have been paper ballots to recount?
How many times has the system not worked? If the audit logs aren’t rock solid, how can we ever know?
Our votes are too important to be counted by secret software running on black box machines.
===
Mitch Trachtenberg is a member of the Humboldt County Election Transparency Project, and the author of Ballot Browser, free and open source ballot counting software. He is also a partner in Trachtenberg Election Verification Systems (TEVSystems), which provides support for Ballot Browser.
Is that software rigging 101 or 202?
Malicious intent
OMG why isnt this the lead story on every network in america????!!!!!!!! thirty states…gameable voting machines. this is nuts!
This was and still is huge. It is beautiful in its simplicity. Perform a full public audit and find the errors. Investigate fully and find how illegal, error-prone and easily manipulable the secret vote-counting GEMS system is.
OK, so now the entire nation knows, right? And we’re going to trash that corrupt POS GEMS system and hold its owners/makers accountable and all the failed certification groups accountable, right? RIGHT?
It is clear that NASED and the ITA lab were negligent in their duties. No less culpable was the so-called testing of GEMS 1.18.19 by the sole consultant who has been repeatedly hired by the state to test this voting system. This testing was supposedly done in collaboration with the Secretary’s Election Division. Both the Secretary’s staff and the consultant, Steven V Freeman, issued reports stating that the voting system met all the requirements of law. This supposed testing was done in the fall of 2003.
The Procedures for Use, a requirement under California law, both before this version of GEMS, and subsequently, required:
“5.8. Securing audit logs and backup records
Procedures should be in place to insure that all audit logs are retrieved and retained and back up copies of all records should be retained as part of the official election. Audit logs from the OS and TS units should be retained and may be printed at the elections office as part of the semi-official canvass. Audit logs from the GEMS server should be printed and retained as part of the official records for the time period required by law.” (from Procedures for Use prior to GEMS 1.18.19)
“12.6. Print audit logs
Following election close, Audit Logs should be printed from all GEMS transmission consoles, then filed. Where possible, these logs should also be saved following election close. Audit Logs should not be deleted from GEMS until the election has closed and Audit Logs have been saved offline, printed, and filed into the election archive.”
(from Procedures for Use 2005).
It is interesting that in the screen shot illustrations in the 2005 procedures the clear button is missing. It also says that in February 2004 the audit procedures included an archiving feature with the print function.
The Report also discusses the claim by a local election official to “accidentally” deleting an audit log by hitting the clear button instead of the print button. Notwithstanding the obvious difficulty with accidentally doing that based upon the distance between the two buttons, there are several other audit logs that could have been printed out to compensate. I would suggest this was more an effort by an election official to hide audit records from the public than an accidental mistake.
I for one am tired of how local election officials are repeatedly allowed to explain away discrepancies, omissions, blame poll workers or human error, hide records such as audit logs, or simple computer “glitches” without questioning by the media or the Secretary’s office. The instances of election officials preventing citizen oversight, disobeying the law, and intentionally hiding what they are doing or any problems is amply documented. We should demand the same accountability of how our elections are run as we do the financial aspects of local government. For too long local election officials have acted with impunity when seeking to cover-up their incompetency, poor decisions, or protecting their voting system vendor from accountability.
Their duty is a sacred trust, and everything they do should be completely open to the public, provable, and accountable. These are our elections, not their’s.
i got my tin hat on and i dont care who knows… is it so far fetched to imagine that someone might discover that the easiest way to control and exploit (to your corporations advantage) the most powerful army on the planet, was to steal what average joe had already discarded. his sacred right to vote. most of us don’t care to vote. we take it for granted. i do mean “we” as a nation. wholesale theft of a nation. aided by voting machines. owned by my corp buddies. voting machines that count secretly. machines that erase votes without detection with a simple touch of the CLEAR button. BRILLIANT! whooda thunk it? why didn’t i think of it! damn
A CLEAR BUTTON on the AUDIT LOG?!! Christ! I’m REEEEeeeling in the yeast, here. Yet another twisted, tricky-sick layer to this fuckery. What a bottom-less tar-pit of (proprietary) treachery ’tis the Diebold GEMS operating system.
Stunning follow up report, Mitch T.!~ Gracious thanks to you and all the Humboldt Heroes for this slightly genius job with your break-through open source ballot tracking system. Fascinating that you didn’t quite know what a Marlin you’d hooked!
It’s gracious of you to say, but no this one’s not just luck (can’t be-we’ve not ridden the luck tsunami much in this never ending trog.) Nay, this fruitful (‘tho dire) resulting evidence is the perfect grouping of talent, unrelenting perseverance, technical will and a dash of (rare) political capital. Unlikely perhaps, but still the result of years of heavy lifting done by so relative few. (Well, that and the fact that these machines are so fatally flawed that if you get within a hundred feet of one with any kind of tracking device — odds are, you’re ‘gonna need a bigger boat’…)
Having a force the likes of a Debra Bowen is a nice touch, (I always think I catch a whiff of our old friend Lowell Finley in her missives); and Carolyn Crnich must be a remarkable woman. Songs should be written in her honor and a giant goblet of wine couriered over asap. As for you and the rest of your team, where can we send you a nice fruit basket? Maybe some nice soap?
Incredible reporting/ grand initiative, Mitch. Humboldt County must be a wonderful place.
John (#3)
HELLyesthisshouldleadEVERYdamnNETWORK. Righto.
Thank you BradBlog, always, for telling me news.
and
Jody (#5) and Lora (#4)~Yes, I also wonder ‘now what?’ Or, more precisely, ‘What’s it gonna take?’…
As a computer software dev for over 30 years, I wouldn’t trust these things any further than I could throw them. I bet I could hide code in somthing like this that would skew votes however you wanted them and be virtually undetectable and programmable through the standard voter touch screen. Paper ballots electronically scanned and then archived seems to be the best way to really do this, IMHO. As a software tester for over 10 years, I have no respect for the tool that actually “tested” this mess, either.
Is this a solution? http://www.heliosvoting.org/
Will this make ANY “news” outlet in America, other than The Brad Blog and a few other sites? No, because they don’t want you to know about it.
Diebold is part of the mass media monopoly and so are the “two” parties. Look at Connecticut. Even after all of the Diebold bad news what does the Secretary of State do? She chooses Diebold!
We need paper ballots with receipts. Elections should be able to be audited by anyone at anytime.
Thank you to all who have commented, especially “Jeannie Dean in LA-13” — I wish I could write like you!
If you want to see this hit mainstream media, I have some suggestions: call your local newspaper or radio or TV stations and tell them about the story.
Find the person responsible for covering local elections, and give them the link.
Find the person covering their crime beat, and give them the link.
Find the person covering their “quirky tech stories” and give them the link.
Email them the text of the article.
Email them the text of the Sec of State report.
Print it and fax it to them.
If you want this on a TV station, go to your seat of government with something visual: a big cookie monster eating votes? (It would be great to be dignified, but it’s probably more important to make a big visual statement if you want TV news coverage.) Have a handout with the critical aspects of the story reduced to ten words: I DON’T WANT MY VOTE LOST BY DIEBOLD.
It looks like they trusted the completeness of their media control when they allowed the CA Secretary of State to conduct a proper investigation.
Trust me, no one will ever hear about this, except you lucky readers of the Brad Blog and maybe a few other sites.
Watch the movie “The Brotherhood of the Bell” and you will understand that our society not only can be but actually is highly controlled. You’ll see.
Let’s just hit the “clear” button, so they don’t know that we changed the votes!!!
Of course, obviously, the fact that there is a “clear” button proves e-vote machines were purposely designed to be hacked and control the vote; IE: our votes don’t count!
The fact that it isn’t a law that there is no way a log could be cleared or no way a machine could be accessed remotely, is also proof. That means they are designed to be hacked into. Does anyone get this???
Just goes to show that the foundation of elections should NEVER be privatized. We of course all remember the promise that Walden O’Dell, chief executive of Diebold Inc. made in 2003 — that he is “committed to helping Ohio deliver its electoral votes to the president next year.” Link.
We all know how that election turned out!
brad i am a 19 year old kid and i live in ohio i cant believe what i have just stumbled upon it makes me wonder if these diebold systems were not used it the bush elections would we have had this war? i think that the only reason that bush was in office was to finish what his dad started bush was not in charge his dad was that is not fair to the American people to not have control of there own votes if there is any thing that i can do to more inform the people of these problems with the voting systems i would because it seems to me that the government is trying to bury this problem to keep a monopoly on voting and electing who they want in office for control the government controls allot of things but controlling the American vote is not fair it feels like the country is not a democracy but a republican dictatorship. i am glad i have stumbled onto this site if you do thank you for taking the time to read this