New CA SoS report confirms findings of 'Humboldt Transparency Project,' discovers even more egregious failures in widely-used Diebold voting counting system
One of the citizens who designed the software, responsible for the startling discoveries, offers his thoughts...
By Mitch Trachtenberg on 3/4/2009, 1:40pm PT  

Guest Blogged by Mitch Trachtenberg of the Humboldt County Election Transparency Project

California Secretary of State Debra Bowen has released a remarkable 13-page report [PDF] of her office's investigation into how the Diebold/Premier GEMS software silently dropped all votes contained on 197 ballots from the Humboldt County, California, November 2008 general election.

The report shows that this version of GEMS not only deleted a batch of ballots without any request by --- or alert to --- the elections staff, but also failed to note the deletion in the system's audit log. The report also points to other startling deficiencies with Diebold/Premier's software: "Key audit trail logs in GEMS version 1.18.19 do not record important operator interventions such as deletion of decks of ballots, assign inaccurate date and time stamps to events that are recorded, and can be deleted by the operator."

That's right. The Diebold/Premier vote tabulation system in question not only fails to record all events accurately, and sometimes at all, it also allows anyone with access to the system to completely delete audit logs, covering the tracks of any tampering that may have occurred, at any time, on the system.

Any of these flaws, the report concludes, "appears to violate the 1990 Voting System Standards to an extent that would have warranted failure of the GEMS version 1.18.19 system had they been detected and reported by the [federal] Independent Testing Authority [ITA] that tested the system."...

Discovering Diebold's Failure...

I'm proud to be a member of the Humboldt County Election Transparency Project, the volunteer group that Humboldt County Clerk and Registrar of Voters Carolyn Crnich invited into her office to conduct an independent scan of all ballots cast. The project, proposed by Humboldt commercial fisherman Kevin Collins, and founded by Collins and Crnich along with Tom Pinto, a tech staffer at the district attorney's office, David Cobb, a former Green Party Presidential Candidate, and Parke Bostrom, a concerned citizen. I was there when we ended up scanning more ballots than were counted in the official results. I thought we'd made a mistake ourselves.

I'd developed an open-source vote counting program called Ballot Browser, and I ran it on our collection of scans the night we finished scanning, to try to hunt down our mistake. The transparency project had also kept careful logs of our scanning. With a few exceptions, those logs would also tell us how many ballots we'd scanned for each precinct. (The exceptions were a few mixed batches; most of the ballots were batched by precinct, but not all.)

When I got Ballot Browser's results and compared them with the official count, precinct by precinct, I was able to quickly point to precinct 1E45 as having 197 more ballots in Ballot Browser's count than in the official count. The discovery would lead to corrected, re-certified election results, the CA Secretary of State's investigation and recent finding, the likely decertification of the Diebold software in question, and --- hopefully --- decertification in the approximately thirty other states which use the same software, or a version with the same bug, to count ballots cast in their elections.

When the apparent mistake in the Ballot Browser's results orignally came to light, Carolyn Crnich was checking both the official Diebold scanning logs and our independent scanning logs and also narrowed in on precinct 1E45. It looked like, somehow, a batch of 197 vote-by-mail ballots from precinct 1E45 had been scanned but then deleted. The deletion of this batch, not-so-coincidentally the very first deck of vote-by-mail ballots that the elections staff had scanned into its Diebold/Premier equipment, seemed to have taken place after the election night results, but before the official numbers were certified.

It turned out that the problem was not that the Transparency Project had mistakenly double-scanned a batch of ballots, and not that the elections staff had somehow missed a batch of ballots. The problem was a bug in Diebold's GEMS software: under fairly common circumstances, the first deck of ballots might be silently deleted, with no notice given to election officials overseeing the computerized tabulation.

Since this news broke last December, some have questioned why normal ballot reconciliation procedures didn't flag the dropped ballots. The Secretary of State's report makes this clear [emphasis in original]:

Whether the fact that ballots had been omitted from the tally could or should have been discovered through ballot reconciliation processes that are part of the standard canvass process misses what was at issue in Humboldt County. No software error affecting the accuracy of election results should ever be excused based on claims that the effects of the error could or should be detected and corrected through adherence to sound election administration procedures. In this particular case, a reconciliation of the Registrar's count of vote-by-mail ballots returned by voters with the count reported by GEMS was performed on November 1, the day Deck 0 was tallied, and no discrepancy was found. GEMS reports generated on Election Day and on November 23, 2008, two and a half weeks after the election, continued to accurately reflect the 197 ballots in Deck 0.

It was only later, after the GEMS Central Count Server was re-opened and new decks of vote-by-mail ballots that had been received on Election Day were tallied for the first time, that Deck 0 was deleted, without any warning or notification to the elections official, as a result of the software programming flaw. Because the deletion of the votes from the 197 ballots in Deck 0 occurred long after they were counted and after repeated reports showed them properly accounted for, nothing indicated any need to recheck the earlier reconciliation for a third time.

It would be reasonable to think that the Humboldt County Election Transparency Project had discovered a previously unknown bug in GEMS. Reasonable, but wrong. The Secretary of State's report confirms what The BRAD BLOG reported in one of its follow-ups last December: the error was long known by Diebold.

As the Secretary of State's report now confirms:

Diebold knew of this serious software error no later than October 2004. The company, however, did not notify the Election Assistance Commission (EAC), the National Association of State Election Directors (NASED) or the California Secretary of State. Instead, the company sent a vague email to elections officials in the 11 California counties using GEMS version 1.18.19 with the Central Count Server at the time. (Six other counties used GEMS version 1.18.19, but did not use it with the Central Count Server.) The email ... advised the county officials to create and immediately delete an empty Deck 0 before scanning any real ballots, but did not explain why this new procedure was necessary.

The email and attachment did not inform the elections officials that failure to follow these instructions would likely result in deletion of tallied votes by GEMS without any warning or notice to the system operator. The email and attachment also failed to inform counties that it was a programming flaw in the GEMS software that made the special instructions necessary. The chief elections official for Humboldt County, Registrar of Voters Carolyn Crnich, states that she never saw the email or the attached instructions. The former county elections officer apparently had received the email in 2004, but left Humboldt County in 2007 to work in another county's elections office. That county elections officer did not, according to Registrar of Voters Carolyn Crnich, pass the information along to her or anyone else in her office. This helps to explain why the Deck 0 software error manifested itself in the November 2008 election.

Diebold Audit Logs Also Found Useless, Gameable...

It is stunning enough that the lost ballots flaw, known to Diebold/Premier since 2004, has been allowed to remain in versions of GEMS still in use. But there turn out to be more problems, arguably more serious than even dropping ballots, as noted in the SoS report:

A second set of serious problems related to electronic audit logs was discovered during the Secretary of State Office's investigation of the Deck 0 software programming flaw. First, GEMS version 1.18.19 fails to record in any log important system events such as the deletion of decks of optical scan ballots after they have been scanned and entered into the GEMS election results database. Second, it records the wrong entry date and time for certain decks of ballots. Third, it permits deletion of certain audit logs that contain - or should contain - records that would be essential to reconstruct operator actions during the vote tallying process.

The problems with the audit logs, apparently undetected during the federal certification of GEMS by EAC/NASED, call the entire certification process into question.

I don't think it's possible to overstate the importance of 100% confidence in a voting system's audit logs. Without confidence that a voting system is providing what the Federal Election Commission's 1990 voting system standards call "a concrete, indestructible archival record of all system activity related to the vote tally," there can simply be no confidence in the reported results of an election.

That a "Clear" button was found in the software, capable of deleting an entire audit log, without even confirmation of the clear required by the system operative, is unimaginable in such a system, not to mention, in strict violation of the voting system standards under which this system had, several times, been certified by the federal so-called Independent Testing Authority (ITA).

When election integrity advocates have criticized touch-screen voting, the touch-screen vendors have pointed to the audit logs, and how any tampering will show up in them. And yet here is a system, federally certified, where the audit logs are not only undependable, but can be (and have been!) inadvertently deleted by an elections staffer.

Any of the problems that have turned up, according to the Secretary of State's report, "would warrant a finding by an Independent Testing Authority (ITA) of 'Total Failure'". But that never happened despite multiple federal certifications of both this GEMS system, version 1.18.19, but also "subsequent GEMS versions (1.18.20, 1.18.21, 1.18.22 and 1.18.23) that contain the same software error," as the report notes.

Although the Secretary of State's report is written in calm language, it is a bombshell. In Humboldt County, nearly two hundred citizens were nearly disenfranchised by a four year old bug in a system that had passed federal certification.

Fortunately, Humboldt County still uses paper ballots, so a recount was possible.

Fortunately, Humboldt County Clerk and Registrar of Voters Carolyn Crnich was willing to stick her neck out and allow a group of volunteers to come in and conduct an independent recount.

Fortunately, once the Humboldt County Election Transparency Project turned up a four year old bug in GEMS, California's Secretary of State Debra Bowen conducted a thorough investigation.

Fortunately, that investigation was able to discover what those who conducted federal certification did not: the lack of integrity of the GEMS audit logs.

Does this mean the system worked, or does it just mean that this time, we lucked out? In how many jurisdictions nationwide would the ballots have been recounted, even though no candidate questioned the results? In how many jurisdictions nationwide would there even have been paper ballots to recount?

How many times has the system not worked? If the audit logs aren't rock solid, how can we ever know?

Our votes are too important to be counted by secret software running on black box machines.

===

Mitch Trachtenberg is a member of the Humboldt County Election Transparency Project, and the author of Ballot Browser, free and open source ballot counting software. He is also a partner in Trachtenberg Election Verification Systems (TEVSystems), which provides support for Ballot Browser.

Share article...
                 

Article Categories: Diebold/Premier, Election Irregularities, Debra Bowen, California, Election Reform, Voting System Certification, Election Fraud, Election 2008, EAC, Accountability, Humboldt County