Jupyter notebooks for post-election audits
The following is a guest blog post authored by Kellie Ottoboni, describing her recent work where they used Jupyter to support statistical audits of election results.
In December 2018, I facilitated pilot post-election risk-limiting audits in three cities in Michigan. This was the first time that SUITE, a new method for “hybrid” risk-limiting audits that I helped develop, has been used in practice. I wrote a Jupyter notebook tool to do the SUITE risk calculations, to determine the necessary sample size, and to sample the ballots using a cryptographically secure pseudo-random number generator.
What are risk-limiting audits?
A risk-limiting audit (RLA) is a statistical check that the reported outcome (the reported winner(s), as opposed to exact vote totals) of an election is correct. The procedure limits the chance that an incorrect outcome will go uncorrected, if the reported outcome is in fact wrong. RLAs involve sampling and examining paper ballots. 34 RLA pilots have been done in California, Colorado, Indiana, New Jersey, Ohio, Virginia, and Denmark. Colorado began requiring RLAs by law in 2017, and Rhode Island will begin requiring RLAs in 2019.
SUITE is a general method for conducting RLAs of stratified samples, where a population of ballots is divided into non-overlapping strata and samples are drawn independently from each stratum. We began developing SUITE for Colorado, treating absentee votes and in-precinct votes as two strata. Aside from this special case, SUITE may be useful for auditing states where counties work independently.
Michigan used the specific two-stratum version of SUITE, which combines two types of RLAs: ballot polling, which involves “polling” the ballots and recording the proportion of votes for each candidate, and ballot-level comparison, which involves comparing paper ballots to their electronic record and counting the number of discrepancies. Ballot-level comparison audits require looking at fewer ballots than ballot polling, but can only be done when ballots can be matched to their electronic records. In two of the Michigan cities, absentee ballots could be matched to their electronic record. SUITE allows you to use these two strategies side-by-side for a single RLA.
The Jupyter tool
Previous RLA pilots have used HTML pages with Javascript code, written by Philip Stark, to conduct the audits. With a short timeframe to create a similar SUITE tool, Philip and I decided that the best solution was to build out the Python library we had begun and use a Jupyter notebook for the interface.
The SUITE notebook made the audits more transparent. In each city, I projected the notebook running locally on my laptop on a screen for about 30 local election officials. They could observe me entering the reported vote totals for each candidate and the 20-digit random seed to initialize the pseudo-random number generator, then see the sampled ballots appear on the screen in a nicely formatted table. These folks are used to working in Excel spreadsheets, so the interactivity of a Jupyter notebook is more familiar interface than simply running an executable file in the terminal. Each step of the audit was interactive and annotated with Markdown.
Our hope is that the SUITE notebook can serve as a proof of concept for further development. Free & Fair turned the HTML RLA tools into industrial strength software and Democracy Works built upon it to create the program that Colorado uses for statewide RLAs.
Pitfalls
One main difference between the HTML RLA tools and the SUITE notebook is that the webpages hide all the code. While we moved most of the code to modules and limited most code cells to one function, the notebook still contained a lot of code to scroll through. Voila, a tool for hiding code in notebooks with interactive widgets, is one possible solution to display only the crucial input and output pieces of the tool.
Another issue we faced was the sequential nature of notebooks. RLAs are iterative: if the risk of the ballots from a first round of sampling is too high, then the audit proceeds to more rounds, until either the risk is sufficiently small or all ballots have been counted. A more proficient software developer than I might have come up with an elegant way to make this possible in a Jupyter notebook while tracking the data from each round of sampling. My hack was to assume that in Michigan we would need at most two rounds of sampling and to copy the code for the first round, with some modifications. (In fact, none of the three cities used more than one round.)
The RLA tools written by Free & Fair and Democracy Works are written in Java and Clojure. Academics continue to improve the statistics, so it would be helpful to have an API accessible with Python. It would enable the open source community to contribute to the codebase rather than reinvent the wheel to build a tool every time new statistics are developed.
Michigan pilots
Rochester Hills, Lansing, and Kalamazoo participated in the pilots of their November, 2018 election. I traveled to each city with a team of election auditing experts from MIT, the NYU Brennan Center for Justice, the Electoral Assistance Commission, and Democracy Works. We hope that this Jupyter notebook helped demystify the code and math, and illustrate that RLAs are feasible and efficient way to insure election integrity.
