license:cc-by-nc-sa-4.0 doi:10.57967/hf/2802 region:us
---
license: cc-by-nc-sa-4.0
---

This corpus is distributed under the cc-by-nc-sa-4.0 license. Please review before accessing. https://creativecommons.org/licenses/by-nc-sa/4.0/

Please cite as:

Khosmood, F., Dekhtyar, A., Ellwein, S., White, B., "The Digital Democracy Corpus: Comprehensive Proceedings of Four State Legislatures 2015-2018", Digital Humanities, Washington, DC, August 2024.

CONTENT

This a corpus of a comprehensive proceedings of the state legislature in four largest US States: California, Texas, New York and Florida. It contains two full sessions for California (2015-2016 and 2017-2018 sessions) and for the other three states only the 2017-2018 session. 
The proceedings are mainly transcripts of what was said in public committee hearings of the legislature including floor sessions. They were developed from videos released by the legislature, but they were transcribed and annotated by the project. In addition to the speeches, speaker identities, committees, bills, sessions, hearings and links to hosted videos are provided.
The format of the corpus is a series of CSV files closely resembling relational database tables of utterances, people, committees, bills, hearings, discussions and videos.
The corpus contains over 1.6 Million speeches spoken by about 116,000 individually identified speakers in four states. The data is over 1.66GB uncompressed.

COLUMN IDS

A short explanation of column names for all 9 CSV files is provided here: https://docs.google.com/spreadsheets/d/1QJA9OkhLQQVduc3FD532N0h1e5a3Z7lYIfQs7Ex2Op4/edit?usp=sharing 

QUALITY

While no formal assessment of the content quality has been published, this corpus is considered of the highest quality mainly because every auto-transcribed utterance was subsequently verified and edited for quality and annotated by people trained and employed by the project. Unfortuantely since unlike the US Federal Government, none of the 50 states produce official transcript records, there's no way to verify against an authoritive source, and there are no other projects that have produced the same data.

BRIEF HISTORY

This corpus was developed by the Cal Poly Institute for Advanced Technology and Public Policy as part of the "Digital Democracy" government transparency project 2015-2018. The key individuals were Sen. Sam Blakeslee (former California Senate minority leader and founder of IATPP), Christine Robertson (Sam's chief of staff and designer of the project), Hans Poschman (IATPP staff and former legislative staff), Toshihiro Kuboi (Software Engineer), Thomas Gerrity (Software Engineer) and Cal Poly Computer Science Professors Foaad Khosmood (NLP), Alex Dekhtyar (Databases and Data Science), Franz Kurfess (AI) and Davide Falessi (Software Engineering). Help and consultation was also provided by Dr. Michael Latner (Political Science) and Pat Howe (Journalism). Dozens of Cal Poly students in Computing and Liberal Arts also made valuable contributions during this period. 
The project was a web-accessible portal (digitaldemocracy.org) where citizens of each state could search the records, browse hearings, bills, votes and donations, and view auto-synched videos with transripts. However live data gathering and transcription was expensive to maintain and the project's public service mission did not allow for monetization, so the project was halted by December 2018. But research continued on the same concepts involving Cal Poly students and visitors from Graz University of Technology, Munich University of Applied Sciences and University of Miami. Special thanks to Professors Christian Guetl (Graz), Gudrun Socher (Munich) and Lindsay Grace (Miami) and students Thorsten Ruprechter, Anastasia Klimashevskaya, Michelle Perkonigg, Alexander Ehm, Sasha Prostota, Katy Huang.
The project was revived by 2022 and became part of the nonprofit news agency CalMatters, covering the California 2023-2024 legislative session (https://digitaldemocracy.calmatters.org). A new UI was created by 10UP. More information, financial and news data was integrated into the public facing site and a new tool "AI TipSheets" was created to help reporters cover the legislature better. 

GOOGLE COLAB

We have created a Google Colab that demonstrates some basic operations such as downloading and reading in the data and displaying full transcripts
https://colab.research.google.com/drive/1RkV4bvbjJmHG5VEznZjndZKU_8XYyZGH?usp=sharing 

CONTACTS

IATPP https://iatpp.calpoly.edu

Foaad Khosmood https://foaad.net (foaad -at- calpoly -dot- edu)