license:apache-2.0 size_categories:100K<n<1M format:json modality:text library:datasets library:dask library:mlcroissant region:us
---
license: apache-2.0
---

# Congressional Data

* This dataset contains speeches extracted from government institutions of the US, UK and Canada. 
* Each speech is a single partition and contains metadata such as speaker name, date, country, and the source URL.
* This dataset is intended to evaluate federated learning algorithms. If using one speech as a client, then there are 134k clients for date range 2023-04-01 to 2024-08-31; If using one speaker as a client, then there are 1k clients for the same period. Since there are more speeches that are later added, there will be more clients as we increase the date range.
* Github: https://github.com/hazylavender/fl_llm_benchmark_dataset 

## Dataset Details

The raw dataset is a list of dictionaries object like below
```
{
    "url": "https://www.ourcommons.ca/Content/House/441/Debates/335/HAN335-E.XML",
    "date_str": "2024-06-19",
    "title": "Prime Minister's Award for Excellence in Early Childhood Education",
    "speaker": "Mr. Mike Kelloway",
    "data": "it is a privilege to rise in the House today to recognize a constituent of mine who has received the Prime Minister's Award for Teaching Excellence….",
    "chamber": "HOUSE OF COMMONS",
    "country": "CA"
  },
```

## Uses

To use this dataset, use any libraries of your liking, for example
```
from mlcroissant import Dataset
ds = Dataset(jsonld="https://huggingface.co/api/datasets/hazylavender/CongressionalDataset/croissant")
records = ds.records("default")
```

To filter on dates (or any other fields), you can do
```
import itertools
import pandas as pd

df = (
    pd.DataFrame(list(itertools.filterfalse(lambda x: x['default/date_str'].decode() < '2024-01-01', records)))
)
```



## Dataset statistics
(cutoff 2024-08-31)
* Date range 2023-04-01 to 2024-08-31
* Number of speeches: 134k
* Token distribution
![token_distribution.png](https://cdn-uploads.huggingface.co/production/uploads/66c104f1ef1f45cd71d1896c/VusQi5COxo_iCCeoLgSra.png)
* Number of speeches grouped by countries
    * Total: 133,926
![speeches_by_country.png](https://cdn-uploads.huggingface.co/production/uploads/66c104f1ef1f45cd71d1896c/WhMBRUNvqIDRy77DhvM7Z.png)
* Number of unique speakers grouped by countries
    * Total: 1930
![speakers_by_country.png](https://cdn-uploads.huggingface.co/production/uploads/66c104f1ef1f45cd71d1896c/oP5x1HZcgrSlbndV_oWec.png)

## Data sources and license information
**US**
* Source: https://www.congress.gov/congressional-record 
* License: https://www.loc.gov/collections/publications-of-the-law-library-of-congress/about-this-collection/rights-and-access/#:~:text=Publications%20of%20the%20Law%20Library%20of%20Congress%20are%20works%20of,free%20to%20use%20and%20reuse. 
* API: https://api.govinfo.gov/docs/
  
**Canada**
* Source: https://www.ourcommons.ca/documentviewer/en/44-1/house/sitting-340/hansard 
* License: https://www.ourcommons.ca/en/open-data 
* API: https://api.openparliament.ca
  
**UK**
* Source: https://hansard.parliament.uk/ 
* License: https://www.parliament.uk/site-information/copyright/ 
* API: https://www.theyworkforyou.com/ 

