license:cc-by-4.0 size_categories:1K<n<10K format:parquet modality:tabular modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
license: cc-by-4.0
size_categories:
- 1K<n<10K
---

# Overview
Parsed from a series of PDFs named "「AI と著作権に関する考え方について（素案）」のパブリックコメントの結果について（個人）", published by 文化庁.

Original Website: https://www.bunka.go.jp/seisaku/bunkashingikai/chosakuken/hoseido/r05_07/

This project is WIP. Currently 7 out of X files are parsed.

# Note
* Sentences censored by the publisher are converted into `<censored>`.
* Some entries are filtered out if their texts are probably broken.
* Some comments are missing because they are scanned as images and we couldn't parse them.

# Method
We parsed the PDFs by running the following program.
<details>

```python
#! pip install pdfminer.six tqdm pandas pyarrow fastparquet

from pdfminer.high_level import extract_pages
from pdfminer.layout import LTTextBoxHorizontal, LTTextLineHorizontal, LTFigure, LTRect
from tqdm import tqdm
import pandas as pd
from pathlib import Path
import re

files = sorted(Path("pdfs").glob("*.pdf"))
initial_id = 185001345000000001
pattern = re.compile(r"●受付番号  (\d+)")
QUANT_Y = 10
WIDTH = 5000
CENSORED_TAG = "<censored>"

def commit(receipt_number: int, content: str, filename: str, page: int, table: list, filtered: list):
    # initial validation
    if len(table) == 0:
        assert receipt_number == initial_id

    # number validation
    if len(table) > 0 and table[-1][0] + 1 != receipt_number:
        # number is skipped, previous entry maybe broken
        # print("receipt number %d may be broken. skipped." % table[-1][0], file=sys.stderr)
        filtered.append(table[-1])
        table.pop()
    table.append((receipt_number, content.replace("\n", ""), filename, page))


if __name__ == "__main__":
    ignored_classes = set()
    table = []
    filtered = []
    for file in files:
        print("extracting", file.name)
        pages = extract_pages(file)

        processing_id = None
        content_buffer = ""
        content_head_page_no = None
        for page_idx, page in enumerate(tqdm(pages)):
            flatten_elements = []
            for elm in page:
                if isinstance(elm, LTTextBoxHorizontal):
                    for el in elm:
                        assert isinstance(el, LTTextLineHorizontal)
                        flatten_elements.append(el)
                elif isinstance(elm, LTFigure):
                    for el in elm:
                        if isinstance(el, LTRect):
                            flatten_elements.append(el)
                        else:
                            ignored_classes.add(elm.__class__)
                elif isinstance(elm, (LTTextLineHorizontal, LTRect)):
                    flatten_elements.append(elm)
                else:
                    ignored_classes.add(elm.__class__)
            
            # higher (larger top_y) textbox and then more left (smaller left_x) textbox comes early
            # NOTE: QUANT_Y must be shorter than the height of textline
            sorted_elements = sorted(flatten_elements, key=lambda e: -e.y1//QUANT_Y * WIDTH + e.x0)

            for elm in sorted_elements:
                if isinstance(elm, LTTextLineHorizontal):
                    element_text = elm.get_text().strip()
                    m = pattern.match(element_text)
                    if m: # receipt title
                        if processing_id is None:
                            # this is the first receipt
                            processing_id = int(m.group(1))
                            content_buffer = ""
                            content_head_page_no = page_idx + 1
                        else:
                            # found next title. Commit the current buffer.
                            commit(processing_id, content_buffer, file.name, content_head_page_no, table, filtered)
                            processing_id = int(m.group(1))
                            content_buffer = ""
                            content_head_page_no = page_idx + 1
                    else: # content
                        content_buffer += elm.get_text().strip()
                elif isinstance(elm, LTRect):
                    content_buffer += CENSORED_TAG
                else:
                    raise NotImplementedError
        # End of file. Commit last content
        commit(processing_id, content_buffer, file.name, content_head_page_no, table, filtered)
    df = pd.DataFrame(table, columns=["receipt_number", "content", "filename", "page"])
    df_filt = pd.DataFrame(filtered, columns=["receipt_number", "content", "filename", "page"])

    print("entries:", len(df))
    print("rejected entries:", len(df_filt))
    print("ignored classes:", ignored_classes)

    df.to_parquet("cleaned.parquet")
    df_filt.to_parquet("broken.parquet")

# extracting 94024601_01.pdf
# 1129it [00:12, 91.09it/s]
# extracting 94041201_01.pdf
# 1089it [00:07, 153.65it/s]
# extracting 94059201_01.pdf
# 1058it [00:09, 108.85it/s]
# extracting 94091801_01.pdf
# 1097it [00:11, 94.81it/s]
# extracting 94112801_01.pdf
# 0it [00:00, ?it/s]Ignoring (part of) ToUnicode map because the PDF data does not conform to the format. This could result in (cid) values in the output. The start object is not a byte.
# 1146it [00:14, 64.23it/s]Ignoring (part of) ToUnicode map because the PDF data does not conform to the format. This could result in (cid) values in the output. The start object is not a byte.
# 1152it [00:14, 79.02it/s]
# entries: 4996
# rejected entries: 2
# ignored classes: {<class 'pdfminer.layout.LTFigure'>}

```

</details>


# License
This dataset is licensed under CC-BY 4.0. This is compatible with the original website's [Term of Use](https://www.mext.go.jp/b_menu/1351168.htm).

## Credit in Japanese

[「AI と著作権に関する考え方について（素案）」のパブリックコメントの結果について（個人）](https://www.bunka.go.jp/seisaku/bunkashingikai/chosakuken/hoseido/r05_07/)（©文化庁, [クリエイティブ・コモンズ・ライセンス（表示4.0 国際）](https://creativecommons.org/licenses/by/4.0/)）を加工して作成.


## Credit in English

Created by modifying [「AI と著作権に関する考え方について（素案）」のパブリックコメントの結果について（個人）](https://www.bunka.go.jp/seisaku/bunkashingikai/chosakuken/hoseido/r05_07/) © Agency for Cultural Affairs, Government of Japan, [Licensed under CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).