license:mit region:us
---
license: mit
---

# Dataset Card for nrw-bplan-scrape

## Dataset Description
 
  **Homepage:** [DSSGx Munich](https://sites.google.com/view/dssgx-munich-2023/startseite) organization page. 
 
  **Repository:** [GitHub](https://github.com/DSSGxMunich/land-sealing-dataset-and-analysis). 

### Dataset Summary

This dataset contains all inputs needed as well as outputs of running the full pipeline for creating the NRW land sealing dataset. This can be reproduced by running [this notebook](https://github.com/DSSGxMunich/land-sealing-dataset-and-analysis/blob/main/src/1_execute_pipeline.ipynb).

## Dataset Structure

* nrw
  * bplan
    * features
      * keywords
        * exact_search
          * ```baunvo_keywords.csv```: Results y/n of keywords found in documents relating to baunvo and article 13b.
        * fuzzy_search:
          * ```keyword_dict_hochwasser.json```: Results of keywords found in documents relating to "hochwasser", e.g. hqhäufig and hq100
          * contains 7 csv files with results of fuzzy key search for keywords. The file name indicates the key being searched for and the text around this keyword is extracted in a row for each document
    * raw
      * images: images from [here](https://huggingface.co/datasets/DSSGxMunich/nrw-bplan-images) can be added to this folder
      * links:
          * ```NRW_BP.geojson```: The file downloaded from the NRV geoportal, containing all raw data on URLs to land parcel bplans. 
          * ```land_parcels.geojson```: A processed version of NRW_BP.geojson.
          * ```NRW_BP_parsed_links.csv```: A csv formatted version of NRW_BP.geojson.
          
      * text: 
          * ```bp_text.json```: Raw output of the text text extraction of each pdf. Contains only columns for the filename and the extracted text.
          * ```document_texts.json```: Enriched version of bp_texts.json in which columns about the documents have been appended. 
      * pdfs: pdfs extarcted from the NRW Geoportal and are found [here](https://huggingface.co/datasets/DSSGxMunich/nrw-bplan-pdfs), can be added to this folder
    * knowledge_extraction_agent: Contains 6 json files. The filename corresponds to the key looked for in the fuzzy keyword search (e.g. ```fh.json``` cooresponds to ```firsthöhe.csv```, ```gfz.json``` corrresponds to ```geschossflächenzahl.csv```). More unfo can be found [here](https://huggingface.co/datasets/DSSGxMunich/bplan_keyword_extraction) 
    * ```knowledge_agent_output.json```: Is a toy example for 10 files of the output of the pipeline for the knowledge agent (merging of results in ```nrw/bplan/knowledge_extraction_agent```)
  * clean
    * ```document_texts.xlsx```: See [here](https://huggingface.co/datasets/DSSGxMunich/document_text) for more information
    * ```exact_keyword.xlsx```: This corresponds to baunvo_keywords.csv.
    * ```fuzzy_keyword.xlsx```: Is the merged version of the files found in ```nrw/bplan/fuzzy_search````
    * ```knowledge_agent.xlsx```: The .xlsx version of ```nrw/bplan/knowledge_agent_output.json```)
    * ```land_parcels.xlsx```: See [here](https://huggingface.co/datasets/DSSGxMunich/land_parcels) for more information
    * ```regional_plans.xlsx```: The .xlsx version of the data table found [here](https://huggingface.co/datasets/DSSGxMunich/regional_plan_sections)
  * rplan
    * features: contains ```regional_plan_sections.json```, the output of the pipeline - a more detailed can be found [here](https://huggingface.co/datasets/DSSGxMunich/regional_plan_sections)
    * raw
      * geo: contains ```regions_map.geojson``` with information on the geolocations of the regional plans
      * pdfs: contains pdfs of regional plans for NRW - used as input to run the pipeline
      * text: contains text extracted with Tika from all pdf regional plans

