source_datasets:THUDM/webglm-qa language:en license:apache-2.0 size_categories:10K<n<100K format:csv modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
language:
- en
license: apache-2.0
source_datasets:
- THUDM/webglm-qa
---

Conversion of [THUDM/webglm-qa](https://huggingface.co/datasets/THUDM/webglm-qa) dataset to be used in pretraining.

Python code used for conversion:

```python
from datasets import load_dataset
import pandas
import re

dataset = load_dataset("THUDM/webglm-qa", split="train")

def format(columns):
    return re.sub(r'\[\d\]', '', columns["answer"].strip())

pandas.DataFrame({"text": [format(columns) for columns in dataset]}).to_csv("train.csv", index=False)
```
