license:other region:us
---
license: other
duplicated_from: jeffwan/sharegpt_vicuna
---

Duplicated from `philschmid/sharegpt-raw` which is marked as duppliated from `jeffwan/sharegpt_vicuna` until step 3. 

Then processed with the [Better Uncensored (BUn) pipeline](https://huggingface.co/sudoaza/better-uncensored). A version with long conversations split is also provided.

Now we have the cleaned uncensored dataset in `sharegpt_20230401_clean_bun.json` (57058 conversations) and the same with split long conversations in `sharegpt_20230401_clean_split_bun.json` (103152 conversations).

This latest one should be a drop-in replacement for `anon8231489123/ShareGPT_Vicuna_unfiltered`.

**Note:** the BUn pipeline removes mostly unicode conversations, so this would not be usable for mainly non-ASCII languages like Chineese, Russian, etc.

## Prepraration

```
pip3 install -r requirements.txt
```

## Data Cleaning

1. merge two raw json files and json beautify the merged file

```
python merge.py sharegpt_90k_raw_dataset/sg_90k_part1.json sharegpt_90k_raw_dataset/sg_90k_part2.json  sharegpt_20230401_html_unformatted.json
python pretty_json.py --in sharegpt_20230401_html_unformatted.json --out sharegpt_20230401_html.json
```

2. (Optional) Verify the json file

```
if jq empty sharegpt_20230401_html.json 2>/dev/null; then
  echo "JSON is valid"
else
  echo "JSON is invalid"
fi

jq length sharegpt_90k_raw_dataset/sg_90k_part1.json
jq length sharegpt_90k_raw_dataset/sg_90k_part2.json
jq length sharegpt_20230401_html.json
```

3. clean data - remove html tags etc

```
python3 clean_sharegpt.py --in sharegpt_20230401_html.json --out sharegpt_20230401_clean.json
....
100%|███████████████████████████████████████████████████████████████████| 90665/90665 [06:32<00:00, 230.98it/s]
total: 90665, skip: 13745, new: 76920
```

4. uncensor with BUn

```
python uncensor_sharegpt.py --in-file sharegpt_20230401_clean.json --out-file sharegpt_20230401_clean_bun.json
....
total: 76920, skip: 19862, new: 57058, uncen: 0
```

5. Split the long conversation

```
python -m fastchat.data.split_long_conversation --in sharegpt_20230401_clean_bun.json --out sharegpt_20230401_clean_split_bun.json --model-name meta-llama/Llama-2-13b-hf
...
#in: 57058, #out: 103152
```

Now we have the cleaned uncensored dataset in `sharegpt_20230401_clean_bun.json` (57058 conversations) and the same with split long conversations in `sharegpt_20230401_clean_split_bun.json` (103152 conversations).

This latest one should be a drop-in replacement for `anon8231489123/ShareGPT_Vicuna_unfiltered`.
 
