task_categories:question-answering language:zh size_categories:10K<n<100K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us art humor
---
task_categories:
- question-answering
language:
- zh
tags:
- art
- humor
size_categories:
- 10K<n<100K
---
# Note

## 预处理方式

```python
from datasets import load_dataset
import jsonlines
import matplotlib.pyplot as plt

ds_ruozhiba = load_dataset("kirp/wisdomBar")

_data = []
for item in ds_ruozhiba["train"]:
    instruct = item["title"] if item["detail"] is None else item["title"] + ("," if item["title"][-1] not in [",", "，","。", ".", "！", "!", "？", "?"] else "") + item["detail"]
    if instruct:
        _data.append(instruct)
_data_to_dump = [[{"from": "human", "value": value}] for value in _data]

with jsonlines.open('ruozhiba/ruozhiba.jsonl', mode='w') as writer:
    writer.write_all(_data_to_dump)

def string_length_distribution(strings, step=10, gate_coef=0.005):
    distribution = {i: 0 for i in range(0, 100)}
    for string in strings:
        length = len(string)
        distribution[length // step] += 1
    distribution = {
        f"{k*step}-{(k+1)*step}": v for k, v in distribution.items() if v > gate_coef * max(distribution.values())
    }
    return distribution


dist = string_length_distribution(_data)

plt.bar(list(dist.keys()), list(dist.values()))
plt.xlabel("String Length")
plt.ylabel("Count")
plt.title("String Length Distribution")
plt.show()
```

注意事项:

-   拼接方式不是完全合理,部分 title 是初始问题或者梗,detail 部分是答案,部分是梗的进一步补充(语义上并列的同义表述/进一步的阐述),因此需要根据实际情况进行调整.
-   当前暂不对所属细分的处理方式单独处理,统一将拼接后的内容作为 human 的输入.