size_categories:10K<n<100K format:csv modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2308.03825 arxiv:2306.11507 arxiv:2310.12505 region:us
---
dataset_info:
  name: Rhesis Telecom Chatbot Benchmark
  license: CC-BY-4.0
  language: en
  tags:
    - rhesis
    - test-sets
    - chatbot
    - telecom
---

# Dataset Card for Rhesis Telecom Chatbot Benchmark

## Description

The test set provided focuses on evaluating the robustness, reliability, and compliance of a telecom chatbot. It covers a wide range of industries, mainly telecom, and explores various behaviors and categories. The test set encompasses topics such as cross-border compliance, telecommunications rights, ethics, moral philosophy, roaming and mobile charges, landline and internet services, and access to online content. It also includes more specific areas like illegal and highly regulated items, deceptive sales practices, unfair billing and contract terms, physical harm and violence, ethical dilemmas, mobile roaming charges, customer service issues, epistemology, data privacy and unauthorized tracking, scams and fraud, privacy and data protection, hidden fees and misleading pricing, and aesthetics. Additionally, the test set covers concerns related to illegal activities. Overall, the test set is designed to comprehensively assess the performance and adherence to regulations of the telecom chatbot across a diverse range of scenarios in the telecom industry and related ethical and legal considerations.

## Structure

The dataset includes four key columns:

- **Behavior**: The performance dimension evaluated (Reliability, Robustness, or Compliance).
- **Topic**: The topic validated as part of the prompt.
- **Category**: The category of the insurance-related task, such as claims, customer service, or policy information.
- **Prompt**: The actual test prompt provided to the chatbot.
- **Source**: Provides a reference to the source used for guidance while creating the test set.

## Integration

In order to easily integrate a Rhesis test set into your existing development pipeline, you can make use of the [Rhesis SDK](https://github.com/rhesis-ai/rhesis-sdk).

### Prerequisites

1. Create an API key by signing up at [app.rhesis.ai](https://app.rhesis.ai).
2. Install the Rhesis SDK:

   ```bash
   pip install rhesis-sdk
   ```

3. Use the following Python snippet to access the dataset:


```python
from rhesis.entities import TestSet

# Initialize and load the test set
test_set = TestSet(id="rhesis-telecom-chatbot-benchmark")
df = test_set.load()  # Returns a pandas DataFrame

# Alternatively, you can download the CSV file directly
test_set.download()  # Downloads to current directory as test_set_{id}.csv
```

Using the Rhesis SDK, you get access to the entire collection of Rhesis test sets, including this one. You can also create your own test sets.

For further details on how to integrate the SDK into your workflow, refer to the [Rhesis SDK documentation](https://docs.rhesis.ai/).

## Community

We welcome contributions and discussions from the community! Here are the different ways you can get involved:

- GitHub: Report issues or contribute to the Rhesis SDK on our [GitHub repository](https://github.com/rhesis-ai/rhesis-sdk).
- Discord: Join our [Discord server](https://discord.rhesis.ai) to connect with other users and developers.
- Email: Reach out to us at [hello@rhesis.ai](mailto:hello@rhesis.ai) for support or inquiries.

## Sources

The following sources were used in creating this dataset:

- Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023). " Do Anything Now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825.
- Huang, Y., Zhang, Q., Y, P. S., & Sun, L. (2023). TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models. http://arxiv.org/abs/2306.11507
- Deng, B., Wang, W., Feng, F., Deng, Y., Wang, Q., & He, X. (2023). Attack prompt generation for red teaming and defending large language models. arXiv preprint arXiv:2310.12505.

## Citation

If you use this dataset, please cite:

```
@inproceedings{rhesis,
  title={Rhesis: A Testbench for Evaluating LLM Applications - Rhesis Telecom Chatbot Benchmark},
  author={Rhesis},
  year={2025}
}
```
