task_categories:text-generation license:mit size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2405.08355 region:us function-calling
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: conversations
    list:
    - name: from
      dtype: string
    - name: value
      dtype: string
  - name: domain
    dtype: string
  splits:
  - name: train
    num_bytes: 52531050
    num_examples: 12022
  - name: validation
    num_bytes: 3066470
    num_examples: 700
  - name: test
    num_bytes: 5929698
    num_examples: 1354
  download_size: 17783138
  dataset_size: 61527218
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: validation
    path: data/validation-*
  - split: test
    path: data/test-*
license: mit
task_categories:
- text-generation
tags:
- function-calling
---
# Seal-Tools
<img src="https://cdn-uploads.huggingface.co/production/uploads/635966506805fab09b09b8cc/9dyWupH5IjI5fwvbmLpnB.png" alt="Seal Tools Logo" style="width:25%; height:auto;">

This Huggingface repository contains the dataset generated in [Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark](https://arxiv.org/abs/2405.08355).

## Abstract
Seal-Tools contains self-instruct API-like tools. Seal-Tools not only offers a large
number of tools, but also includes instances
which demonstrate the practical application
of tools. Seeking to generate data on a large
scale while ensuring reliability, we propose a
self-instruct method to generate tools and instances, allowing precise control over the process. 
Moreover, our Seal-Tools contains hard
instances that call multiple tools to complete
the job, among which some are nested tool
callings. For precise and comprehensive evaluation, we use strict format control and design
three metrics from different dimensions. Therefore, Seal-Tools can serve as a new benchmark
to evaluate the tool-calling ability of LLMs.
Finally, we evaluate several prevalent LLMs
and our finetuned model on Seal-Tools. The results show that current systems are far from
perfect. The code, data and experiment results are available [here](https://github.com/fairyshine/Seal-Tools).

https://link.springer.com/chapter/10.1007/978-981-97-9434-8_29

## Results
### Overall Results
| Model                                      | Format ACC | Tool P  | Tool R  | Tool F1 | Parameter P | Parameter R | Parameter F1 |
|--------------------------------------------|------------|---------|---------|---------|-------------|-------------|--------------|
| **ChatGPT** _(gpt-3.5-turbo-0613)_        | 96.16      | 83.20   | 74.73   | 78.74   | 68.63       | 66.85       | 67.73        |
| **GPT4** _(gpt-4-0613)_                   | 97.12      | 90.02   | 74.71   | 81.65   | 80.52       | 67.57       | 73.48        |
| **LLaMA2** _7B_                           | 40.55      | 47.91   | 26.74   | 34.33   | 33.52       | 20.43       | 25.39        |
| **LLaMA2-Chat** _7B_                      | 78.73      | 62.10   | 53.91   | 57.72   | 44.92       | 43.24       | 44.06        |
| **Vicuna** _7B-v1.5_                      | 70.83      | 67.33   | 49.81   | 57.26   | 49.11       | 42.26       | 45.43        |
| **Mistral** _7B-Instruct-v0.2_            | 77.03      | 76.84   | 59.65   | 67.16   | 64.81       | 50.25       | 56.61        |
| **ToolLLaMA2** _7B-v2_                    | 13.44      | 19.35   | 0.96    | 1.84    | 18.98       | 0.84        | 1.61         |
| **Ours** _(finetuned on LLaMA2-7B)_       |            |         |         |         |             |             |              |
| &emsp; w/ BM25                            | 95.57      | 79.67   | 74.79   | 77.15   | 73.51       | **70.76**   | 72.11        |
| &emsp; w/ DPR                             | **95.86**  | **82.81** | **77.84** | **80.25** | **75.95**   | 70.23       | **72.98**   |

### Single-Tool Results
| Model           | Format ACC             | Tool P    | Tool R    | Tool F1   | Parameter P | Parameter R | Parameter F1 |
|-----------------|------------------------|-----------|-----------|-----------|-------------|-------------|--------------|
| **ChatGPT**     | 98.98                  | 88.01     | 94.90     | 91.33     | 74.28       | 83.94       | 78.82        |
| **GPT4**        | 98.64                  | 88.16     | 96.26     | 92.03     | 82.00       | 85.16       | 83.55        |
| **LLaMA2**      | 44.22                  | 25.83     | 42.18     | 32.04     | 15.93       | 28.66       | 20.48        |
| **LLaMA2-Chat** | 85.37                  | 40.27     | 81.63     | 53.93     | 26.54       | 63.21       | 37.38        |
| **Vicuna**      | 76.53                  | 47.65     | 72.45     | 57.49     | 33.79       | 59.76       | 43.17        |
| **Mistral**     | 86.73                  | 72.99     | 86.39     | 79.13     | 66.14       | 68.29       | 67.20        |
| **ToolLLaMA**   | 21.77                  | 12.50     | 2.72      | 4.47      | 11.94       | 1.63        | 2.86         |
| **Ours**        |                        |           |           |           |             |             |              |
|   w/ BM25       | **98.30**              | 91.81     | 91.50     | 91.65     | 84.31       | 85.16       | 84.73        |
|   w/ DPR        | **98.30**              | **93.13** | **92.18** | **92.65** | **85.54**   | **85.37**   | **85.45**    |

### Multiple-Tool Results
| Model           | Format ACC            | Tool P    | Tool R    | Tool F1   | Parameter P | Parameter R | Parameter F1 |
|-----------------|-----------------------|-----------|-----------|-----------|-------------|-------------|--------------|
| **ChatGPT**     | 95.38                 | 82.70     | 73.01     | 77.55     | 68.11       | 65.49       | 66.77        |
| **GPT4**        | 96.70                 | 90.24     | 72.86     | 80.62     | 80.37       | 66.17       | 72.58        |
| **LLaMA2**      | 39.53                 | 54.52     | 25.42     | 34.68     | 38.43       | 19.78       | 26.11        |
| **LLaMA2-Chat** | 76.89                 | 67.02     | 51.54     | 58.27     | 49.03       | 41.64       | 45.03        |
| **Vicuna**      | 69.25                 | 71.13     | 47.88     | 57.23     | 51.85       | 40.87       | 45.71        |
| **Mistral**     | 74.34                 | 77.36     | 57.36     | 65.88     | 64.67       | 48.81       | 55.63        |
| **ToolLLaMA**   | 11.13                 | 22.95     | 0.81      | 1.57      | 21.05       | 0.78        | 1.50         |
| **Ours**        |                       |           |           |           |             |             |              |
|   w/ BM25       | 94.81                 | 78.57     | 73.36     | 75.87     | 72.61       | **69.61**   | 71.08        |
|   w/ DPR        | **95.19**             | **81.88** | **76.61** | **79.16** | **75.12**   | 69.02       | **71.94**    |

### Nested-Tool Results
| Model         | Format ACC | Tool P  | Tool R  | Tool F1 | Parameter P | Parameter R | Parameter F1 |
|--------------|------------|---------|---------|---------|-------------|-------------|--------------|
| **LLaMA2-Chat** | 79.86      | 73.04   | 58.39   | 64.90   | 37.23       | 34.66       | 35.90        |
| **Mistral**     | 68.43      | 84.16   | 57.67   | 68.44   | 52.00       | 36.94       | 43.20        |
| **Ours**        | **96.76**  | 89.64   | 85.82   | 87.69   | 77.32       | 74.15       | 75.70        |
| ├ has seen (501) | 96.41      | 91.03   | 86.61   | 88.76   | 78.88       | 75.43       | 77.12        |
| └ still unseen (85) | **98.82**  | 81.71   | 81.08   | 81.40   | 67.66       | 66.02       | 66.83        |

```
@misc{wu2024sealtoolsselfinstructtoollearning,
      title={Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark}, 
      author={Mengsong Wu and Tong Zhu and Han Han and Chuanyuan Tan and Xiang Zhang and Wenliang Chen},
      year={2024},
      eprint={2405.08355},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2405.08355}, 
}
```