Evaluating Audio Reasoning with Big Bench Audio
mhillsmith, georgewritescode
The emergence of native Speech to Speech models offers exciting opportunities to increase voice agent capabilities and simplify speech-enabled workflows. However, it's crucial to evaluate whether this simplification comes at the cost of model performance or introduces other trade-offs.

To support analysis of this, Artificial Analysis is releasing **[Big Bench Audio](https://huggingface.co/datasets/ArtificialAnalysis/big_bench_audio)**, a new evaluation dataset for assessing the reasoning capabilities of audio language models. This dataset adapts questions from **[Big Bench Hard](https://arxiv.org/pdf/2210.09261)** - chosen for its rigorous testing of advanced reasoning - into the audio domain.

This post introduces the Big Bench Audio dataset alongside initial benchmark results for GPT-4o and Gemini 1.5 series models. Our analysis examines these models across multiple modalities: native Speech to Speech, Speech to Text, Text to Speech and Text to Text. We present a summary of results below, and on the new Speech to Speech page on the [**Artificial Analysis** website](https://artificialanalysis.ai/models/speech-to-speech). Our initial results show a significant "speech reasoning gap": while GPT-4o achieves 92% accuracy on a text-only version of the dataset, its Speech to Speech performance drops to 66%.

## The Big Bench Audio Dataset

[Big Bench Audio](https://huggingface.co/datasets/ArtificialAnalysis/big_bench_audio) comprises **1,000 audio questions** selected from four categories of Big Bench Hard, each chosen for their suitability for audio evaluation:

- **Formal Fallacies**: Evaluating logical deduction from given statements
- **Navigate**: Determining if navigation steps return to a starting point
- **Object Counting**: Counting specific items within collections
- **Web of Lies**: Evaluating Boolean logic expressed in natural language

Each category contributes 250 questions, creating a balanced dataset that avoids tasks heavily dependent on visual elements or text that could be potentially ambiguous when verbalized. 

Each question in the dataset is structured as:


```json
{
  "category": "formal_fallacies",
  "official_answer": "invalid",
  "file_name": "data/question_0.mp3",
  "id": 0
}
```


The audio files were generated using **23 synthetic voices** from top-ranked Text to Speech models in the **[Artifical Analysis Speech Arena](https://artificialanalysis.ai/text-to-speech/arena?tab=Leaderboard)**. Each audio generation was rigorously verified using Levenshtein distance against transcriptions, and edge cases were reviewed manually. To find out more about how the dataset was created, check out the **[dataset card](https://huggingface.co/datasets/ArtificialAnalysis/big_bench_audio)**.


## Evaluating Audio Reasoning

To assess the impact of audio on each model's reasoning performance, we tested **four different configurations** on Big Bench Audio:

1. **Speech to Speech**: An input audio file is provided and the model generates an output audio file containing the answer.
2. **Speech to Text**: An input audio file is provided and the model generates a text answer.
3. **Text to Speech**: A text version of the question is provided and the model generates an output audio file containing the answer.
4. **Text to Text**: A text version of the question is provided and the model generates a text answer.

Based on these configurations we conducted **eighteen experiments**:

<center>

| Model                                      | Speech to Speech | Speech to Text | Text to Speech | Text to Text |
|