overview.md

NVIDIA NemotronLabs VoiceChat 12B

▶ Hear it first. Natural turn-taking, barge-in and live tool calling.

Sample	What to listen for	Player
Natural turn-taking	Smooth back-and-forth, ~450 ms response
Barge-in / interruption	User cuts in; model yields instantly
Tool calling live	Tools are called when needed

Model Overview

Description:

NVIDIA NemotronLabs VoiceChat is a 12B end-to-end, real-time speech full duplex (FD) model for conversational AI that jointly performs streaming speech understanding and speech generation [1]. Unlike traditional cascaded stacks (ASR → LLM → TTS), this model achieves full duplex, real-time, seamless voice interaction in one unified architecture, eliminating the need for multiple models or API handoffs, thus reducing end-to-end latency. It sets new benchmarks by bringing open, robust, and highly natural conversation capabilities. Moreover, NVIDIA NemotronLabs VoiceChat is the first open full-duplex model to support tool calling while maintaining a natural conversation flow during tool execution. For each tool a specific “on-hold” message can be defined that will be spoken by the agent as soon as the LLM generates the text that will trigger the tool call and response.

The model operates on audio signals, which are encoded using a fast conformer module. The resulting audio tokens are inputted into a Nemotron Nano V2 9B LLM backbone to predict text tokens, which are fed to a TTS decoder [2] to predict audio codes for generating the agent's speech. A separate output channel is used to predict tool calling scripts. NemotronLabs VoiceChat offers an unprecedented trade-off between intelligence and latency in the space of open-source voice agents, as highlighted by our benchmarking results below.

This model is ready for research purposes only.

Highlights

Parameters	Turn-taking latency	VoiceBench (open FD)	Tool calling	Architecture
12B	~450 ms	#2	1st open FD	1 unified model

License/Terms of Use

GOVERNING TERMS: The use of this model is governed by the OpenMDW License Agreement, version 1.1.

Use Case:

NemotronLabs VoiceChat is targeted for researchers, developers, and professionals in the field of natural language processing (NLP) and speech technology for purposes such as automatic speech recognition (ASR), text-to-speech synthesis (TTS), and voice assistant development.

Deployment Geography

Global

Release Date:

July 16, 2026.

Model Architecture

Architecture Type: Hybrid Mamba/Transformer  ·  Parameters: 12B

Network Architecture:

NemotronLabs VoiceChat uses:

• Fast Conformer Speech Encoder from Nemotron-Speech-Streaming-En-0.6b
• NVIDIA Nemotron Nano v2 LLM backbone
• NVIDIA TTS decoder and codec [2]
• Separate output channel for tool-calling scripts

Input

Property	Value
Input Type(s)	Text (prompt), Audio (user speech)
Input Format	String, WAV/WebAudio
Sample rate	16 kHz

Output

Property	Value
Output Type(s)	Text (agent), Audio (agent speech), Text (user transcription)
Output Format	String, WAV/WebAudio
Sample rate	22.05 kHz

How NemotronLabs VoiceChat Compares to Other Open FD Models

Model	Params	Tool calling	Big Bench Audio
NemotronLabs VoiceChat	12B	✔	37.0%
PersonaPlex 7B	7B	✘	19.1%
Moshi	7B	✘	4.4%
Freeze-Omni	7B	✘	33.4%

Software Integration

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Runtime Engine: vLLM

Supported Hardware Microarchitecture Compatibility:

• NVIDIA A100
• NVIDIA H100
• NVIDIA B200
• NVIDIA RTX-6000

Preferred/Supported Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

• v1.0

Quickstart

A tutorial is under preparation and will be shared shortly. Visit back soon!

Training, Testing, and Evaluation Datasets:

Training Dataset:

Data Modality: Audio (speech) and Text
Audio Training Data Size: ~550k hours

VoiceChat has been trained on a blend of different datasets comprising both real audio datasets and synthetic speech datasets generated using various TTS systems on text training corpora, including:

• Nemotron 5.5 pre-training and SFT text data
• Brainy-mantis text data
• Greteal AI v1 and v2 text data
• Ultrachat text data
• Blackwell studio recordings real speech data
• Fisher real speech data
• LibriVox
• LibriTTS
• HiFi-TTS
• Riva Speakers: Internal Dataset
• Publicly available internet scale data
• PromptTTS
• VCTK
• Voxmovies
• JL-Corpus
• Nemotron Nano v3 function calling data
• Persona Plex training datasets

Data Collection Method by dataset: Hybrid: Human, Synthetic, Automated.
Labeling Method by dataset: Automated.

Testing/Evaluation Dataset:

VoiceBench

VoiceBench is a benchmark dataset developed to evaluate large language model (LLM)-based voice assistants, focusing on real-world spoken interactions rather than just text or clean speech recognition. It combines audio and text data. The dataset includes multiple subsets covering tasks like open-ended questions, multiple-choice QA, instruction following, and adversarial cases — sourced from both real human speech and synthetic text-to-speech examples.

Link: VoiceBench
Data Collection Method by dataset: Hybrid: Human, Synthetic, Automated.
Labeling Method by dataset: Automated.
Properties: VoiceChat is #2 amongst all open full-duplex models on VoiceBench.

Benchmark Scores:

Metric	Value
Text-output average accuracy	55.1

Artificial Analysis Big Bench Audio

Big Bench Audio is a benchmark dataset used to evaluate the reasoning capabilities of audio-language and native speech-to-speech models. It adapts 1,000 questions from Big Bench Hard into audio format, testing whether models can understand spoken prompts and solve reasoning tasks directly from audio rather than relying only on text input. The benchmark is used by Artificial Analysis to compare speech models on audio reasoning performance across challenging question-answering tasks.

Link: Artificial Analysis Big Bench Audio
Data Collection Method by dataset: Hybrid: Synthetic, Automated.
Labeling Method by dataset: Automated.
Properties: TODO

Benchmark Scores:

Metric	Value
Text-output average accuracy	37.0%

Full-Duplex-Bench 1.0

Full-Duplex-Bench 1.0 is a benchmark designed to evaluate the interactive capabilities of full-duplex spoken dialogue models like VoiceChat. It focuses on measuring natural, human-like conversational behaviors such as pause handling, backchanneling, smooth turn-taking, and user interruption management, using automatic metrics to provide consistent, reproducible assessments of model performance.

Link: Full-Duplex-Bench 1.0
Data Collection Method by dataset: Hybrid: Human, Synthetic, Automated.
Labeling Method by dataset: Automated.
Properties: NemotronLabs VoiceChat is #2 amongst all open models on FullDuplexBench 1.0.

Benchmark Scores:

Metric	Value
Pause Handling(Synthetic): TOR↓	0.153
Pause Handling(Candor): TOR↓	0.255
Smooth Turn Taking: TOR↑	0.82
Smooth Turn Taking: Latency↓	448 ms
User Interruption: TOR↑	1
User Interruption: Latency↓	480 ms
User Interruption: GPT-4o↑	4.33

AU Harness BFCL-v3 (tool calling)

This is the subset of the AU Harness benchmark focused on the evaluation of tool calling within audio conversation. It was created by systematically converting the original BFCL-v3 textual instructions into spoken counterparts.

Link: AU Harness
Data Collection Method by dataset: Hybrid: Human, Synthetic, Automated.
Properties: NemotronLabs VoiceChat is the first open full-duplex model to support tool calling while maintaining a natural conversation flow during tool execution.

Benchmark Scores:

Metric	Value
Simple	58.5%
Multiple	62.5%
Parallel	42.5%
Parallel Multiple	27.5%
Irrelevance	89.6%
Average	56.1%

Full-Duplex-Bench v3 (tool calling)

Full-Duplex-Bench-v3 (FDB-v3) is a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use.

Link: Full-Duplex-Bench v3
Data Collection Method by dataset: Human.
Properties: NemotronLabs VoiceChat is competitive with frontier models in terms of tool selection accuracy on this benchmark.

Benchmark Scores:

Metric	Value
Tool Selection	82.5%
Argument accuracy	44.2%
Pass@1	33%

Inference:

Acceleration Engine: vLLM
Test Hardware: NVIDIA H100

Known Limitations

NemotronLabs VoiceChat is trained with no more than 2-min long audio context windows. Conversational context beyond this window may not be retained reliably.

NemotronLabs VoiceChat has been optimized to achieve a trade-off between general knowledge and natural conversation, hence it may not be as performant as the LLM backbone it is derived from, namely NVIDIA Nemotron Nano v2, in terms of knowledge, instruction-following capabilities and safety.

The model can make reasoning errors or provide incorrect/incomplete information NemotronLabs VoiceChat was not explicitly trained for reasoning or alignment. Its performance on tasks requiring multi-step reasoning, arithmetic, or safety-aligned behavior may therefore be limited.

Further, the model may not yet be able to systematically handle user backchannelling.

As far as tool calling is concerned, we recommend a maximum of 5 tools per session; more tools may degrade performance. Also, the model cannot yet reliably call multiple tools simultaneously. Long tool responses may cause delays before the agent speaks (mitigate with on-hold messages). The user cannot interrupt the agent during tool calling execution. Further, in mixed conversations (general chat + tool requests), the model may answer from its own knowledge instead of calling the appropriate tool.

Finally, the model is not suitable for noisy or highly reverberant environments, especially where background speech may occur.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

References:

[1] SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model
[2] Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
[3] PersonaPlex: Voice and role control for full duplex conversational speech models