fusion-embedding-2-2b-preview

fusion-embedding-2-2b-preview is the second generation of Eximius Labs' unified multimodal embedding models: text, images, video, and audio in one vector space. Audio is added to a byte-frozen vision-language base through modality-gated deep adapters, so gaining audio never changes a single existing text, image, or video vector. For the first-generation architecture see fusion-embedding-1-2b-preview (that line is final at v0.3).

GitHub | Technical report: arXiv:2607.18666

Model Overview

fusion-embedding-2-2b-preview embeds all four modalities on a Qwen3-VL-Embedding-2B base that is byte-identical to its original release: its text, image, and video behaviour, and its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters (about 2.3% of the stack): a perceiver-resampler that maps frozen Qwen2.5-Omni audio-tower features into the base's input space, plus 28 gated adapters (44.2M) that give the frozen language model in-layer capacity for audio. The adapters run only while audio is being encoded; every other forward pass returns the frozen layers' output untouched, so the invariance is bitwise, not approximate. Adding audio never invalidates an index you have already built.

This v0.3 release ships two revisions of the same model. The default is tuned for audio-text retrieval; a keyword-tuned revision is available for spoken-command use.

Versions in this release

Revision	Use it for	AudioCaps a2t R@10	SpeechCommands
v0.3-preview (default)	general audio-text retrieval, RAG, clustering	0.785	0.894
v0.3-kw-preview	spoken-keyword and command retrieval	0.749	0.929

The default (v0.3-preview) is the recommended model for almost all use. The keyword-tuned revision trades audio-text retrieval quality for higher zero-shot keyword-spotting accuracy; use it only when spoken-command matching is the priority, and use the default otherwise.

Highlights (v0.3-preview)

Feature	Value
Parameters	~2.06B frozen base + 640M frozen audio tower; 60.6M trained
Modalities	text, image, video, audio
AudioCaps 883 (audio-to-text)	R@1 0.333 · R@5 0.646 · R@10 0.785 · mAP@10 0.255
AudioCaps 883 (text-to-audio)	R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445
MAEB(audio-only), 19 tasks	mean 0.454; leads the board on IEMOCAP speaker-gender (0.938)
Non-audio preservation	text, image, video outputs bit-for-bit the base model's
Embedding dimension	2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048)
Base model	Qwen/Qwen3-VL-Embedding-2B (byte-frozen)
Audio tower	Qwen/Qwen2.5-Omni-7B audio encoder (frozen)

Evaluation

AudioCaps retrieval (883-clip test, min-rank over references)

Model	a2t R@10	t2a R@10
fusion-embedding-2 v0.2	0.743	0.775
v0.3-preview	0.785	0.782
v0.3-kw-preview	0.749	0.771

The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.

MAEB(audio-only), full 19-task board — v0.3-preview

Mean 0.4536. Scores are the main metric per task.

Task	Score
IEMOCAPGender	0.938
BeijingOpera	0.924
NMSQAPairClassification	0.794
GTZANAudioReranking	0.717
JamAltArtistA2ARetrieval	0.687
GTZANGenre	0.644
VoxPopuliLanguageID	0.616
MInDS14	0.578
CREMADPairClassification	0.543
MridinghamTonic	0.540
VoxPopuliAccentPairClassification	0.509
CREMA_D	0.277
VoxCelebSA	0.273
BirdCLEF	0.184
CommonLanguageAgeDetection	0.171
SIBFLEURS	0.115
VoxPopuliGenderClustering	0.079
VehicleSoundClustering	0.025
CREMA_DClustering	0.006

IEMOCAP speaker-gender classification is the strongest cell relative to the field. Clustering tasks are the weakest and are a target for the next release.

Non-audio preservation and other modalities

Text, image, and video outputs are bit-for-bit identical to the frozen Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their benchmark scores are therefore the base model's own, unchanged by anything added for audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs only, with no audio-image supervision.

Evaluation integrity

We audited our training corpus against the benchmark tasks we report. Two findings matter to anyone comparing these numbers.

Common Voice overlap (why one cell is not reported)

Our pretraining corpus includes 70,000 Common Voice clips sampled from the validated pool. In Common Voice, validated is the superset from which the train, dev, and test splits are drawn, and the corpus is cumulative across releases. We replayed our ingest to recover the exact clips we trained on and measured the overlap directly:

measure	value
eval clips also in our training set	44 / 200 (22%)
eval sentences appearing verbatim in our captions	94 / 200 (47%)

The cause is structural. Our ingest streams the archive in order and stops at a target count; archive order tracks clip identifier, and low identifiers are the older recordings that the v21 test split draws from. "The first N that pass the filters" therefore over-samples the evaluation era. The effect is compounded because our spoken-word corpus is itself derived from Common Voice.

Consequently this cell is reported only under MTEB's training_datasets disclosure (the leaderboard marks the model as trained on this task; see mteb#5096 for the maintainer guidance we follow). The raw number is inflated by the overlap above and must not be read as a zero-shot result. CommonLanguageAgeDetection derives from Common Voice and is declared the same way.

Tasks we do not train on

The results we do report are drawn from tasks whose source corpora are not in our training mix. We flag the general hazard because it is not specific to us: several corpora in wide use as training data — Common Voice, CREMA-D, IEMOCAP, VoxPopuli, FLEURS, GigaSpeech — are simultaneously tasks in audio embedding benchmarks. A model trained on the obvious speech data is contaminated on several cells by default.

What we changed

• Future ingests record source clip identifiers, so overlap can be audited without replaying the ingest.
• Speech corpora are sampled from a designated train split, never from a pooled validated split that contains the test data.

Notes and limitations

• Emotion recognition regressed relative to v0.2. On RAVDESS zero-shot emotion the score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central to your use, evaluate before adopting. Why this happens, and what does and does not recover it, is the subject of a dedicated paper: the corpus's discriminative structure, not its size or caption vocabulary, decides which attributes the embedding encodes (arXiv:2608.01560).
• One evaluation cell, CommonLanguageAgeDetection, derives from Common Voice, which is part of the training data. It scores low (0.171) and does not inflate the reported mean, but the potential overlap is noted for completeness.
• These are research previews under CC-BY-NC-4.0.

Usage

Requirements

• fusion_embedding package: pip install fusion-embedding[hf] (the Sentence Transformers section below uses the lighter [sense] extra)
• transformers>=4.46, torch (CUDA), torchvision, pillow, soundfile, librosa
• ~14 GB GPU memory at bf16

via inference.py (this repository)

from inference import FusionEmbedder

fe = FusionEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-2b-preview",
    revision="v0.3-preview",           # default, retrieval-tuned
    # revision="v0.3-kw-preview",      # keyword-tuned alternative
)

a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
t = fe.embed_text("a dog barks")         # uses the base's native chat template
i = fe.embed_image("dog.jpg")            # PIL image or path

print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared space

# Matryoshka: pass dim= for smaller embeddings (64..2048)
t_small = fe.embed_text("a dog barks", dim=256)

The checkpoint contains the gated adapters and the loader refuses to run without them. All inputs use the base model's chat-template format; embedding quality is sensitive to this formatting, so use the templates provided by FusionEmbedder.

Cross-modal ranking tip

When ranking a gallery of one modality against queries of another, per-modality mean-centering of the gallery improves cross-modal recall:

gallery = FusionEmbedder.center(gallery_embeddings)

Sentence Transformers

fusion-embedding-2 works directly with Sentence Transformers (v5.5.1+, multimodal encode). The custom module on this repository is a thin adapter over the fusion-embedding package, so embeddings are identical to the native fusion_embedding.UnifiedEmbedder.from_pretrained path.

pip install -U "sentence-transformers>=5.5.1" "fusion-embedding[sense]>=0.3.0" torchvision

# only to embed a video by file path; torchcodec also needs FFmpeg installed
pip install -U torchcodec

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "EximiusLabs/fusion-embedding-2-2b-preview",
    trust_remote_code=True,
)

# Text
text_embs = model.encode([
    "a dog barks in the distance while rain falls on a tin roof",
    "an orchestra tuning up before a performance",
])

# Images: local path, PIL image, or HxWxC uint8 array
image_embs = model.encode(["photo_of_a_bicycle.jpg"])

# Audio: local file path, or a waveform with its sampling rate
waveform = np.sin(2 * np.pi * 440.0 * np.arange(44100) / 44100).astype(np.float32)
audio_embs = model.encode([
    "dog_bark.wav",
    {"audio": {"array": waveform, "sampling_rate": 44100}},
])

# Video: local file path (requires torchcodec) or a [T, C, H, W] uint8 frame
# tensor
video_embs = model.encode(["clip.mp4"])

# One shared space: rank any modality against any other
scores = model.similarity(text_embs, audio_embs)

All vectors are L2-normalized at the full interoperability dimension (2048).

Queries, documents, and instructions

The text side carries a retrieval instruction through the base model's chat template. encode uses the query instruction by default; encode_query / encode_document select the released query/document instructions, and a custom instruction can be passed as prompt= (it replaces the instruction inside the chat template rather than being prepended to the text):

q = model.encode_query(["engine failure sounds"])
d = model.encode_document(["The compressor stalls with a loud metallic bang."])
custom = model.encode(["dog"], prompt="Retrieve audio by sound description.")

Image, video, and audio inputs use the model's fixed native formats; instructions and prompts only affect text.

Matryoshka dimensions

The checkpoint was trained with Matryoshka rungs (2048, 1536, 1024, 512, 256, 128, 64). Truncate and renormalize to use a shorter rung (this matches the native MRL readout):

vecs = model.encode(
    ["a dog barks in the distance"],
    truncate_dim=1024,
    normalize_embeddings=True,
)

Notes

• Revisions: the integration files are on main, which is what loads when no revision= is given, and on the v0.3-kw-preview branch, so both the flagship and the keyword-tuned model load through Sentence Transformers. The immutable release tags, v0.3-preview among them, predate the integration and do not carry these files, so the revision= pin shown in the inference.py example above does not apply here. main carries the same flagship checkpoint as v0.3-preview. For a reproducible load, pin the commit hash of a revision that carries the integration files.
• Each input item is a single modality; embed each modality in its own encode call (the shared space makes the vectors directly comparable). Fused multi-modality items (for example {"text": ..., "audio": ...} in one dict) are not part of the released model contract and raise an error.
• Audio arrays must carry a sampling rate ({"array": ..., "sampling_rate": ...}); any rate is accepted and resampled to 16 kHz exactly as the native path does.
• Cross-modal ranking of a full gallery benefits from per-modality mean-centering, worth a couple of points of R@1; see the "Cross-modal ranking tip" above. The package equivalent of that section's FusionEmbedder.center is fusion_embedding.UnifiedEmbedder.center.
• The first load downloads the frozen base (Qwen/Qwen3-VL-Embedding-2B) and the frozen audio tower (Qwen/Qwen2.5-Omni-7B audio encoder) plus this repository's trained connector checkpoint, exactly like the native loader.
• model.save(...) writes configuration only; the 2B weight stack always loads from the Hugging Face repositories above, so saved directories stay small and reloads fetch weights from the hub cache.

Changelog

• v0.3-preview (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785 (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender. RAVDESS emotion regressed to 0.21.
• v0.3-kw-preview: keyword-tuned revision. SpeechCommands zero-shot 0.929; AudioCaps a2t R@10 0.749.
• v0.2-preview: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
• v0.1-preview: first modality-gated-adapter release.

Deploy on RunPod

One-click from the RunPod Hub.

curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \n  -H "Authorization: Bearer $RUNPOD_API_KEY" \n  -H "Content-Type: application/json" \n  -d '{"input": {"text": "a dog on a beach"}}'

Swap text for image, video, or audio (value = an https URL, data URI, or base64). Audio needs the audio tower: set FE2_ENABLE_AUDIO=1 on the endpoint (a 24 GB+ GPU). Returns 1024-d embeddings; add "dim": 512 to truncate.

Serve with vLLM or SGLang

The fusion-embedding pip package carries serving plugins for both engines. Each registers through the engine's own plugin entry point, so the standard launch command works with no source changes and no trust_remote_code.

vLLM (smoke-tested against vllm==0.26.0):

pip install 'vllm[audio]==0.26.0' fusion-embedding
vllm serve EximiusLabs/fusion-embedding-2-2b-preview --runner pooling

SGLang (smoke-tested against sglang==0.5.16):

pip install sglang==0.5.16 'fusion-embedding[sense]'
python -m sglang.launch_server \
  --model-path EximiusLabs/fusion-embedding-2-2b-preview --is-embedding

Text embeds over the OpenAI-compatible /v1/embeddings route; images and audio go through each engine's multimodal inputs, and the dimensions parameter serves the Matryoshka ladder. Both plugins are parity-gated against the reference implementation on every release: at fp32, text cosine 0.999998 or higher, audio 0.999999 or higher on real clips, and loading the adapters leaves text and image outputs bit-for-bit those of the frozen base. Single GPU (roughly 8 GB for bf16 weights); the plugins force eager execution. Prompt formats and the full parity harnesses are documented in the plugin READMEs: vLLM and SGLang.

Sense packs

Optional sense packs extend this space to more sensors without changing any existing vector:

• fusion-embedding-2-ember: thermal infrared images.
• fusion-embedding-2-tremor: inertial motion (body-worn or robot IMU), with a Unitree-humanoid head, fusion-embedding-2-tremor-g1.
• fusion-embedding-2-tactus: touch, 32x32 pressure/taxel arrays (FSR gloves, e-skins, robot hands); matches to exceeds the STAG (Nature 2019) supervised baseline while remaining open-vocabulary.

Engram

This model plugs into Engram, the open cross-modal memory layer for physical AI. Engram indexes a robot's video, audio, and motion into one embedding space and answers questions about it in plain language, including temporal reasoning that retrieval alone cannot. Engram is built directly on Fusion Embedding 2.

pip install engram-robomem

Repo: https://github.com/Eximius-Labs/engram · PyPI: https://pypi.org/project/engram-robomem · Playground: https://www.eximiuslabs.com/playground

License

Weights (all revisions): CC-BY-NC-4.0 (research preview). The code is Apache-2.0. The frozen base and audio tower retain their original licenses.

Citation

Model:

@software{fusion_embedding_2_2026,
  title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
            Unified Text, Image, Video, and Audio Embedding Space},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
}

Technical report:

@article{tonmoy2026fusion,
  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
             Video, and Audio},
  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
             Arham, Md. Shahrier Islam and Luthra, Arman},
  journal = {arXiv preprint arXiv:2607.18666},
  year    = {2026}
}

Corpus analysis (what the audio training corpus does and does not teach):

@article{tonmoy2026axis,
  title   = {Discriminative Axis, Not Data Volume: What a Contrastive
             Corpus Teaches an Audio Embedding},
  author  = {Tonmoy, Abdul Basit},
  journal = {arXiv preprint arXiv:2608.01560},
  year    = {2026}
}