|
|
--- |
|
|
language: |
|
|
- en |
|
|
- zh |
|
|
- ru |
|
|
- ja |
|
|
- de |
|
|
- fr |
|
|
- es |
|
|
- pt |
|
|
- vi |
|
|
- th |
|
|
- ar |
|
|
- ko |
|
|
- it |
|
|
- pl |
|
|
- nl |
|
|
- sv |
|
|
- tr |
|
|
- he |
|
|
- cs |
|
|
- uk |
|
|
- ro |
|
|
- bg |
|
|
- hu |
|
|
- el |
|
|
- da |
|
|
- fi |
|
|
- nb |
|
|
- sk |
|
|
- sl |
|
|
- hr |
|
|
- lt |
|
|
- lv |
|
|
- et |
|
|
- mt |
|
|
pipeline_tag: sentence-similarity |
|
|
tags: |
|
|
- qwen |
|
|
- embedding |
|
|
- onnx |
|
|
- int8 |
|
|
- quantized |
|
|
- text-embeddings-inference |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# Qwen3-Embedding-0.6B ONNX INT8 for Text Embeddings Inference |
|
|
|
|
|
This is an INT8 quantized ONNX version of [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) optimized specifically for [Text Embeddings Inference (TEI)](https://github.com/huggingface/text-embeddings-inference) with CPU acceleration. |
|
|
|
|
|
## Key Features |
|
|
|
|
|
- **INT8 Quantization**: ~8x smaller model size (0.56GB vs 4.7GB) |
|
|
- **CPU Optimized**: 2-4x faster inference on CPU compared to float32 |
|
|
- **TEI Compatible**: Properly formatted for Text Embeddings Inference |
|
|
- **Multilingual**: Supports 29 languages including English, Chinese, Russian, Japanese, etc. |
|
|
- **Mean Pooling**: Configured for mean pooling (handled by TEI) |
|
|
|
|
|
## Performance |
|
|
|
|
|
- **Model size**: 0.56 GB (vs 4.7 GB float32) |
|
|
- **Expected speedup**: 2-4x on CPU |
|
|
- **Accuracy**: Minimal loss (1-3%) compared to float32 |
|
|
- **Best for**: CPU deployments, edge devices, high-throughput scenarios |
|
|
|
|
|
## Usage with Text Embeddings Inference |
|
|
|
|
|
### Docker Deployment (CPU) |
|
|
|
|
|
```bash |
|
|
docker run -p 8080:80 \ |
|
|
-e OMP_NUM_THREADS=$(nproc) \ |
|
|
-e KMP_AFFINITY=granularity=fine,compact,1,0 \ |
|
|
-e ORT_THREAD_POOL_SIZE=$(nproc) \ |
|
|
ghcr.io/huggingface/text-embeddings-inference:cpu-latest \ |
|
|
--model-id YOUR_USERNAME/qwen3-embedding-0.6b-int8-tei-onnx |
|
|
``` |
|
|
|
|
|
### Python Client |
|
|
|
|
|
```python |
|
|
from huggingface_hub import InferenceClient |
|
|
|
|
|
client = InferenceClient("http://localhost:8080") |
|
|
|
|
|
# Single embedding |
|
|
response = client.post( |
|
|
json={"inputs": "What is Deep Learning?"}, |
|
|
) |
|
|
embedding = response.json() |
|
|
|
|
|
# Batch embeddings |
|
|
response = client.post( |
|
|
json={"inputs": ["What is Deep Learning?", "深度学习是什么？"]}, |
|
|
) |
|
|
embeddings = response.json() |
|
|
``` |
|
|
|
|
|
## CPU Optimization |
|
|
|
|
|
For optimal CPU performance, set these environment variables: |
|
|
|
|
|
```bash |
|
|
export OMP_NUM_THREADS=$(nproc) # Use all physical cores |
|
|
export KMP_AFFINITY=granularity=fine,compact,1,0 |
|
|
export ORT_THREAD_POOL_SIZE=$(nproc) |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
