vincolle/reranker-Italian-ModernBERT-base-gooaq-mnrl

Italian ModernBERT Reranker

A cross-encoder reranker for Italian, fine-tuned from DeepMount00/Italian-ModernBERT-base on a tailored Italian QA dataset.

---

Why this model

Italian-language information retrieval has very few dedicated reranker models — most pipelines either fall back to multilingual cross-encoders (diluted by 100 languages) or skip reranking entirely. This model is trained natively in Italian, on Italian-specific hard negatives, to fill that gap.

Results

Evaluated with CrossEncoderRerankingEvaluator on a held-out 2,500-sample dev split of the training distribution:

Metric	Score	Δ over base retriever
MAP	0.9105	+0.0703
MRR@10	0.9102	+0.0712
NDCG@10	0.9235	+0.0576

"Base retriever" refers to nickprock/multi-sentence-BERTino — a strong, Italian-specialized bi-encoder — used to mine hard negatives during training. The gain reported here is therefore a realistic estimate of what this reranker adds on top of a competitive existing Italian retriever, not a comparison against a weak baseline.

Out-of-distribution evaluation (mMARCO-it): pending — will be added here once complete.

Training data

This model was trained on vincolle/gooaq-italian-80k, an Italian translation of GooAQ that I produced myself (translated with DeepSeek). To my knowledge this is the first public Italian version of GooAQ — it does not exist as a native dataset in any other language besides English.

GooAQ-style data consists of real Google autocomplete-style questions paired with featured-snippet-style answers, giving the model exposure to natural, informational queries rather than synthetic or templated ones.

Hard negative mining

Hard negatives were mined using nickprock/multi-sentence-BERTino, a native Italian bi-encoder trained with Matryoshka Representation Learning. Using an Italian-specific embedder for mining (rather than a generic multilingual one) was a deliberate choice — early experiments with multilingual embedders produced negatives that were either too easy (trivially distinguishable) or, in the worst case, contained answers just as valid as the labeled positive. Switching to an Italian-native retriever for mining produced negatives that were genuinely hard without being false negatives, which is what made margin-based filtering during mining meaningful at all.

• 74,961 training triples (question, positive, 5 hard negatives)
• 2,500 held-out dev samples, reranking-evaluation format (30 candidates per query, positive included)

Model architecture

• Base model: DeepMount00/Italian-ModernBERT-base — a from-scratch Italian ModernBERT, cased, native Italian pretraining
• Max sequence length: 8,192 tokens (vs. 512 for classic BERT-based rerankers — useful for long-form answer reranking)
• Output: single relevance logit (num_labels=1)
• Loss: CachedMultipleNegativesRankingLoss (sigmoid activation, scale 10.0, 4 in-batch negatives, mini-batch size 32)

A cased, natively-Italian base model was chosen deliberately over multilingual or uncased alternatives: the training data is cased, and ModernBERT's architecture (RoPE, longer context, modern pretraining recipe) offers a meaningfully higher ceiling than older Italian BERT variants for this task.

Usage

pip install -U sentence-transformers

from sentence_transformers import CrossEncoder

model = CrossEncoder("vincolle/reranker-Italian-ModernBERT-base-gooaq-mnrl")

# Score query-document pairs directly
pairs = [
    ("Quante calorie si bruciano correndo 5 miglia?",
     "Correndo 5 miglia si brucia una grande quantità di calorie. In generale, la maggior parte delle persone che corre 5 miglia brucia circa 500 calorie."),
    ("Quante calorie si bruciano correndo 5 miglia?",
     "Le pinzette sono consentite nel bagaglio a mano, così come i rasoi elettrici."),
]
scores = model.predict(pairs)
print(scores)
# [0.99..., 0.05...]

# Or rerank a list of candidate documents for a single query
ranks = model.rank(
    "Quante calorie si bruciano correndo 5 miglia?",
    [
        "Correndo 5 miglia si brucia una grande quantità di calorie...",
        "Le pinzette sono consentite nel bagaglio a mano...",
        "Guidare con ammortizzatori difettosi può essere estremamente pericoloso...",
    ],
)

Suggested pipeline

This reranker is designed as the second stage of a retrieve-then-rerank pipeline:

1. Retrieve top-k candidates with a fast bi-encoder (e.g. nickprock/multi-sentence-BERTino)
2. Rerank those candidates with this model to get the final ranking

from sentence_transformers import SentenceTransformer, CrossEncoder

retriever = SentenceTransformer("nickprock/multi-sentence-BERTino")
reranker = CrossEncoder("vincolle/reranker-Italian-ModernBERT-base-gooaq-bce")

query = "Quante calorie si bruciano correndo 5 miglia?"
candidates = retriever.similarity(...)  # your top-k retrieval step
reranked = reranker.rank(query, candidates)

Training details

Full hyperparameters

Hyperparameter	Value
per_device_train_batch_size	128
num_train_epochs	1
learning_rate	2e-5
lr_scheduler_type	linear
warmup_steps	0.1
fp16	True
optimizer	adamw_torch_fused
seed	42

Training curve

Step	Training Loss	gooaq-it-dev NDCG@10
1	2.4481	–
100	1.3025	0.8877
200	0.4025	0.9110
300	0.3212	0.9165
400	0.2890	0.9197
500	0.2640	0.9220
586 (final)	–	0.9235

Total training time: 2.6 hours on a single T4 GPU.

Framework versions

• Python 3.12.13 · Sentence Transformers 5.5.1 · Transformers 5.10.2 · PyTorch 2.11.0+cu128 · Datasets 4.0.0

Limitations

• Trained on a single dataset distribution (GooAQ-style: short factoid questions, paragraph-length answers). Performance on other query styles (long-form, multi-hop, document-style) is not yet verified — see the pending OOD evaluation above.
• The training data was machine-translated, not human-verified. While quality appears strong on manual inspection, some translation artifacts may be present.

Citation

If you use this model, please cite the base model and the Sentence Transformers library:

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}