rahulseetharaman/reranker-msmarco-v1.1-bert-uncased_L-12_H-256_A-4-listnet

CrossEncoder based on bansalaman18/bert-uncased_L-12_H-256_A-4

This is a Cross Encoder model finetuned from bansalaman18/bert-uncased_L-12_H-256_A-4 on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: bansalaman18/bert-uncased_L-12_H-256_A-4
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("rahulseetharaman/reranker-msmarco-v1.1-bert-uncased_L-12_H-256_A-4-listnet")
# Get scores for pairs of texts
pairs = [
    ['cost to replace faucet in garage', "Purchasing extra remotes and getting openers set up for operation will typically range from $100 to $400, which will add to the overall cost of the garage door installation. If your opener works with the new door, you won't need to have it replaced. Garage door opener repair can cost between $100 and $300 depending on the extent of the problem. Return to Top. The type of garage door you select -- and any extra pieces or labor required -- will influence how much you pay to have it professionally installed."],
    ['cost to replace faucet in garage', 'Here are some factors to consider as you look to invest in a new garage door: A garage door will cost anywhere from $200 for single doors to $4,000 for two or more doors, with better materials accounting for the higher cost. Most homeowners report spending about $1,060 for the door and installation. Garage door opener repair can cost between $100 and $300 depending on the extent of the problem. Return to Top. The type of garage door you select -- and any extra pieces or labor required -- will influence how much you pay to have it professionally installed.'],
    ['cost to replace faucet in garage', 'Get an INSTANT estimate of the cost to Replace a Tub Faucet! Our free calculator uses recent, trusted data to estimate costs for your Tub Faucet Replacement project. For a basic 1 faucet project in zip code 47474, the benchmark cost to Replace a Tub Faucet ranges between $350.80 - $579.54 per faucet. To estimate costs for your project: 1.'],
    ['cost to replace faucet in garage', 'A plumber will charge $250 to remove an old faucet and replace it with a good quality and stylish new one. That includes the labor and new faucet. You can do it for $145, the cost of the new unit, and save 42 percent. You need basic plumbing tools, along with a basin wrench, a specialty tool that twists and turns so it can loosen and adjust tough-to-reach fasteners on a faucet assembly.'],
    ['cost to replace faucet in garage', 'Compare the cost to replace a kitchen faucet yourself with the labor and material cost to hire a plumber for the job. That includes the labor and new faucet. You can do it for $145, the cost of the new unit, and save 42 percent. You need basic plumbing tools, along with a basin wrench, a specialty tool that twists and turns so it can loosen and adjust tough-to-reach fasteners on a faucet assembly.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'cost to replace faucet in garage',
    [
        "Purchasing extra remotes and getting openers set up for operation will typically range from $100 to $400, which will add to the overall cost of the garage door installation. If your opener works with the new door, you won't need to have it replaced. Garage door opener repair can cost between $100 and $300 depending on the extent of the problem. Return to Top. The type of garage door you select -- and any extra pieces or labor required -- will influence how much you pay to have it professionally installed.",
        'Here are some factors to consider as you look to invest in a new garage door: A garage door will cost anywhere from $200 for single doors to $4,000 for two or more doors, with better materials accounting for the higher cost. Most homeowners report spending about $1,060 for the door and installation. Garage door opener repair can cost between $100 and $300 depending on the extent of the problem. Return to Top. The type of garage door you select -- and any extra pieces or labor required -- will influence how much you pay to have it professionally installed.',
        'Get an INSTANT estimate of the cost to Replace a Tub Faucet! Our free calculator uses recent, trusted data to estimate costs for your Tub Faucet Replacement project. For a basic 1 faucet project in zip code 47474, the benchmark cost to Replace a Tub Faucet ranges between $350.80 - $579.54 per faucet. To estimate costs for your project: 1.',
        'A plumber will charge $250 to remove an old faucet and replace it with a good quality and stylish new one. That includes the labor and new faucet. You can do it for $145, the cost of the new unit, and save 42 percent. You need basic plumbing tools, along with a basin wrench, a specialty tool that twists and turns so it can loosen and adjust tough-to-reach fasteners on a faucet assembly.',
        'Compare the cost to replace a kitchen faucet yourself with the labor and material cost to hire a plumber for the job. That includes the labor and new faucet. You can do it for $145, the cost of the new unit, and save 42 percent. You need basic plumbing tools, along with a basin wrench, a specialty tool that twists and turns so it can loosen and adjust tough-to-reach fasteners on a faucet assembly.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.0784 (-0.4112)	0.2779 (+0.0169)	0.0850 (-0.3346)
mrr@10	0.0537 (-0.4238)	0.3681 (-0.1317)	0.0640 (-0.3627)
ndcg@10	0.0739 (-0.4665)	0.2468 (-0.0783)	0.0827 (-0.4179)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.1471 (-0.2430)
mrr@10	0.1619 (-0.3061)
ndcg@10	0.1345 (-0.3209)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 11 characters mean: 33.62 characters max: 100 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  how long does a engine take to cool down	["He says if you drive your car around and then park it in the garage, it won't get as cold as if you kept it outside. Henein: When you stop the engine, the engine is hot. It takes a long time for the internal components of the engine to cool down. It can take up to 8 or to 12 hours. That's because there's no wind to carry the heat away. So even if it feels cold in your garage, the engine may still be warm. ", "This helps the engine and the fan circulate air quicker, getting cool air and coolant to the engine and helping you remove some heat from the car. If you're in stop/start traffic, this could be a great way to keep the engine moving when the car can't. 5. Add water to the radiator if you are out of coolant. Add warm water to your radiator, but only after the engine has cooled. Cool water can cause a crack in your engine block due to the extreme change in temperature. 6. Drive for short stretches, turn the car off, and repeat if you need to keep moving.", 'Making the world better,...	[1, 0, 0, 0, 0, ...]
  average cost of dessert bar	['The average cost of the dessert table for a wedding with 100 guest would range from $1.75 to $3.00 per serving. Price is dependent upon which desserts you choose and is based on base pricing. Other charges may apply. The following guide will give you an example of the average cost of a tiered cake for a wedding with a guest list of 100. Three tiers with a base of 12″, a middle tier of 9″ and and 6″ topper. The cake can be round or square. The round cake feeds 102 guests and the starting price is $408', 'This average, of course, can vary depending on where you live. A couple in a busy metropolitan areas is likely to spend more on a wedding cake than a couple in a more rural area. A dessert bar’s cost can vary depending on the desserts you choose. Plated desserts at a buffet could cost up to $15 per plate, while a self-serve dessert bar can go as low as a few dollars per guest. Dessert and catering prices averaged $63 per person in 2012, but menu items certainly impact how much you’ll ...	[1, 0, 0, 0, 0, ...]
  average salary for attorney in utah	['$60. Salaries Down From Five Years Ago. Lawyers in this metropolitan area make $126,720 per year ($60.92 per hour). Looking at historical salaries for Salt Lake City, Utah, the average pay for this job has decreased by $2,596 (-2.11%) from a year ago, and decreased by $14,995 (-11.07%) from five years ago. Across all metropolitan areas, pay for this job has decreased by 5.33% since 2009.', 'The salary for someone with the job title Attorney I in Salt Lake City, UT varies depending on a number of factors.', "The average pay for an Attorney / Lawyer is $77,252 per year. Experience has a moderate effect on salary for this job. People in this job generally don't have more than 20 years' experience.", 'Average Salary of Jobs Matching Your Search. Average Attorney salaries for job postings in Salt Lake City, UT are 11% lower than average Attorney salaries for job postings nationwide.', 'Average Attorney salaries for job postings in Utah are 22% lower than average Attorney salaries for job ...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 9 characters mean: 33.57 characters max: 112 characters	min: 1 elements mean: 5.89 elements max: 10 elements	min: 1 elements mean: 5.89 elements max: 10 elements

• Samples:

  query	docs	labels
  cost to replace faucet in garage	["Purchasing extra remotes and getting openers set up for operation will typically range from $100 to $400, which will add to the overall cost of the garage door installation. If your opener works with the new door, you won't need to have it replaced. Garage door opener repair can cost between $100 and $300 depending on the extent of the problem. Return to Top. The type of garage door you select -- and any extra pieces or labor required -- will influence how much you pay to have it professionally installed.", 'Here are some factors to consider as you look to invest in a new garage door: A garage door will cost anywhere from $200 for single doors to $4,000 for two or more doors, with better materials accounting for the higher cost. Most homeowners report spending about $1,060 for the door and installation. Garage door opener repair can cost between $100 and $300 depending on the extent of the problem. Return to Top. The type of garage door you select -- and any extra pieces or labor req...	[1, 1, 0, 0, 0, ...]
  what does diffusion mean	['diffusion. 1. the state or process of being widely spread. 2. the spontaneous mixing of the molecules or ions of two or more substances resulting from random thermal motion; its rate is proportional to the concentrations of the substances and it increases with the temperature. diffusion. 1. the process of becoming diffused, or widely spread. 2. the spontaneous movement of molecules or other particles in solution, owing to their random thermal motion, to reach a uniform concentration throughout the solvent, a process requiring no addition of energy to the system.', 'Diffusion. Dissolved substances have to pass through the cell membrane to get into or out of a cell. Diffusion is one of the processes that allows this to happen. Diffusion diffusion: The movement of particles (molecules or ions) from an area of higher concentration to an area of lower concentration occurs when particles spread.', 'diffusion. 1. The process of rapid random movement of the particles of a liquid or gas which...	[1, 0, 0, 0, 0, ...]
  how long will bacon keep	["1 Cooked bacon in strips lasts for around a week post-cooking, in the fridge. 2 If you freeze it, it will last up to 6 months. 3 Just cook it a bit less than you like it, so when you go to warm it up, it doesn't overcook. 4 Bacon bits will last for about 6 weeks in the fridge, and 6 months in the freezer.", 'Different types of bacon keep for different lengths of time. 1 Cooked bacon in strips lasts for around a week post-cooking, in the fridge. 2 Bacon bits will last for about 6 weeks in the fridge, and 6 months in the freezer.', "1 If you freeze it, it will last up to 6 months. 2 Just cook it a bit less than you like it, so when you go to warm it up, it doesn't overcook. 3 Bacon bits will last for about 6 weeks in the fridge, and 6 months in the freezer.", 'Bacon Cindy 2015-04-21T07:10:21+00:00. How long does bacon last? The shelf life of bacon depends on a variety of factors, such as the sell by date, the preparation method and how the meat was stored. Bacon is tasty for bre...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	-	0.0278 (-0.5126)	0.2798 (-0.0453)	0.0345 (-0.4662)	0.1140 (-0.3414)
0.0002	1	2.0871	-	-	-	-	-
0.0508	250	2.0921	-	-	-	-	-
0.1016	500	2.0867	2.0913	0.0300 (-0.5104)	0.2532 (-0.0719)	0.0218 (-0.4788)	0.1017 (-0.3537)
0.1525	750	2.0918	-	-	-	-	-
0.2033	1000	2.0904	2.0907	0.0513 (-0.4891)	0.2946 (-0.0305)	0.0415 (-0.4591)	0.1291 (-0.3262)
0.2541	1250	2.092	-	-	-	-	-
0.3049	1500	2.0921	2.0901	0.0463 (-0.4941)	0.2430 (-0.0821)	0.0457 (-0.4549)	0.1117 (-0.3437)
0.3558	1750	2.089	-	-	-	-	-
0.4066	2000	2.0866	2.0897	0.0449 (-0.4955)	0.2610 (-0.0641)	0.0484 (-0.4523)	0.1181 (-0.3373)
0.4574	2250	2.0873	-	-	-	-	-
0.5082	2500	2.0866	2.0894	0.0606 (-0.4798)	0.2722 (-0.0528)	0.0636 (-0.4371)	0.1321 (-0.3232)
0.5591	2750	2.0844	-	-	-	-	-
0.6099	3000	2.086	2.0892	0.0579 (-0.4826)	0.2606 (-0.0644)	0.0584 (-0.4422)	0.1256 (-0.3297)
0.6607	3250	2.0879	-	-	-	-	-
0.7115	3500	2.0925	2.0893	0.0668 (-0.4737)	0.2546 (-0.0704)	0.0812 (-0.4194)	0.1342 (-0.3212)
0.7624	3750	2.0851	-	-	-	-	-
0.8132	4000	2.0804	2.0892	0.0739 (-0.4665)	0.2468 (-0.0783)	0.0827 (-0.4179)	0.1345 (-0.3209)
0.8640	4250	2.0844	-	-	-	-	-
0.9148	4500	2.0904	2.0891	0.0733 (-0.4671)	0.2487 (-0.0763)	0.0758 (-0.4248)	0.1326 (-0.3227)
0.9656	4750	2.0845	-	-	-	-	-
-1	-1	-	-	0.0739 (-0.4665)	0.2468 (-0.0783)	0.0827 (-0.4179)	0.1345 (-0.3209)

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.0.0
• Transformers: 4.56.0.dev0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ListNetLoss

@inproceedings{cao2007learning,
    title={Learning to Rank: From Pairwise Approach to Listwise Approach},
    author={Cao, Zhe and Qin, Tao and Liu, Tie-Yan and Tsai, Ming-Feng and Li, Hang},
    booktitle={Proceedings of the 24th international conference on Machine learning},
    pages={129--136},
    year={2007}
}