Marco127/Base_Test1_

SentenceTransformer based on sentence-transformers/multi-qa-mpnet-base-dot-v1

This is a sentence-transformers model finetuned from sentence-transformers/multi-qa-mpnet-base-dot-v1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/multi-qa-mpnet-base-dot-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Dot Product

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Marco127/Base_Test1_")
# Run inference
sentences = [
    '\nA hotel guest may not leave the room to another person, even if the time for which he or she has paid has\nnot expired.',
    '\nA hotel guest may not leave the room to another person, even if the time for which he or she has paid has\nnot expired.',
    'Orders for accommodation services made in writing or by other means, which have been\nconfirmed by the hotel and have not been cancelled by the customer in a timely manner, are\nmutually binding. The front office manager keeps a record of all received and confirmed\norders.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

• Evaluated with BinaryClassificationEvaluator

Metric	Value
dot_accuracy	0.6671
dot_accuracy_threshold	48.9305
dot_f1	0.4987
dot_f1_threshold	33.9523
dot_precision	0.3325
dot_recall	0.9964
dot_ap	0.3126
dot_mcc	0.0

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,362 training samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	label
  type	string	string	int
  details	min: 11 tokens mean: 48.75 tokens max: 156 tokens	min: 11 tokens mean: 48.75 tokens max: 156 tokens	0: ~69.20% 1: ~30.80%

• Samples:

  sentence1	sentence2	label
  Hotel guests may receive visits in their hotel rooms from guests not staying in the hotel. Visitors must present a personal document at the hotel reception and register in the visitors' book. These visits can last for only a maximum of 2 hours and must finish until 10:00 pm.	Hotel guests may receive visits in their hotel rooms from guests not staying in the hotel. Visitors must present a personal document at the hotel reception and register in the visitors' book. These visits can last for only a maximum of 2 hours and must finish until 10:00 pm.	0
  We do not guarantee that any special requests will be met, but we will use our best endeavours to do so as well as using our best endeavours to advise you if that is not the case.	We do not guarantee that any special requests will be met, but we will use our best endeavours to do so as well as using our best endeavours to advise you if that is not the case.	0
  Pool and Fitness Room hours and guidelines are provided at check in. All rules and times will be enforced to allow efficient operation of the hotel and for the comfort and safety of all guests.	Pool and Fitness Room hours and guidelines are provided at check in. All rules and times will be enforced to allow efficient operation of the hotel and for the comfort and safety of all guests.	1

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 841 evaluation samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 841 samples:

  	sentence1	sentence2	label
  type	string	string	int
  details	min: 11 tokens mean: 48.1 tokens max: 156 tokens	min: 11 tokens mean: 48.1 tokens max: 156 tokens	0: ~66.71% 1: ~33.29%

• Samples:

  sentence1	sentence2	label
  In the case of fire, guests are obliged to notify the reception without hesitation, either directly, or on the phone (0) and may use a portable fire extinguisher located at the corridors of each floor to extinguish the flames. The use of the elevator in case of fire is prohibited!	In the case of fire, guests are obliged to notify the reception without hesitation, either directly, or on the phone (0) and may use a portable fire extinguisher located at the corridors of each floor to extinguish the flames. The use of the elevator in case of fire is prohibited!	0
  Children should be accompanied in locations such as stairways etc. The rooms are for accommodation service. Each individual staying in a room must be registered at the reception.	Children should be accompanied in locations such as stairways etc. The rooms are for accommodation service. Each individual staying in a room must be registered at the reception.	0
  Towels for the Fitness Room and Pool are located in those areas. Towels from guest rooms are not to be taken to the Pool or Fitness Room.	Towels for the Fitness Room and Pool are located in those areas. Towels from guest rooms are not to be taken to the Pool or Fitness Room.	0

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	dot_ap
-1	-1	-	-	0.3126
0.4739	100	0.0011	0.0001	-
0.9479	200	0.0002	0.0000	-
1.4218	300	0.0	0.0000	-
1.8957	400	0.0001	0.0000	-
2.3697	500	0.0	0.0000	-
2.8436	600	0.0	0.0000	-
3.3175	700	0.0	0.0000	-
3.7915	800	0.0	0.0000	-
4.2654	900	0.0	0.0000	-
4.7393	1000	0.0	0.0000	-

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}