pavanmantha/distilroberta-pubmed-embeddings

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1 on the pumed-finetuning dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • pumed-finetuning

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pavanmantha/distilroberta-pubmed-embeddings")
# Run inference
sentences = [
    'Are peripheral blood lymphocytes from patients with rheumatoid arthritis differentially sensitive to apoptosis induced by anti-tumour necrosis factor-alpha therapy?',
    'The efficacy of anti-tumour necrosis factor-alpha (TNF-alpha) therapies in rheumatoid arthritis (RA) has been mainly attributed to TNF-alpha neutralisation. Other mechanism as immune cell apoptosis, which is impaired in RA, may also be induced by anti-TNF-alpha therapies. The aim of our study was to investigate whether TNF-alpha inhibitors could induce apoptosis in vitro of the peripheral blood lymphocytes of RA patients. Peripheral blood mononuclear cells (PBMC) isolated from 24 patients with RA and 18 healthy donors were incubated with anti-TNF-alpha agents, infliximab or etanercept, in comparison with no agent and including an isotypic control, for 48 hours. Apoptosis was detected and quantified by annexin V labelling of phosphatidylserine externalization using cytofluorometric analysis and compared with PBMC production TNF-alpha in vitro. In healthy donors, induced apoptosis was observed in 0.3% to 3.8% of lymphocytes with both therapies. In RA patients the treatment induced lymphocyte apoptosis in 17 of 24 patients with a percentage of annexin V-positive lymphocytes ranging from 0.1% to 25%. Among these 17 RA patients, a significant in vitro lymphocyte apoptosis (> 4%) was observed in 11 patients (46%) compared with healthy donors (p < 0.01). The variability of the response to anti-TNF-alpha within the RA population was not dependent on TNF-alpha synthesis or disease activity.',
    'This study examined links between DNA methylation and birth weight centile (BWC), and explored the impact of genetic variation. Using HumanMethylation450 arrays, we examined candidate gene-associated CpGs in cord blood from newborns with low (<15th centile), medium (40-60th centile) and high (>85th centile) BWC (n = 12). Candidates were examined in an investigation cohort (n = 110) using pyrosequencing and genotyping for putative methylation-associated polymorphisms performed using standard PCR. Array analysis identified 314 candidate genes associated with BWC extremes, four of which showed ≥ 4 BWC-linked CpGs. Of these, PM20D1 and MI886 suggested genetically determined methylation levels. However, methylation at three CpGs in FGFR2 remained significantly associated with high BWC (p = 0.004-0.027).',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: ai-pubmed-validation
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	1.0

Training Details

Training Dataset

pumed-finetuning

• Dataset: pumed-finetuning at 1ba143a
• Size: 8,000 training samples
• Columns: instruction, context, and context_neg
• Approximate statistics based on the first 1000 samples:

  	instruction	context	context_neg
  type	string	string	string
  details	min: 11 tokens mean: 26.36 tokens max: 67 tokens	min: 31 tokens mean: 320.03 tokens max: 512 tokens	min: 82 tokens mean: 321.22 tokens max: 512 tokens

• Samples:

  instruction	context	context_neg
  Do competency assessment of primary care physicians as part of a peer review program?	To design and test a program that assesses clinical competence as a second stage in a peer review process and to determine the program's reliability. A three-cohort study of Ontario primary care physicians. Reference physicians (n = 26) randomly drawn from the Hamilton, Ontario, area; volunteer, self-referred physicians (n = 20); and physicians referred by the licensing body (n = 37) as a result of a disciplinary hearing or peer review. Standardized patients, structured oral examinations, chart-stimulated recall, objective structured clinical examination, and multiple-choice examination. Test reliability was high, ranging from 0.73 to 0.91, and all tests discriminated among subgroups. Demographic variables relating to the final category were age, Canadian or foreign graduates, and whether or not participants were certified in family medicine.	Static stretch is frequently observed in the lung. Both static stretch and cyclic stretch can induce cell death and Na(+)/K(+)-ATPase trafficking, but stretch-induced alveolar epithelial cell (AEC) functions are much less responsive to static than to cyclic stretch. AEC remodeling under static stretch may be partly explained. The aim of this study was to explore the AEC remodeling and functional changes under static stretch conditions. We used A549 cells as a model of AEC type II cells. We assessed F-actin content and cell viability by fluorescence staining at various static-stretch magnitudes and time points. Specifically, we used scanning electron microscopy to explore the possible biological mechanisms used by A549 cells to 'escape' static-stretch-induced injury. Finally, we measured choline cytidylyltransferase-alpha (CCT alpha) mRNA and protein by real-time PCR and Western blot to evaluate cellular secretory function. The results showed that the magnitude of static stretch was the...
  Is age an important determinant of the growth hormone response to sprint exercise in non-obese young men?	The factors that regulate the growth hormone (GH) response to physiological stimuli, such as exercise, are not fully understood. The aim of the present study is to determine whether age, body composition, measures of sprint performance or the metabolic response to a sprint are predictors of the GH response to sprint exercise in non-obese young men. Twenty-seven healthy, non-obese males aged 18-32 years performed an all-out 30-second sprint on a cycle ergometer. Univariate linear regression analysis was employed to evaluate age-, BMI-, performance- and metabolic-dependent changes from pre-exercise to peak GH and integrated GH for 60 min after the sprint. GH was elevated following the sprint (change in GH: 17.0 +/- 14.2 microg l(-1); integrated GH: 662 +/- 582 min microg l(-1)). Performance characteristics, the metabolic response to exercise and BMI were not significant predictors of the GH response to exercise. However, age emerged as a significant predictor of both integrated GH (beta ...	We have previously reported the crucial roles of oncogenic Kirsten rat sarcoma viral oncogene homolog (KRAS) in inhibiting apoptosis and disrupting cell polarity via the regulation of phosphodiesterase 4 (PDE4) expression in human colorectal cancer HCT116 cells in three-dimensional cultures (3DC). Herein we evaluated the effects of resveratrol, a PDE4 inhibitor, on the luminal cavity formation and the induction of apoptosis in HCT116 cells. Apoptosis was detected by immunofluorescence using confocal laser scanning microscopy with an antibody against cleaved caspase-3 in HCT116 cells treated with or without resveratrol in a two-dimensional culture (2DC) or 3DC. Resveratrol did not induce apoptosis of HCT116 cells in 2DC, whereas the number of apoptotic HCT116 cells increased after resveratrol treatment in 3DC, leading to formation of a luminal cavity.
  Is terlipressin more effective in decreasing variceal pressure than portal pressure in cirrhotic patients?	Terlipressin decreases portal pressure. However, its effects on variceal pressure have been poorly investigated. This study investigated the variceal, splanchnic and systemic hemodynamic effects of terlipressin. Twenty cirrhotic patients with esophageal varices grade II-III, and portal pressure > or =12 mmHg were studied. Hepatic venous pressure gradient, variceal pressure and systemic hemodynamic parameters were obtained. After baseline measurements, in a double-blind administration, 14 patients received a 2mg/iv injection of terlipressin and six patients received placebo. The same measurements were repeated 60 min later. No demographic or biochemical differences were observed in basal condition between groups. Terlipressin produced significant decreases in intravariceal pressure from 20.9+4.9 to 16.3+/-4.7 mmHg (p<0.01, -21+/- 16%), variceal pressure gradient from 18.9+/-4.8 to 13.5+/-6.0 mmHg (p<0.01, -28+/-27%), estimated variceal wall tension from 78+/-29 to 59+/-31 mmHg x mm (p<0...	Based on the theories of brain reserve and cognitive reserve, we investigated whether larger maximal lifetime brain growth (MLBG) and/or greater lifetime intellectual enrichment protect against cognitive decline over time. Forty patients with multiple sclerosis (MS) underwent baseline and 4.5-year follow-up evaluations of cognitive efficiency (Symbol Digit Modalities Test, Paced Auditory Serial Addition Task) and memory (Selective Reminding Test, Spatial Recall Test). Baseline and follow-up MRIs quantified disease progression: percentage brain volume change (cerebral atrophy), percentage change in T2 lesion volume. MLBG (brain reserve) was estimated with intracranial volume; intellectual enrichment (cognitive reserve) was estimated with vocabulary. We performed repeated-measures analyses of covariance to investigate whether larger MLBG and/or greater intellectual enrichment moderate/attenuate cognitive decline over time, controlling for disease progression. Patients with MS declined in...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

pumed-finetuning

• Dataset: pumed-finetuning at 1ba143a
• Size: 1,000 evaluation samples
• Columns: instruction, context, and context_neg
• Approximate statistics based on the first 1000 samples:

  	instruction	context	context_neg
  type	string	string	string
  details	min: 11 tokens mean: 26.08 tokens max: 52 tokens	min: 38 tokens mean: 311.42 tokens max: 512 tokens	min: 45 tokens mean: 316.2 tokens max: 512 tokens

• Samples:

  instruction	context	context_neg
  Are pre-transplant impedance measures of reflux associated with early allograft injury after lung transplantation?	Acid reflux has been associated with poorer outcomes after lung transplantation. Standard pre-transplant reflux assessment has not been universally adopted. Non-acid reflux may also induce a pulmonary inflammatory cascade, leading to acute and chronic rejection. Esophageal multichannel intraluminal impedance and pH testing (MII-pH) may be valuable in standard pre-transplant evaluation. We assessed the association between pre-transplant MII-pH measures and early allograft injury in lung transplant patients. This was a retrospective cohort study of lung transplant recipients who underwent pre-transplant MII-pH at a tertiary center from 2007 to 2012. Results from pre-transplant MII-pH, cardiopulmonary function testing, and results of biopsy specimen analysis of the transplanted lung were recorded. Time-to-event analyses were performed using Cox proportional hazards and Kaplan-Maier methods to assess the associations between MII-pH measures and development of acute rejection or lymphocytic...	The yeast cell cycle is largely controlled by the cyclin-dependent kinase (CDK) Cdc28. Recent evidence suggests that both CDK complex stability as well as function during mitosis is determined by precise regulation of Swe1, a CDK inhibitory kinase and cyclin binding partner. A model of mitotic progression has been provided by study of filamentous yeast. When facing nutrient-limited conditions, Ras2-mediated PKA and MAPK signaling cascades induce a switch from round to filamentous morphology resulting in delayed mitotic progression. To delineate how the dimorphic switch contributes to cell cycle regulation, temperature sensitive cdc28 mutants exhibiting constitutive filamentation were subjected to epistasis analyses with RAS2 signaling effectors. It was found that Swe1-mediated inhibitory tyrosine phosphorylation of Cdc28 during filamentous growth is in part mediated by Ras2 activation of PKA, but not Kss1-MAPK, signaling. This pathway is further influenced by Cks1, a conserved CDK-bind...
  Is predictive accuracy of the TRISS survival statistic improved by a modification that includes admission pH?	To determine if pH measured at the time of hospital admission and corrected for PCO2 was an independent predictor of trauma survival. Phase 1 was a retrospective case-control analysis of 1708 patients, followed by multivariate multiple logistic regression analysis of a subset of 919 patients for whom the Revised Trauma Score (RTS), Injury Severity Score (ISS), and pH were available. Phase 2 was a prospective comparison of a mathematical model of survival derived in phase 1 (pH-TRISS) with the TRISS method in 508 of 1325 subsequently admitted trauma patients. Urban level 1 trauma center. All patients admitted with blunt or penetrating trauma during the study period. Survival vs mortality. In phase 1, factors significantly associated with mortality by t test and chi 2 analysis included the RTS, ISS< Glasgow Coma Scale, corrected pH (CpH), and sum of the head, chest, and abdominal components of the Abbreviated Injury Scale-85 (AIS85) (HCAISS) (for all, P < .0001). The TRISS statistic was ...	Ovarian cancer is the most lethal gynecologic malignancy, and there is an unmet clinical need to develop new therapies. Although showing promising anticancer activity, Niclosamide may not be used as a monotherapy. We seek to investigate whether inhibiting IGF signaling potentiates Niclosamide's anticancer efficacy in human ovarian cancer cells. Cell proliferation and migration are assessed. Cell cycle progression and apoptosis are analyzed by flow cytometry. Inhibition of IGF signaling is accomplished by adenovirus-mediated expression of siRNAs targeting IGF-1R. Cancer-associated pathways are assessed using pathway-specific reporters. Subcutaneous xenograft model is used to determine anticancer activity. We find that Niclosamide is highly effective on inhibiting cell proliferation, cell migration, and cell cycle progression, and inducing apoptosis in human ovarian cancer cells, possibly by targeting multiple signaling pathways involved in ELK1/SRF, AP-1, MYC/MAX and NFkB. Silencing IGF...
  Does exposure to intermittent nociceptive stimulation under pentobarbital anesthesia disrupt spinal cord function in rats?	Spinal cord plasticity can be assessed in spinal rats using an instrumental learning paradigm in which subjects learn an instrumental response, hindlimb flexion, to minimize shock exposure. Prior exposure to uncontrollable intermittent stimulation blocks learning in spinal rats but has no effect if given before spinal transection, suggesting that supraspinal systems modulate nociceptive input to the spinal cord, rendering it less susceptible to the detrimental consequences of uncontrollable stimulation. The present study examines whether disrupting brain function with pentobarbital blocks descending inhibitory systems that normally modulate nociceptive input, making the spinal cord more sensitive to the adverse effect of uncontrollable intermittent stimulation. Male Sprague-Dawley rats received uncontrollable intermittent stimulation during pentobarbital anesthesia after (experiment 1) or before (experiment 2) spinal cord transection. They were then tested for instrumental learning at ...	Increased serum hepcidin has been reported in patients receiving chronic hemodialysis, and hypothesized to contribute to the alterations of iron metabolism of end-stage renal disease. However, no quantitative assessment is available to date; the clinical determinants are still under definition; and the role of genetic factors, namely HFE mutations, has not yet been evaluated. The aim of this study was to quantitatively assess serum hepcidin-25 in hemodialysis patients versus controls, and analyze the relationship between hepcidin, iron indices, HFE genotype, and erythropoietic parameters. Sixty-five hemodialysis patients and 57 healthy controls were considered. Hepcidin-25 was evaluated by surface-enhanced laser desorption/ionization time-of-flight mass spectrometry, HFE genotype by restriction analysis. Serum hepcidin-25 was higher in hemodialysis patients compared with controls. In patients, hepcidin-25 correlated positively with ferritin and C reactive protein, and negatively with s...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	ai-pubmed-validation_cosine_accuracy
-1	-1	-	-	1.0
0.8	100	0.0152	0.0085	1.0

Framework Versions

• Python: 3.11.10
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.4.1+cu124
• Accelerate: 1.4.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}