Embedding evaluation
How does one even evaluate a fine-tuned model? I don’t want to evaluate it during training to keep things modular and since things take a while. I have been using a Triplet dataset for embedding that basically has a question, positive example and a negative example.This is my code for evaluating so far:model = SentenceTransformer(model_path, device='cuda')
model.eval()
ds = load_dataset('json', data_files=dataset_path ,split='train')
ds = ds.select(range(1000))

dev_evaluator = TripletEvaluator(
    anchors=ds["question"],
    positives=ds["positive_example"],
    negatives=ds["negative_example"],
    batch_size= 64,
    show_progress_bar= True,
    main_distance_function= SimilarityFunction.COSINE
)
print ("Beginning Evaluation...")
evaluation_score = dev_evaluator(model)
print(evaluation_score["cosine_accuracy"])However there are a few problems with the code.  First of all it somehow runs over all the data three times for some reason, and the second and third time are quite a bit slower than the first time.Additionally i dont know if its just me but i struggled to even find an example for the main_distance_function, or even an example for a simple evaluation like this.Any tips what im doing wrong or if i can somehow optimize things further with my GPU ?
2025-01-08T02:28:39Z
Is the batch size too large and the data not fitting into the GPU’s VRAM, causing a slowdown?
2025-01-08T13:44:01Z
Shouldn’t be the case. The GPU got 12 GB and the data isn’t that large either.
2025-01-08T14:05:42Z
Hmm. Anyway, there are no documents…huggingface.coTraining and Finetuning Embedding Models with Sentence Transformers v3We’re on a journey to advance and democratize artificial intelligence through open source and open science.an example for the main_distance_functionMaybe these.github.comUKPLab/sentence-transformers/blob/master/sentence_transformers/util.py#L111Tensor: Matrix with res[i][j] = cos_sim(a[i], b[j])"""a = _convert_to_batch_tensor(a)b = _convert_to_batch_tensor(b)a_norm = normalize_embeddings(a)b_norm = normalize_embeddings(b)return torch.mm(a_norm, b_norm.transpose(0, 1))def pairwise_cos_sim(a: Tensor, b: Tensor) -> Tensor:"""Computes the pairwise cosine similarity cos_sim(a[i], b[i]).Args:a (Union[list, np.ndarray, Tensor]): The first tensor.b (Union[list, np.ndarray, Tensor]): The second tensor.Returns:Tensor: Vector with res[i] = cos_sim(a[i], b[i])"""fromsentence-transformers/sentence_transformers/similarity_functions.py at master · UKPLab/sentence-transformers · GitHub