Today Unsupervised Sentence Transformers, Tomorrow Skynet (how TSDAE works)
To adapt a pretrained transformer to produce meaningful sentence vectors, we typically need a more supervised fine-tuning approach. We can use datasets like natural language inference (NLI) pairs, labeled semantic textual similarity (STS) data, or parallel data (pairs of translations).

For some domains and languages, such as finance and English, this data is fairly easy to find or gather. But many domains and many languages have very little labeled data. If you can find semantic similarity pairs for the agriculture industry, please let me know. There are many languages, such as Dhivehi, where unlabelled data is hard to find and labelled data practically non-existent.

This means you either spend a very long time gathering tens of thousands of labeled samples or you can try an unsupervised fine-tuning approach.

Unsupervised training methods for sentence transformers are not as effective as their supervised counterparts, but they do work. And if you have no other choice, why not?

In this video, we will introduce the concept of unsupervised fine-tuning for sentence transformers. We will learn to train these models using the unsupervised Transformer-based Sequential Denoising Auto-Encoder (TSDAE) approach.

🌲 Pinecone article:
https://www.pinecone.io/learn/unsupervised-training-sentence-transformers/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Why Language Embedding Matters
05:12 Supervised Methods
05:29 Natural Language Inference
07:15 Semantic Textual Similarity
07:43 Multilingual Training
10:00 TSDAE (Unsupervised)
18:50 Data Preparation
29:05 Initialize Model
32:39 Model Training
36:25 NLTK Error
37:15 Evaluation
41:01 TSDAE vs Supervised Methods
42:42 Why TSDAE is Cool