size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: instruction
    dtype: string
  - name: output
    dtype: string
  - name: cluster
    dtype: int64
  splits:
  - name: train
    num_bytes: 131460545
    num_examples: 70000
  download_size: 69216368
  dataset_size: 131460545
---
# Dataset Card for "Wizard-EvolInstruct70k-k8"

`centers.pt` in the files is a 8x384 matrix including the centers of each cluster. I use `sentence-transformers/all-MiniLM-L6-v2` to encode text.

```python
import torch
from sentence_transformers import SentenceTransformer

sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
embeddings = torch.tensor(model.encode(sentences))
centers = torch.load("centers.pt")

# mse based cluster choice
clusters = (embeddings - centers).pow(2).mean(1).argmin().tolist()

# or you could load the sklearn kmeans classifier
# todo: documentation for that
# todo: figure out how to do that
# todo: cant you push sklearn classifiers to the hub with some weird code introduced earlier this year or something
```