K fold cross validation
Hi,I use Trainer() to fine-tune bert-base-cased model on NER task.I split my dataset with sklearn.model_selection.train_test_split .Now, I want to use k fold cross validation to split dataset and fine-tune the model.Does anyone try the same way? plz tell me if you have any ideas.
2021-04-26T10:16:52Z
one suggestion would be to use thesplitfunctionality ofdatasetsto create your folds as described here:Splits and slicing — datasets 1.6.0 documentationthen you could use a loop to fine-tune on each fold with the trainer and aggregate the predictions per fold
2022-04-19T14:29:50Z
yucheng:Now, I want to use k fold cross validation to split dataset and fine-tune the model.Very old thread I know, but here’s an alternative to@lewtun’s solution that I like:import numpy as np
from sklearn.model_selection import StratifiedKFold

from datasets import load_dataset

# First make the kfold object
folds = StratifiedKFold(n_splits=5)

# Then get the dataset
datasets = load_dataset("glue", "mrpc")

# Now make our splits based off of the labels. 
# We can use `np.zeros()` here since it only works off of indices, we really care about the labels
splits = folds.split(np.zeros(datasets["train"].num_rows), datasets["train"]["label"])

# Finally, do what you want with it
# In this case I'm overriding the train/val/test
for train_idxs, val_idxs in splits:
    datasets = load_dataset("glue", "mrpc")
    datasets["test"] = datasets["validation"]
    datasets["validation"] = datasets["train"].select(val_idxs)
    datasets["train"] = datasets["train"].select(train_idxs)For a method without having to reload the dataset, you can also do:from datasets import DatasetDict
fold_dataset = DatasetDict({
    "train":datasets["train"].select(train_idx),
    "validation":datasets["train"].select(val_idx),
    "test":datasets["validation"]
})
2022-11-27T04:53:58Z
import numpy as np
from sklearn.model_selection import StratifiedKFold
from datasets import load_dataset

# First make the kfold object
folds = StratifiedKFold(n_splits=10)

# Now make our splits based off of the labels. 
# We can use `np.zeros()` here since it only works off of indices, we really care about the labels
splits = folds.split(np.zeros(datasets["train"].num_rows), datasets["train"]["tags"])
# Finally, do what you want with it
# In this case I'm overriding the train/val/test
for train_idxs, val_idxs in test:
    fold_dataset = DatasetDict({
        "train":datasets["train"].select(train_idxs),
        "validation":datasets["train"].select(val_idxs),
        "test":datasets["validation"]
    })How to do it the same for multi class label? I get this error…ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.But if I add MultiClassBinarizer I get thissplits = folds.split(np.zeros(datasets["train"].num_rows), MultiLabelBinarizer().fit_transform(datasets["train"]["tags"]))ValueError: Supported target types are: (‘binary’, ‘multiclass’). Got ‘multilabel-indicator’ instead.
2023-01-09T03:57:43Z
I like your approach, as you can use sklean CV methods (in particular, the Grouping ones that aren’t yet in huggingface datasets as of Jan '23).You do have an issue though: you’ve overloaded the name “datasets”. Your code will run but it’s a bit confusing, as@scostinianofound out!@scostiniano: Take@muellerzr's first version of the code up to# Finally, do what I want with it, and replace from there down with the second version of the code, then it should work.Ideally, you should have somehting likeglue_datasets = load_dataset("glue", "mrpc")to avoid the confusion (and retain access to thedatasetsmodule, which is no longer accessible with your current code!)
2023-07-29T04:24:05Z
is there an example script on how to call the Trainer() 5 times(5fold) with different train-test splits?