Loading COLNN already split in sentences
I’m writing a custom pipeline to train a DistilBERT for token classification on some data that I got in a COLLN file. This file is already split in sentences, for example:Hello,0world,0This,0is,0a,0post,0The split between sentences is done with just an empty line as it is required by the run_ner.py script. Right now, I’m loading the dataset with the datasets package, such as:dataset = load_dataset('csv', data_files={
    'train': DATA_FOLDER + 'train.csv',
    'test': DATA_FOLDER + 'test.csv',
    'dev': DATA_FOLDER + 'dev.csv'
}, sep="\t", column_names=["tokens", "tags"])But it looks like the split on sentences is not considered and all the tokens are loaded in a consecutive manner.Is there something to make load_dataset understand that the data is split in sentences and tokens?