Trying to Build Datasets, Random Items Get Added
Hi all,I’m currently trying to load in fastai’s version of theIMDBdataset, to learn how to build aDatasetfrom a folder of.txt's. I’m preparing my data with the following:# Downloading the dataset
from fastai.data.external import untar_data, URLs
from fastai.data.transforms import get_files

path = untar_data(URLs.IMDB, dest='./IMDB')From there I can get all the training.txt's with:texts = get_files(path/'train', extensions='.txt')
texts = [str(t) for t in texts]In turn, this is a list of 25,000 text files.However, when I use theload_datasetapi to bring this in, suddenly my dataset has 25,682 items! Can anyone help me figure out why? This is an issue as I need to useadd_columnto add in a label, and there is a mismatch between the number of actual training items vs the onesDatasetspicked up. Here is how I’m building the dataset:dset = load_dataset('text', data_files={'train':texts})TIA!
2021-07-27T13:20:18Z
I think it’s because the “text” loader creates a new sample for each “\n” it sees, so the texts you have that contain some of those are then split into several samples.@lhoestqor@albertvillanovaif you could conifrm?PS: it would be easier to just dodsets = load_dataset('imdb'):-p
2021-07-27T13:28:11Z
sgugger:I think it’s because the “text” loader creates a new sample for each “\n” it sees, so the texts you have that contain some of those are then split into several samples.@lhoestqor@albertvillanovaif you could conifrm?I don’tthinkso, when I tried checking for that I still got 25,000. Or a better way to put that is this returns zero:from fastcore.xtras import open_file

count = 0
for text in texts:
    t = open_file(text).read()
    if '\n' in t or '\r' in t: count += 1sgugger:PS: it would be easier to just dodsets = load_dataset('imdb'):-pOf course it would be! However I’m trying to write a high-level data API for adaptnlp currently, so I’m only using IMDB as a situational test caseEdit: Trying a new way to verify, will update with those resultsAha!@sguggerthank you! There were some hidden\x85characters, which is the source of the breakage.I can work with that now. Thank you!(If you have recommendations for fixes, I’m all ears, I was just going to take that into account while mapping labels from folder names)