Loading and saving the German Train.1h split of the multilingual_librispeech (MLS) dataset
I want to download just the GermanTrain.1hsplit of themultilingual_librispeech(MLS) dataset and save it to my local machine. After some trial and error, I ended up using the following command:mls_german_train_1h_dataset = load_dataset('multilingual_librispeech', 'german', split='Train.1h', cache_dir='/path/to/mls-german-train-1h')However, as I started running it, my terminal told me I was downloading 123 GB of data. I thought, “Can that be right? Is the smallest training split of the MLS German dataset really so big?” Not wanting to download so much data, I interrupted the command.For comparison, I then started downloading the GermanTrain.9hsplit instead:mls_german_train_9h_dataset = load_dataset('multilingual_librispeech', 'german', split='Train.9h', cache_dir='/mnt/nvdl/usr/schilton/mls-german-train-9h')Once again, the terminal told me that I was downloading 123 GB.Have I done something wrong in my calls toload_dataset(), or will Python sift through all 123 GB of data and only save the specified split?
2022-05-02T13:39:57Z
Hi! The data for bothtrain.1handtrain.9hcomes from the same folder hence the equal download size, and this dataset doesn’t support streaming (currently), so downloading the entire data is the only option.