How to get file paths when iterating over a custom dataset with KeyDataset?
Hi!I created a dataset from a folder with some mp3 files and tried to iterate over them with KeyDataset:import datasets
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset
from tqdm.auto import tqdm

dataset_mp3 = load_dataset("audiofolder", data_dir="/content/drive/MyDrive/Temp/Mp3", drop_metadata=True).cast_column("audio", Audio(sampling_rate=16000))

# KeyDataset (only *pt*) will simply return the item in the dict returned by the dataset item
# as we're not interested in the *target* part of the dataset. For sentence pair use KeyPairDataset

for out in tqdm(pipe(KeyDataset(dataset_mp3["train"], "audio"))):
    print(out)
    # {"text": "My audio transcription"}
    # {"text": ....}
    # ....For now I’m getting the output for the processing I want (“text”), but how to get the file path corresponding to each output?P.S: I’m using Google Colab.Thanks!
2023-10-06T13:22:07Z
I am not sure if this is nice solution but it works…First of all createmetadata.csvin yourdata_dirwhere you put thefile_nameand someid.Load the dataset as before… Now thedataset_mp3["train"][1] == {
   "audio": "path/to/audio/file",
   "id": "id from the metadata.csv"
}Now change the for loop into:for item in tqdm(KeyPairDataset(dataset_mp3["train"], "audio", "id")):
    out = pipe(item["text"])
    print(out, item["text_pair"])It is strange, but take a look herehttps://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/pt_utils.pyand you will see how it worksIt can be simplifed if you need justfile_namejust iterate over KeyDataset and run pipe inside the loop.Hope this helps.