Creating a dataset with custom data
Hey there, I’m trying to create a DatasetDict with two datasets(train and dev) for fine tuning a bart model.I’ve created lists of source sentences, target sentences and id’s, they are lists of strings.data = DatasetDict({
    "train": Dataset.from_dict({
        "id": train_idxs,
        "translation": {
            "source": train_inputs,
            "target": train_labels
        }
    }, features=Features({"id": Value(dtype='string'), "translation": {"source": Sequence, "target": Sequence}})),
    "dev": {
        "id": dev_idxs,
        "translation": {
            "source": dev_inputs,
            "target": dev_labels
        }
    }
})is the code I’m using to create the DatasetDict, but I get errorTypeError: string indices must be integersI want the object to have the same structure as the “Books” datasetdict that is used in this guideTranslationif anyone has any suggestions please let me know, as well as if I need to provide more information!Thank you!
2022-09-02T23:48:55Z
Hello! I don’t think that you should need to manually invokeDatasetDict. Instead, since you’re loading text data you can use one of the methods listed here:LoadIf you have a bunch of CSV fileswith the headerssourceandtarget, you can load them withfrom datasets import load_dataset
data = load_dataset("csv", data_files={"train": ["my_text_1.txt", "my_text_2.txt"], "validate": "my_test_3.txt"})In general, I think it’s best to avoidDatasetDictand instead rely on the helper methods as much as possible.Hope that helps!
2022-09-05T10:51:00Z
Hi@edensn! You can use the following features dictionary instead of the current one to match the features:Features({"id": Value(dtype='string'), "translation": Translation(languages=["source", "target"])})
2022-09-05T13:57:33Z
Thank you both of you guys for the suggestions! I actually figured it out using the method that@mariosaskosuggested by just opening the Books dataset I was trying to mimic and doing just looking at books.features!