Create large dataset with PIL image loaded in 'image' column
I have a dataset of 4M images with metadata that I already formatted as HF dataset from the pandas DF.I would like to add a column that has the PIL image as a lot of these dataset have.With the size of the dataset I need to process I am not sure what is the best approach to do it as running a lip or map will def go OOMAny ideas?also how to create the json file so that if I save to disk then I can load it usingload_datasetand notload_from_disk?