Dataset map return only list instead torch tensors
when i use the map on Dataset object the 'input_ids return as list instead of tensorsdef tokenize(batch):
  return tokenizer(batch['text'],padding=True, return_tensors='pt', truncation=True).to(DEVICE)data.map(tokenize, batched=False, batch_size=None)→ return listtokenizer(data['text'],padding=True, return_tensors='pt', truncation=True).to(DEVICE)→ return tensor
2022-03-21T12:18:12Z
Hi!mapignores tensor formatting while writing a cache file, so to get PyTorch tensors under theinput_idscolumn, you need to explicitly callset_format("pt", columns=["input_ids"], output_all_columns=True)on the dataset object (aftermap).
2022-09-29T22:36:57Z
I got burned by this as well. Maybe the documentation (examples) for map could be updated to explain this?
2022-12-21T04:18:44Z
I got burned by this too! Definitely seems like an issue in the library that.mapdoesn’t return the objects that the mapped function returns.And I’d just finished the course which has awhole sectionabout bugs caused by not settingreturn_tensors, but the examples never mention this issue with.map.
2022-12-22T19:49:18Z
Hi! When it comes to tensors, PyArrow (the storage format we use) only understands 1D arrays, so we would have to store (potentially) a significant amount of metadata to be able to restore the types aftermapfully. Also, amaptransform can return different value types for the same column (e.g. PyTorch tensors or Python lists), which would make this process ambiguous (e.g. should we return mixed types or only one type when indexing the dataset afterwards). So for the sake of simplicity, we return Python objects by default, and for other formats, one can usewith_format/with_transform.
2023-06-27T11:01:18Z
I ended up using a data collator to take the lists and turn them back into tensors
2024-03-01T16:12:13Z
Yep, I also got burned by this. What is the point here to return list of lists when it’s a torch tensor? Inside the map function, it’s a tensor, then after it’s a list?
2024-04-10T15:48:41Z
Also burned by this coercion, would be helpful to note the PyArrow limitation (althoughTensormay be helpful here?) and workarounds in the docs…Edit: looks like there’s some discussion on how to use PyArrow’s Tensor type atUse pyarrow Tensor dtype · Issue #5272 · huggingface/datasets · GitHub