Datasets + Arrow Help
Hi, I’m just getting started and am excited that Datasets is built on Arrow. But I haven’t seen how to access the Arrow data. For example, how do I use pyarrow or Polars on loaded training data?
2022-06-09T07:33:20Z
You should be able to access the underlying Arrow data through a datasets_data. Note that such usage is not intended, though. EDIT: see@mariosasko’s reply. I was a bit too quick, you also have a public propertydatathat you can use.github.comhuggingface/datasets/blob/747f0c8612d4929dbdd1c72cca201815911ce660/src/datasets/arrow_dataset.py#L640arrow_table: Table,info: Optional[DatasetInfo] = None,split: Optional[NamedSplit] = None,indices_table: Optional[Table] = None,fingerprint: Optional[str] = None,):info = info.copy() if info is not None else DatasetInfo()DatasetInfoMixin.__init__(self, info=info, split=split)IndexableMixin.__init__(self)self._data: Table = _check_table(arrow_table)self._indices: Optional[Table] = _check_table(indices_table) if indices_table is not None else Nonemaybe_register_dataset_for_temp_dir_deletion(self)self._format_type: Optional[str] = Noneself._format_kwargs: dict = {}self._format_columns: Optional[list] = Noneself._output_all_columns: bool = Falseself._fingerprint: str = fingerprint# Read metadata
2022-06-09T11:27:38Z
Hi! The underlying Arrow table can be accessed using thedset.data.tableattribute, which can then be loaded in Polars as follows:import polars as pl
from datasets import load_dataset
dset = load_dataset(...)
df = pl.from_arrow(dset.data.table)