Args:
            df (`pyspark.sql.DataFrame`):
                The DataFrame containing the desired data.
            split (`NamedSplit`, *optional*):
                Split name to be assigned to the dataset.
            features (`Features`, *optional*):
                Dataset features.
            cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
                Directory to cache data. When using a multi-node Spark cluster, the cache_dir must be accessible to both
                workers and the driver.
            keep_in_memory (`bool`):
                Whether to copy the data in-memory.
            working_dir (`str`, *optional*)
                Intermediate directory for each Spark worker to write data to before moving it to `cache_dir`. Setting
                a non-NFS intermediate directory may improve performance.
            load_from_cache_file (`bool`):
                Whether to load the dataset from the cache if possible.