Filtering Dataset
I’m trying to filter a dataset based on the ids in a list. This approach is too slow. The dataset is an Arrow dataset.responses = load_dataset('peixian/rtGender', 'responses', split = 'train')
# post_id_test_list contains list of ids
responses_test = responses.filter(lambda x: x['post_id'] in post_id_test_list)
2021-09-24T13:43:02Z
Hi baumstan.I’m not sure I understand the question.   Why does it matter if it is slow?I would expect you to create and then save your train/test datasets only once, before you start using your model.  If it takes a long time, just leave it running.Are you trying to use a dynamic post_id_test_list, or to train with transient data, or what?I suspect you might find better answers on Stack Overflow, as this doesn’t look like a Huggingface-specific question.
2021-09-26T18:16:59Z
Have tried Stackoverflow.python - pyarrow Table Filtering -- huggingface - Stack Overflow
2024-04-08T08:36:39Z
I think settinginput_columnsis what you want. Seemy answeron Stack Overflow.