IOB tagging for NER / DatasetDict
Hello everyone!I have created a custom dataset in the IOB tagging format. I want to follow the HF tutorial for Token Classification.Sydney _ _ B-ORGAirport _ _ I-ORGannounces _ _ Orange _ _ Oof _ _ Onew _ _ Osustainability _ _ Ogoals _ _ OFinland  _ _  B-LOCATIONis _ _ Oon _ _ Otrack _ _ Oto _ _ Obe _ _ Ocarbon _ _ B-SUSTAINABILITYneutral _ _ I-SUSTAINABILITYby _ _ O2035 _ _ B-DATEWhen I am loading my dataset with load_dataset(). I only got “text”… But not the “labels”.DatasetDict({train: Dataset({features: [‘text’],num_rows: 285996})})When I look at the structure in the data loader it’s as follows.‘Sydney _ _ B-ORG’,‘Airport _ _ I-ORG’,‘announces _ _ O’,‘range _ _ O’,‘of _ _ O’,‘new _ _ O’,‘sustainability _ _ O’,‘goals _ _ O’,‘’,‘’,Did I miss something?How can I create a similar structure of the WNUT 17 dataset (“id”, “ner_tags”, and “tokens” ) in the tutorial?Thank you for your help.Roman
2023-11-20T09:23:06Z
Did you find a solution for that? I’m having the same trouble as well.