How to apply a trained token classification model to tokens?
This feels like something that should be easy, but I could not find out how to do it:I have trained a token classification model for NER like shown in the courseToken classification - Hugging Face Courseand the trained model annotates texts when I use the pipeline for token-classification as suggested in the chapter.However, I want to instead apply the model to basically exactly the same kind of data that was used for training.So for training, a single example would look like{'id': '0', 'tokens': ['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.'], 'ner_tags': [3, 0, 7, 0, 0, 0, 7, 0, 0]}(or with the NER tag ids replaced with the actual tag strings B-PER etc.)Now I want to apply the model to basically something like['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.']or a whole batch of such sequences and want to get back the corresponding tag ids or tag labels.How does one do that?