TFBertModel for classification task with no CLS token
Hello, I’m reading a paper where BERT (TFBertModel) and RoBerta (TFRobertaModel) are used to solve a text classification task.PreambleGoing through the implementation, I noticed that each text sample is tokenized with no special tokenstokenizer.encode(sentence, add_special_tokens=False).Later on the outputs of the tokenizers are passed to the respective models and thepooled outputis retrieved, as follows:embedding_BERT = encoder_BERT(
    input_ids_BERT,
    token_type_ids=token_type_ids_BERT,
    attention_mask=attention_mask_BERT
)['pooler_output']QuestionsThe authors claim to be using the[CLS]tokens produced by both models. However, how can this be the case if the tokenizers encoded the text samples without including the special tokens?Ifadd_special_tokensisFalse, does the first token of each text sample still encode knowledge about the whole sequence as it usually is the case with[CLS]?The authors actually use thepooled output, which is produced byBertPooler. Can its output still be considered as theCLStoken?Referencespaper:https://ceur-ws.org/Vol-3202/politices-paper1.pdfcode:PoliticES2022/PoliticES.ipynb at main · ssantamaria94/PoliticES2022 · GitHub