Does the BERT model choose only the inputs it needs?
Hi, I am a new user of Transformers and I have successfully fine-tuned a BERT model following the tutorial.However, I have one question about the features I send to the Trainer and those accepted by the BERT model.Specifically, my original dataset contains two columns named “text” and “label”. After tokenizing the “text”, the dataset object now have three more columns named “input_ids”, “token_type_ids”, and “attention_mask”. I understand that these three columns are required by the BERT model, but I didn’t drop the original “text” column when I fed the dataset to the Trainer API.So my question is, does BERT automatically ignore non-relevant features? (maybe this is achieved quietly by the Trainer API) Or should I remove these columns, leaving only “input_ids”, “token_type_ids”, and “attention_mask”?Thank you!
2022-07-01T09:36:51Z
The Trainer will ignore irrelevant columns in your input data.github.comhuggingface/transformers/blob/d0acc9537829e7d067edbb791473bbceb2ecf056/src/transformers/trainer_utils.py#L678-L692def _remove_columns(self, feature: dict) -> dict:if not isinstance(feature, dict):return featureif not self.message_logged and self.logger and self.model_name:ignored_columns = list(set(feature.keys()) - set(self.signature_columns))if len(ignored_columns) > 0:dset_description = "" if self.description is None else f"in the {self.description} set"self.logger.info(f"The following columns {dset_description} don't have a corresponding argument in "f"`{self.model_name}.forward` and have been ignored: {', '.join(ignored_columns)}."f" If {', '.join(ignored_columns)} are not expected by `{self.model_name}.forward`, "" you can safely ignore this message.")self.message_logged = Truereturn {k: v for k, v in feature.items() if k in self.signature_columns}