Selective masking in Language modeling
Hi HuggingfacersI have a number of questions regarding finetuning a language model:How to mask a selective portion of a given input sentence instead of masking randomly.For example, if I am using ALBERT as a model, and I am aiming to do a different kind of loss function than the standard MLM loss for the masked tokens, how to access the model output of the masked tokens
2020-08-13T14:26:22Z
Refer to:https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=M1oqh0F6W3adMask Code:github.comhuggingface/transformers/blob/master/src/transformers/data/data_collator.py#L157if are_tensors_same_length:return torch.stack(examples, dim=0)else:if self.tokenizer._pad_token is None:raise ValueError("You are attempting to pad samples but the tokenizer you are using"f" ({self.tokenizer.__class__.__name__}) does not have one.")return pad_sequence(examples, batch_first=True, padding_value=self.tokenizer.pad_token_id)def mask_tokens(self, inputs: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:"""Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original."""if self.tokenizer.mask_token is None:raise ValueError("This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.")labels = inputs.clone()