[HELP] Special tokens not appearing as predicted tokens!
Hello,I require some urgent help!I trained a masked language model on a Twitter dataset, with each tweet containing one emoji. Then, I used the following code to add the emojis as special tokens:num_added_toks = tokenizer.add_tokens(['😃',
'😄',
'😁',
'😆',
'😅',
'😂',
'🤣',
'🧔🏿‍♂️'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizerFrom adding the special tokens, I added 3311 different emojis successfully, which increased the embedding to (53575, 768) as shown below:We have added 3311 tokens

Embedding(53575, 768)Now, here’s the issue I am facing… When I add the<mask>token to a sentence and input thetop_kas the total number of embeddings, which is 53575, not a single emoji shows up in the predictions.I used this line of code:mask_filler("Are you happy today <mask>", top_k=53575)As you can see in the code above, thetop_kis 53575, the total number of embeddings which should include the 3311 emojis I added, right?However, when I make the predictions and scroll through the list of 53575, not a single emoji is there!Why is this is happening? Like, I have added the emojis to the vocabulary, but they are simple not there when making predictions.SEE FULL CODE HERE:MLM-EMOJIS/mlm_emojis.ipynb at main · saucyhambon/MLM-EMOJIS · GitHub
2021-08-02T11:29:25Z
I see one emoji in the last predictions. Mostly, it’s that the model is not used to seeing those, so it probably needs to be trained longer.
2021-08-02T13:43:18Z
Hello@sgugger. I appreciate the response. How much data would you recommend?
2021-08-03T10:13:43Z
So if I use more data to train, will all the emojis show up as predictions? Only asking, because I am doing this as part of my MSc project, and I am having no success.
2021-08-03T10:40:07Z
@sguggerShouldn’t all tokens have at leastsomeoutput probability? The output predictions are over the full size of the vocabulary after all?
2021-08-03T10:46:48Z
@BramVanroyThat is what I was thinking! If emojis are in the vocab, which they are, they should be predicted tokens when making predictions right? Like, my vocab size with the emojis is 53575, and when making predictions, there is one 1 predicted emoji in the vocab size of 53575 - which means something is going wrong. What do you think?
2021-08-03T10:57:16Z
BramVanroy:@sguggerShouldn’t all tokens have at leastsomeoutput probability? The output predictions are over the full size of the vocabulary after all?I don’t understand the question: if there is one prediction that is an emoji, clearly they have output probabilities. It’s just that for the sentence used (“I am feeling good today my friend ”), those probabilities are lower than punctuation symbols, which seems logical to me given the fast BERT has learned to do that over a huge corpus and a long training, a habit it won’t lose in a small fine-tuning of three epochs over 1700 samples.
2021-08-03T11:09:27Z
Yes, but even changing the sentence, I still get that same predicted emoji.
2021-08-03T16:16:11Z
Update: So…I trained my model on 70,000 records, and I still only get 1 emoji. Something is definitely not right.
2021-08-04T14:00:06Z
If OP adds 3311 emojis to the vocabulary, then you’d expect that the probability vector for the mask token inAre you happy today <mask>includes probabilities for all the original tokens of the model as well as probabilities for all the 3311 emojis.OP seems to suggest that the output probabilities for<mask>do not include probabilities for those added emojis, i.e. the output probabilities do not have the same size as the vocabulary size. That is at least what I understand from OP’s post.
2021-08-04T14:09:14Z
When how did the pipeline predict an emoji if it’s not in the vocab?There is no shape computation anywhere in the notebook, just a pipeline use, so there is no way to see if the model outputted probabilities for each emoji for sure (I’m pretty sure it did, they are just low).
2021-08-04T14:45:24Z
I would think the same, but OP wrote that they triedmask_filler("Are you happy today <mask>", top_k=53575)where53575is the resized embedding size, and that in that output they can only find one emoji.
2021-08-04T14:48:25Z
That is correct indeed. After adding the emojis to the vocab and resizing it, I’d expect all 3311 emojis to appear when viewing thetop_k=53575. Hence, that 53575 contains all the emojis as far as I am aware; therefore, they should be predictions.
2021-08-04T15:02:26Z
Not sure that’s a use case the pipeline properly supports, as it’s a very large value. In any case to debug further, you should dig into the predictions manually and go through the model and its output.
2021-08-04T15:16:36Z
What do you mean by ‘dig into the predictions manually?’ I have already gone through all 53575 predicted tokens when making predictions, and there was only 1 emoji as you know.