title: How to i get word embeddings for out of vocabulary words using a transformer model?
body: <p>When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because &quot;embeddings&quot; is an out of vocabulary word/token, that is being split into <code>em</code>, <code>bed</code> ,<code>ding</code>, <code>s</code>.</p>
  <p>I would like to know if there is any aggregation strategies available that make sense apart from doing a mean of these vectors.</p>
  <pre class="lang-py prettyprint-override"><code>from transformers import AutoTokenizer, AutoModel
  # download and load model
  tokenizer = AutoTokenizer.from_pretrained(&quot;emilyalsentzer/Bio_ClinicalBERT&quot;)
  model = AutoModel.from_pretrained(&quot;emilyalsentzer/Bio_ClinicalBERT&quot;)
  
  sentences = ['This framework generates embeddings for each input sentence']
  
  
  #Tokenize sentences
  encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')
  
  
  #Compute token embeddings
  with torch.no_grad():
      model_output = model(**encoded_input)
  
  print(encoded_input['input_ids'].shape)
  </code></pre>
  <p>Output:
  <code>torch.Size([1, 13])</code></p>
  <pre class="lang-py prettyprint-override"><code>for token in encoded_input['input_ids'][0]:
        print(tokenizer.decode([token]))
  </code></pre>
  <p>Output:</p>
  <pre><code>[CLS]
  this
  framework
  generates
  em
  ##bed
  ##ding
  ##s
  for
  each
  input
  sentence
  [SEP]
  </code></pre>