title: How to train BERT with custom (raw text) domain-specific dataset using Huggingface?
body: <p>I would like to train / fine-tune the BERT model with some own datasets which contain just raw data from a specific domain, similar to BioBERT. In the first step I just want to train BERT with this raw data, without fine-tuning with any specific NLP task. So, I want to build a base which I can potentially use for fine-tuning with annotated train data for specific tasks, such as NER, etc.
  Is this possible in the way I plan to do it?</p>
  <p>I tried to use the example scripts of Huggingface: <a href="https://github.com/huggingface/transformers/tree/master/examples/language-modeling" rel="nofollow noreferrer">https://github.com/huggingface/transformers/tree/master/examples/language-modeling</a></p>
  <p>I used the run_mlm script in the following way:</p>
  <pre><code>python run_mlm.py \
      --model_name_or_path bert-base-uncased\
      --train_file /path/to/text_file.txt \
      --do_train \
      --output_dir /out
  </code></pre>
  <p>I get some output but how can I evaluate the resulting model?
  There are some intrinsic evaluation methods that I wanted to try but I failed, like word similarity or word analogy.</p>
  <p>If you could help me with my approach I would be very thankful.</p>