title: HuggingFace Trainer Segmentation Fault
body: <p>Huggingface Trainer keeps giving <code>Segmentation Fault</code> with this setup code.
  The dataset is around 600MB, and the server has 2*32GB Nvidia V100. Can anyone help find the issue?</p>
  <pre><code>from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling, LineByLineTextDataset
  from transformers import GPT2Config, GPT2LMHeadModel, GPT2TokenizerFast
  
  tokenizer = GPT2TokenizerFast.from_pretrained(&quot;./data/TOKEN&quot;)
  
  config = GPT2Config.from_pretrained('gpt2-large')
  model = GPT2LMHeadModel(config=config)
  tokenizer = GPT2TokenizerFast.from_pretrained(&quot;./data/TOKEN&quot;, model_max_length=1024)
  
  print('loading dataset...')
  dataset = LineByLineTextDataset(
      tokenizer=tokenizer,
      file_path=&quot;./data/kowiki.txt&quot;,
      block_size=128,
  )
  
  training_args = TrainingArguments(
      output_dir='./m',          # output directory
      num_train_epochs=1,              # total # of training epochs
      per_device_train_batch_size=1,  # batch size per device during training - the higher the better, but may OOM
      per_device_eval_batch_size=1,   # batch size for evaluation
      logging_dir='./logs',            # directory for storing logs
      save_steps=10000,
      do_train=True
  )
  
  trainer = Trainer(
      model=model,                         # the instantiated  Transformers model to be trained
      args=training_args,                  # training arguments, defined above
      train_dataset=dataset,         # training dataset
  )
  
  trainer.train()
  </code></pre>
  <p>Error message :</p>
  <pre><code>loading dataset...
  Epoch:   0%|                                              | 0/1 [00:00&lt;?, ?it/s]
  Fatal Python error: Segmentation fault                | 0/99996 [00:00&lt;?, ?it/s]
  
  Thread 0x00007f872dfff700 (most recent call first):
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 299 in wait
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 551 in wait
    File &quot;/opt/conda/lib/python3.6/site-packages/tqdm/_monitor.py&quot;, line 69 in run
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 916 in _bootstrap_inner
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 884 in _bootstrap
  
  Thread 0x00007f8736bb5700 (most recent call first):
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 299 in wait
    File &quot;/opt/conda/lib/python3.6/queue.py&quot;, line 173 in get
    File &quot;/opt/conda/lib/python3.6/site-packages/tensorboard/summary/writer/event_file_writer.py&quot;, line 205 in run
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 916 in _bootstrap_inner
    File &quot;/opt/conda/lib/python3.6/threading.py&quot;, line 884 in _bootstrap
  
  Current thread 0x00007f88273e7740 (most recent call first):
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py&quot;, line 39 in broadcast_coalesced
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py&quot;, line 21 in forward
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py&quot;, line 71 in _broadcast_coalesced_reshape
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py&quot;, line 88 in replicate
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py&quot;, line 159 in replicate
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py&quot;, line 154 in forward
    File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py&quot;, line 577 in __call__
    File &quot;/opt/conda/lib/python3.6/site-packages/transformers/trainer.py&quot;, line 622 in _training_step
    File &quot;/opt/conda/lib/python3.6/site-packages/transformers/trainer.py&quot;, line 499 in train
    File &quot;trainer.py&quot;, line 34 in &lt;module&gt;
  Segmentation fault (core dumped)
  
  </code></pre>
  <p>Python version is 3.7.7 with Pytorch 1.5.1+cu101
  with a 'recently installed' HF transformer &amp; tokenizer(0.8.0).</p>
  <p>First time here :D sorry if I didn't keep trivial stuff</p>
  <p>EDIT : May be a bug - <a href="https://github.com/huggingface/transformers/issues/5590" rel="nofollow noreferrer">https://github.com/huggingface/transformers/issues/5590</a></p>
  <p>EDIT 2: Also segfaults on transformer 3.0.2 and tokenizer 0.7.0, 0.8.1-rc1</p>