title: Sliding window for long text in BERT for Question Answering
body: <p>I've read post which explains how the sliding window works but I cannot find any information on how it is actually implemented.</p>
  <p>From what I understand if the input are too long, sliding window can be used to process the text.</p>
  <p>Please correct me if I am wrong.
  Say I have a text <em><strong>&quot;In June 2017 Kaggle announced that it passed 1 million registered users&quot;</strong></em>.</p>
  <p>Given some <code>stride</code> and <code>max_len</code>, the input can be split into chunks with over lapping words (not considering padding).</p>
  <pre><code>In June 2017 Kaggle announced that # chunk 1
  announced that it passed 1 million # chunk 2
  1 million registered users # chunk 3
  </code></pre>
  <p>If my questions were <em><strong>&quot;when did Kaggle make the announcement&quot;</strong></em> and <em><strong>&quot;how many registered users&quot;</strong></em> I can use <code>chunk 1</code> and <code>chunk 3</code> and <strong>not use</strong> <code>chunk 2</code> <strong>at all</strong> in the model. Not quiet sure if I should still use <code>chunk 2</code> to train the model</p>
  <p>So the input will be:
  <code>[CLS]when did Kaggle make the announcement[SEP]In June 2017 Kaggle announced that[SEP]</code>
  and
  <code>[CLS]how many registered users[SEP]1 million registered users[SEP]</code></p>
  <hr>
  <p>Then if I have a question with no answers do I feed it into the model with all chunks like and indicate the starting and ending index as <strong>-1</strong>? For example <em><strong>&quot;can pigs fly?&quot;</strong></em></p>
  <p><code>[CLS]can pigs fly[SEP]In June 2017 Kaggle announced that[SEP]</code></p>
  <p><code>[CLS]can pigs fly[SEP]announced that it passed 1 million[SEP]</code></p>
  <p><code>[CLS]can pigs fly[SEP]1 million registered users[SEP]</code></p>
  <hr>
  <p>As suggested in the comments, II tried to run <code>squad_convert_example_to_features</code> (<a href="https://github.com/huggingface/transformers/blob/1af58c07064d8f4580909527a8f18de226b226ee/src/transformers/data/processors/squad.py#L134" rel="noreferrer">source code</a>) to investigate the problem I have above, but it doesn't seem to work, nor there are any documentation. It seems like <code>run_squad.py</code> from huggingface uses <code>squad_convert_example_to_features</code> with the <code>s</code> in <code>example</code>.</p>
  <pre class="lang-py prettyprint-override"><code>from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor, squad_convert_example_to_features
  from transformers import AutoTokenizer, AutoConfig, squad_convert_examples_to_features
  
  FILE_DIR = &quot;.&quot;
  
  tokenizer = AutoTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
  processor = SquadV2Processor()
  examples = processor.get_train_examples(FILE_DIR)
  
  features = squad_convert_example_to_features(
      example=examples[0],
      max_seq_length=384,
      doc_stride=128,
      max_query_length=64,
      is_training=True,
  )
  </code></pre>
  <p>I get the error.</p>
  <pre><code>100%|██████████| 1/1 [00:00&lt;00:00, 159.95it/s]
  Traceback (most recent call last):
    File &quot;&lt;input&gt;&quot;, line 25, in &lt;module&gt;
      sub_tokens = tokenizer.tokenize(token)
  NameError: name 'tokenizer' is not defined
  </code></pre>
  <p>The error indicates that there are no <code>tokenizers</code> but it does not allow us to pass a <code>tokenizer</code>. Though it does work if I add a tokenizer while I am inside the function in debug mode. So how exactly do I use the <code>squad_convert_example_to_features</code> function?</p>