title: Fastest way to read a fastq with scikit-bio
body: <p>I am trying to read a fastq formatted text file using <a href="http://scikit-bio.org/" rel="nofollow noreferrer">scikit-bio</a>.</p>
  
  <p>Given that it is a fairly large file, performing operations is quite slow.</p>
  
  <p>Ultimately, I am attempting to dereplicate the fastq file into a dictionary:</p>
  
  <pre><code>f = 'Undetermined_S0_L001_I1_001.fastq'
  seqs = skbio.io.read(f, format='fastq')
  
  seq_dic = {}
  for seq in seqs:
      seq = str(seq)
      if seq in seq_dic.keys():
          seq_dic[seq] +=1
      else:
          seq_dic[seq] = 1
  </code></pre>
  
  <p>Most of the time here is used during the reading of the file:</p>
  
  <pre><code>%%time
  f = 'Undetermined_S0_L001_I1_001.fastq'
  seqs = skbio.io.read(f, format='fastq')
  
  for seq in itertools.islice(seqs, 100000):
      seq
  
  CPU times: user 46.2 s, sys: 334 ms, total: 46.5 s
  Wall time: 47.8 s
  </code></pre>
  
  <p>My understanding is that not verifying the sequences would improve run time, however that does not appear to be the case:</p>
  
  <pre><code>%%time
  f = 'Undetermined_S0_L001_I1_001.fastq'
  seqs = skbio.io.read(f, format='fastq', verify=False, variant='illumina1.8')
  
  for seq in itertools.islice(seqs, 100000):
      seq
  
  CPU times: user 47 s, sys: 369 ms, total: 47.4 s
  Wall time: 48.9 s
  </code></pre>
  
  <p>So my question is, first why isn't <code>verify=False</code> improving run time and second is there a faster way using scikit-bio to read sequences?</p>