title: How can I debug this buffered file queue?
body: <p>I've got a queue object which plays quite an important part in my project and I can't afford for there to be any bugs in it.</p>
  
  <p>The idea of it is to be like the builtin <code>Queue</code>, its base class, but it stores the data, or part of it at least in a file to preserve memory. I decided to keep some of it in memory as this should speed things up. I have put the code <a href="https://github.com/GP89/BufferedGzipQueue" rel="nofollow">here</a>, it might be easier to see it than for me to explain it</p>
  
  <p>It may seem an odd thing to want to do but I need to queue up a lot of work, which queues up much faster than I can get through it, and it will use far too much memory if I used a standard <code>Queue</code>. I can't just put a maxsize on the <code>Queue</code> and block the worker putting things in the queue as I want to know the total amount of data to process as soon as possible. I also can't work out a total first, but not queue it and then go back through putting in the <code>Queue</code> as the total will be different each time I go through the data, and at the end the totals wont match.</p>
  
  <p>My question is how can I test this thoroughly to be sure that there's no items being lost or more importantly any blocking on a getter when there's items still in the buffer or the file, or after complete has been called and the queue is empty.</p>
  
  <p>Some things seem very easy to test and set up unit tests for when you know what the output should be for some given input, but testing something like this I'm not really sure of an effective way to do it. Is it possible to test this kind of thing with unit tests?</p>
  
  <p>I've set up a test program that puts and gets items in at different, varying speeds of different number of items and it seems fine, but I've seen evidence of a getter blocking on <code>.get</code> with items still in the queue, so I'm lead to believe there is a problem.</p>
  
  <p>What's the best way I can test this thoroughly to find any remaining bugs or be near certain it's bug free?</p>
  
  <h2>EDIT</h2>
  
  <p>Can generate some test data similar to what I use with the following code, I only have the checksum for a file under certain conditions in the project and other times it is <code>None</code> so I just generate it some of the time in the code below to try and simulate that</p>
  
  <pre><code>import os
  import hashlib
  
  def hash(f_obj):
      md5 = hashlib.md5()
      while True:
          data = f_obj.read(8192)
          if not data:
              break
          md5.update(data)
      return md5.hexdigest()
  
  def produce(at_once,total_items):
      items=[]
      count=0
      for dir,folders,files in os.walk("/"):
          for f in files:
              try:
                  f_path= os.path.join(dir,f)
                  f_size= os.path.getsize(f_path)
                  f_mtime= os.path.getmtime(f_path)
                  with open(f_path) as file_obj:
                      f_hash= hash(file_obj) if f_size%2 else None
                  items.append((f_path,f_size,f_mtime,f_hash))
                  count+=1
              except Exception as err:
                  print "#####",err,"#####"
              if len(items) &gt;= at_once:
                  yield items
                  items=[]
          if count &gt;= total_items:
              break
      if items:
          yield items
  </code></pre>