title: In Celery, how can I keep long-delayed tasks from blocking newer ones?
body: <p>I have two kinds of tasks. Task A is generated by celerybeat every hour. It runs immediately, and generates a thousand (or many thousand) instances of Task B, each of which has an ETA of one day in the future.</p>
  
  <p>Upon startup, an instance of Task A runs and generates a thousand Bs. And from then on, nothing happens. I should see another A running each hour, with another thousand Bs. But in fact I see nothing.</p>
  
  <p>At the freeze, rabbitmqctl shows 1000 messages, with 968 ready and 32 unacknowledged. An hour later there are 1001 messages, 969 ready and 32 unacknowledged. And so forth, every hour one new message classified as ready. Presumably what's happening is that the worker is prefetching 32 messages but can't act on them, because their ETA is still in the future. Meantime, newer tasks that should run right now can't be run.</p>
  
  <p>What is the right way to handle this? I'm guessing that I need multiple workers, and maybe multiple queues (but I'm not sure of the latter point). Is there a simpler way? I've tried fiddling with CELERYD_PREFETCH_MULTIPLIER and -Ofail (as discussed here: <a href="http://celery.readthedocs.org/en/latest/userguide/optimizing.html" rel="noreferrer">http://celery.readthedocs.org/en/latest/userguide/optimizing.html</a>) but can't get it to go. Is my question the same as this one: <a href="https://stackoverflow.com/questions/12950682/django-celery-celery-blocked-doing-io-tasks">[[Django Celery]] Celery blocked doing IO tasks</a> ?</p>
  
  <p>In any case: I can address this issue only because I know a lot about the nature of the tasks and their timing. Doesn't it seem a design flaw that enough tasks with future ETA can lock up the whole system? If I wait a few hours, then kill and restart the worker, it once again grabs the first 32 tasks and freezes up, even though at this point there are tasks in the queue that are ready to run right now. Shouldn't some component be smart enough to look at ETAs and ignore tasks that aren't runnable?</p>
  
  <p>ADDENDUM: I now think that the problem is a known bug when RabbitMQ 3.3 is used with Celery 3.1.0. More information here:
  <a href="https://groups.google.com/forum/#!searchin/celery-users/countdown|sort:date/celery-users/FiAAESOzezA/499OH-pylacJ" rel="noreferrer">https://groups.google.com/forum/#!searchin/celery-users/countdown|sort:date/celery-users/FiAAESOzezA/499OH-pylacJ</a></p>
  
  <p>After updating to Celery 3.1.1, things seem better. Task A runs hourly (well, it has for a couple of hours) and schedules its copies of Task B. Those seem to be filling up the worker: The number of unacknowledged messages continues to grow. I'll have to see if it can grow without bound.</p>