title: Why does resque use child processes for processing each job in a queue?
body: <p>We have been using Resque in most of our projects, and we have been happy with it.</p>
  
  <p>In a recent project, we were having a situation, where we are making a connection to a live streaming API from the twitter. Since, we have to maintain the connection, we were dumping each line from the streaming API to a resque queue, lest the connection is not lost. And we were, processing the queue afterwards.</p>
  
  <p>We had a situation where the insertion rate into the queue was of the order 30-40/second and the rate at which the queue is popped was only 3-5/second. And because of this, the queue was always increasing. When we checked for reasons for this, we found that resque had a parent process, and for each job of the queue, it forks a child process, and the child process will be processing the job. Our rails environment was quite heavy and the child process forking was taking time.</p>
  
  <p>So, we implemented another rake task of this sort, for the time being:</p>
  
  <pre><code>rake :process_queue =&gt; :environment do
    while true
      begin
        interaction = Resque.pop("process_twitter_resque")
        if interaction
          ProcessTwitterResque.perform(interaction)
        end
      rescue =&gt; e
        puts e.message
        puts e.backtrace.join("\n")
      end
    end
  end
  </code></pre>
  
  <p>and started the task like this:</p>
  
  <pre><code>nohup bundle exec rake process_queue --trace &gt;&gt; log/workers/process_queue/worker.log 2&gt;&amp;1 &amp;
  </code></pre>
  
  <p>This does not handle failed jobs and all.</p>
  
  <p>But, my question is why does Resque implement a child forked process to process the jobs from the queue. The jobs definitly does not need to be processed paralelly (since it is a queue and we expect it to process one after the other, sequentially and I beleive Resque also fork only 1 child process at a time). </p>
  
  <p>I am sure Resque has done it with some purpose in mind. What is the exact purpose behind this parent/child process architecture? </p>