title: Multiprocessing: How to write separate log files for each instance while using pool.map?
body: <p>I want to create a class where each instance writes its own log file. This works fine when I use a function instead of a class (or when I don´t use multiprocessing):</p>
  
  <pre><code>import multiprocessing, logging
  
  def setup_logger(name_logfile, path_logfile):
          logger = logging.getLogger(name_logfile)
          formatter = logging.Formatter('%(asctime)s:   %(message)s', datefmt='%Y/%m/%d %H:%M:%S')
          fileHandler = logging.FileHandler(path_logfile, mode='w')
          fileHandler.setFormatter(formatter)
          streamHandler = logging.StreamHandler()
          streamHandler.setFormatter(formatter)
  
          logger.setLevel(logging.DEBUG)
          logger.addHandler(fileHandler)
          logger.addHandler(streamHandler)
          return logger
  
      def MyFunc(A):
          print A
          logger = setup_logger('Logfile%s' %A, '/dev/shm/Logfile%s.log' %A)
          logger.info('text to be written to logfile')
  
  
      pool = multiprocessing.Pool(2)
      pool.map(MyFunc,[1,2])
      pool.close()
      pool.join()
  </code></pre>
  
  <p>But when I use a class, I get pickling errors:</p>
  
  <pre><code>import multiprocessing, logging
  
  class MyClass(object):
      def __init__(self,A):
          print A
          self.logger = self.setup_logger('Logfile%s' %A, '/dev/shm/Logfile%s.log' %A)
          self.logger.info('text to be written to logfile')
  
      def setup_logger(self,name_logfile, path_logfile):
          logger = logging.getLogger(name_logfile)
          formatter = logging.Formatter('%(asctime)s:   %(message)s', datefmt='%Y/%m/%d %H:%M:%S')
          fileHandler = logging.FileHandler(path_logfile, mode='w')
          fileHandler.setFormatter(formatter)
          streamHandler = logging.StreamHandler()
          streamHandler.setFormatter(formatter)
  
          logger.setLevel(logging.DEBUG)
          logger.addHandler(fileHandler)
          logger.addHandler(streamHandler)
          return logger
  
  pool = multiprocessing.Pool(2)
  pool.map(MyClass,[1,2])
  pool.close()
  pool.join()
  </code></pre>
  
  <p>OUT:</p>
  
  <pre><code>1
  2
  2015/02/12 14:05:09:   text to be written to logfile
  2015/02/12 14:05:09:   text to be written to logfile
  Process PoolWorker-1:
  Traceback (most recent call last):
    File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      self.run()
    File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/lib64/python2.7/multiprocessing/pool.py", line 99, in worker
  Process PoolWorker-2:
      put((job, i, result))
    File "/usr/lib64/python2.7/multiprocessing/queues.py", line 392, in put
  Traceback (most recent call last):
    File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      return send(obj)
  PicklingError: Can't pickle &lt;type 'thread.lock'&gt;: attribute lookup thread.lock failed
      self.run()
    File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/lib64/python2.7/multiprocessing/pool.py", line 99, in worker
      put((job, i, result))
    File "/usr/lib64/python2.7/multiprocessing/queues.py", line 392, in put
      return send(obj)
  PicklingError: Can't pickle &lt;type 'thread.lock'&gt;: attribute lookup thread.lock failed
  </code></pre>
  
  <p>I can´t figure out what is the reason for this error since every logfile has its own output path. I need the logger as an attribute of the object, so how can I work around this pickling error?</p>