title: Loading a pickle in subprocess
body: <p>So I am trying to pickle a large dictionary that includes:
   numpy arrays,
   other dictionaries,
   filenames,
   and simple variables.</p>
  
  <p>But when I boot up a subprocess that will load the data, I'm getting an 'insecure string pickle' error.</p>
  
  <p>If I kill the main process before it calls the subprocess, and I run it manually through command line, it works fine... </p>
  
  <p>The problem may lie with the PIPE-ing with stdout and stderr.</p>
  
  <p>I've tried playing with some protocols for the pickle dump, but to no avail.</p>
  
  <p>Here's a sample from my code:</p>
  
  <pre><code>    dataout = open('Dataout.pickle','wb')
      dictout = { dict of all the values }
      pickle.dump(dictout, dataout)
      for i in xrange(int(self.runs)):
          self.names.append("%s_%i"%(self.name,i))
          if self.rerun == 0:
              self.chains.append(subprocess.Popen('python BaSIC_Model.py %s'%self.names[i], shell=False,stdout=subprocess.PIPE, stderr=subprocess.STDOUT,universal_newlines=True))#bufsize=0,
  
          else:
              self.chains.append(subprocess.Popen('python BaSIC_Model.py %s'%self.names[i], shell=False,stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True))#bufsize=0,
  
      if self.Gui == 1:
          self.cp = Cust_Progress.ProgRun("Running Chains", [0]*int(self.runs),self.itter)
          self.cp.Play()
          while True:
              if not any([True if chain.poll() == None else False for chain in self.chains]):
                  print 'broke'
                  texts = [chain.stdout.read() for chain in self.chains]
                  print texts[0]
                  break
              texts = [chain.stdout.readline() for chain in self.chains]
              print texts[0]
              if texts == None or None in texts:
                  texts = [0]*len(self.chains)
              self.cp.UpdateIters(texts)
              wx.Yield()
              time.sleep(0.05)
          self.cp.Done()
          self.cp.UpdateIters(['Generating Plots']*int(self.runs))
  </code></pre>
  
  <p>And from the subprocess, it fails on:</p>
  
  <pre><code>    inFile = open('Dataout.pickle','rb')
      inputData = pickle.load(inFile)
  </code></pre>
  
  <p>with the insecure string pickle error.
  also, my self.cp from the code is a custom progress that I have, which will generate the progress based off the stdout from each of my subprocess'. I have a working version with Threaded classes, but it is dreadfully slow, and a subprocessed version would give me at least 5x (to 9x) the speed if I am running 5 subprocesses at the same time.</p>
  
  <p>Has anyone else had a problem loading a pickle file in a subprocess before? 
  I've had programs that subprocessed and could load a pickle, but never with a PIPE to stdout.</p>
  
  <p>Thanks!</p>