title: Python - Multiprocessing and Threads for IO
body: <p>Hello I am trying to come up with an example of taking non async code that uses threads and convert it to something that uses both.</p>
  
  <p>My goal:
  Spawn off 4 Processes, and and with each process spawn off 10 threads at the same time.</p>
  
  <pre><code>import requests
  import multiprocessing
  from concurrent import futures
  
  
  def poll_data_1(data):
      response = requests.get('https://breadcrumbscollector.tech/feed/')
      print(f'Got data of length: {len(response.content)} in just {response.elapsed}')
  
  
  def thread_set(data):
      max_workers = 10
      concurrent = futures.ThreadPoolExecutor(max_workers)
      with concurrent as ex:
          ex.map(poll_data_1, data)
  
  
  data =range(40)
  data1 =[]
  for l in data:
      data1.append([l])
  
  # Mutliprocessing
  with multiprocessing.Pool(processes=4, maxtasksperchild=1) as pool:
      pool.imap_unordered(thread_set, data1)     
      pool.close()
      pool.join()
  </code></pre>
  
  <p>So this code "Works" but it looks like it only opens 1 process at a time. So the 10 threads will run, than 10 more.  My goal here would be to run all 40 threads at once.</p>
  
  <p>The reason I am trying to do this is my real application is trying to do 8,000-14,000 IO bound requests.  So threading is not scaling that high.  If I can say have my real server open process=to CPU, and each process spawn 1000 threads I think it would work better.</p>
  
  <p>Or Im super wrong...  Thanks!</p>