title: Use threading to spawn multiple processes to fetch the data in python 3
body: <p>I novice to threads in python. Trying to implement it first time. I want to use threading to spawn multiple processes. I have 
  written code below where I am trying to fetch data from the spark sql query. When I am trying to pull data without threading, this
  query is taking lot of time as it has to scan 70 gb of data. So, I want to do it through threadding to get the better performance.
  I have written code below I am not sure, it is right or not. </p>
  
  <pre><code>from pyspark import sql, SparkConf, SparkContext,SQLContext
  import threading
  
  #conf = SparkConf().setMaster('local[*]').setAppName('appname')
  #conf.set('spark.driver.allowMultipleContexts',True)
  
  def task(sc, i):
      dataframe_mysql = sqlContext.read.format("jdbc")\
                                  .option("url", "jdbc:mysql://path.rds.amazonaws.com:3306/dbname")\
                                  .option("driver", "com.mysql.jdbc.Driver")\
                                  .option("dbtable", "test")\
                                  .option("user", "user")\
                                  .option("password", "pwd")\
                                  .load()
      dataframe_mysql.registerTempTable("data")
      data_2015_00=sqlContext.sql("select * from data where date='2015-01-15 00:00:00' LIMIT 1000")
      print(sc.parallelize(data_2015_00).show())
  
  def run_multiple_jobs():
      conf = SparkConf().setMaster('local[*]').setAppName('appname')
    # Set scheduler to FAIR: http://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application
      conf.set('spark.scheduler.mode', 'FAIR')
      conf.set('spark.driver.allowMultipleContexts', True)
      sc = SparkContext.getOrCreate()
      sqlContext=SQLContext(sc)
  
      for i in range(4):
          t = threading.Thread(target=task, args=(sc, i))
          t.start()
  
  
  run_multiple_jobs()
  </code></pre>
  
  <p>This code is not working fine. And I am getting the below error:</p>
  
  <pre><code>py4j.protocol.Py4JError: An error occurred while calling o249.__getstate__. Trace:
  py4j.Py4JException: Method __getstate__([]) does not exist
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
      at py4j.Gateway.invoke(Gateway.java:272)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:748)
  
  
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "D:\Users\vk\AppData\Local\Continuum\Anaconda3\envs\py35\lib\threading.py", line 914, in _bootstrap_inner
      self.run()
    File "D:\Users\vk\AppData\Local\Continuum\Anaconda3\envs\py35\lib\threading.py", line 862, in run
      self._target(*self._args, **self._kwargs)
    File "&lt;ipython-input-6-8ec48a6ee660&gt;", line 8, in task
      print(sc.parallelize(data_2015_00).show())
    File "D:\Users\vk\spark\spark-2.1.0-bin-hadoop2.7\python\pyspark\context.py", line 470, in parallelize
      os.unlink(tempFile.name)
  PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Users\\vk\\AppData\\Local\\Temp\\spark-7741cdcd-f85d-4f9c-8a41-50bf6f293dbc\\pyspark-fb73a703-e797-44c7-b56a-0e08999aea3a\\tmp4w72p_jj'
  </code></pre>
  
  <p>I really appreciate any kind of help!</p>
  
  <p>Thanks</p>