title: How to load Impala table directly to Spark using JDBC?
body: <p>I am trying to write a spark job with Python that would open a jdbc connection with Impala and load a VIEW directly from Impala into a Dataframe. This question is pretty close but in scala: <a href="https://stackoverflow.com/questions/26634853/calling-jdbc-to-impala-hive-from-within-a-spark-job-and-creating-a-table">Calling JDBC to impala/hive from within a spark job and creating a table</a></p>
  
  <p>How do I do this? There are plenty of examples for other datasources such as MySQL, PostgreSQL, etc. but I haven't seen one for Impala + Python + Kerberos. An example would be of great help. Thank you!</p>
  
  <p>Tried this with information from the web but it didn't work.</p>
  
  <h3>SPARK Notebook</h3>
  
  <pre><code>#!/bin/bash
  export PYSPARK_PYTHON=/home/anave/anaconda2/bin/python
  export HADOOP_CONF_DIR=/etc/hive/conf
  export PYSPARK_DRIVER_PYTHON=/home/anave/anaconda2/bin/ipython
  export PYSPARK_DRIVER_PYTHON_OPTS='notebook --ip=* --no-browser'
  
  # use Java8
  export JAVA_HOME=/usr/java/latest
  export PATH=$JAVA_HOME/bin:$PATH
  
  # JDBC Drivers for Impala
  export CLASSPATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30/*.jar:$CLASSPATH
  export JDBC_PATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30
  
  # --jars $SRCDIR/spark-csv-assembly-1.4.0-SNAPSHOT.jar \
  # --conf spark.sql.parquet.binaryAsString=true \
  # --conf spark.sql.hive.convertMetastoreParquet=false
  
  pyspark --master yarn-client \
          --driver-memory 4G \
          --executor-memory 2G \
          # --num-executors 10 \
          --jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar
          --driver-class-path $JDBC_PATH/*.jar
  </code></pre>
  
  <h3>Python Code</h3>
  
  <pre><code>properties = {
      "driver": "com.cloudera.impala.jdbc41.Driver",
      "AuthMech": "1",
  #     "KrbRealm": "EXAMPLE.COM",
  #     "KrbHostFQDN": "impala.example.com",
      "KrbServiceName": "impala"
  }
  
  # imp_env is the hostname of the db, works with other impala queries ran inside python
  url = "jdbc:impala:imp_env;auth=noSasl"
  
  db_df = sqlContext.read.jdbc(url=url, table='summary', properties=properties)
  </code></pre>
  
  <p>I received this error msg (<a href="http://pastebin.com/y7496a5i" rel="nofollow noreferrer" title="Full Error Log">Full Error Log</a>):<br>
  Py4JJavaError: An error occurred while calling o42.jdbc.
  : java.lang.ClassNotFoundException: com.cloudera.impala.jdbc41.Driver</p>