title: pyspark.sql.utils.AnalysisException: Failed to find data source: kafka
body: <p>I am trying to read a stream from kafka using pyspark. I am using <strong>spark version 3.0.0-preview2</strong> and  <strong>spark-streaming-kafka-0-10_2.12</strong> 
  Before this I just stat zookeeper, kafka and create a new topic:</p>
  
  <pre><code>/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties 
  /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
  /usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic data_wm
  </code></pre>
  
  <p>This is my code: </p>
  
  <pre><code>import pandas as pd
  import os
  import findspark
  findspark.init("/usr/local/spark")
  from pyspark.sql import SparkSession
  
  spark = SparkSession.builder.appName("TestApp").getOrCreate()
  df = spark \
    .readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "data_wm") \
    .load() 
  value = df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") 
  </code></pre>
  
  <p>This how I run my script:</p>
  
  <blockquote>
    <p>sudo --preserve-env=pyspark /usr/local/spark/bin/pyspark --packages
    org.apache.spark:spark-streaming-kafka-0-10_2.12:3.0.0-preview</p>
  </blockquote>
  
  <p>As result for this command I have this :</p>
  
  <pre><code>: resolving dependencies :: org.apache.spark#spark-submit-parent-0d7b2a8d-a860-4766-a4c7-141a902d8365;1.0
          confs: [default]
          found org.apache.spark#spark-streaming-kafka-0-10_2.12;3.0.0-preview in central
          found org.apache.spark#spark-token-provider-kafka-0-10_2.12;3.0.0-preview in central
          found org.apache.kafka#kafka-clients;2.3.1 in central
          found com.github.luben#zstd-jni;1.4.3-1 in central
          found org.lz4#lz4-java;1.6.0 in central
          found org.xerial.snappy#snappy-java;1.1.7.3 in central
          found org.slf4j#slf4j-api;1.7.16 in central
          found org.spark-project.spark#unused;1.0.0 in central :: resolution report :: resolve 380ms :: artifacts dl 7ms
          :: modules in use:
          com.github.luben#zstd-jni;1.4.3-1 from central in [default]
          org.apache.kafka#kafka-clients;2.3.1 from central in [default]
          org.apache.spark#spark-streaming-kafka-0-10_2.12;3.0.0-preview from central in [default]
          org.apache.spark#spark-token-provider-kafka-0-10_2.12;3.0.0-preview from central in [default]
          org.lz4#lz4-java;1.6.0 from central in [default]
          org.slf4j#slf4j-api;1.7.16 from central in [default]
          org.spark-project.spark#unused;1.0.0 from central in [default]
          org.xerial.snappy#snappy-java;1.1.7.3 from central in [default]
  </code></pre>
  
  <p>But I have always this error:</p>
  
  <p>d> f = spark \ ...   .readStream \ ...   .format("kafka") \ ...  </p>
  
  <blockquote>
    <p>.option("kafka.bootstrap.servers", "localhost:9092") \ ...<br>
    .option("subscribe", "data_wm") \ ...   .load()  Traceback (most
    recent call last):   File "", line 5, in    File
    "/usr/local/spark/python/pyspark/sql/streaming.py", line 406, in load
        return self._df(self._jreader.load())   File "/usr/local/spark/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py",
    line 1286, in <strong>call</strong>   File
    "/usr/local/spark/python/pyspark/sql/utils.py", line 102, in deco
        <strong>raise converted pyspark.sql.utils.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the
    deployment section of "Structured Streaming + Kafka Integration</strong>
    Guide".;</p>
  </blockquote>
  
  <p>I don't know the cause of this error, please help</p>