title: Reading data from S3 using pyspark throws java.lang.NumberFormatException: For input string: "100M"
body: <p>I am using the following code to read some json data from S3:</p>
  
  <pre><code>df = spark_sql_context.read.json("s3a://test_bucket/test.json")
  df.show()
  </code></pre>
  
  <p>The above code throws the following exception:</p>
  
  <pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling o64.json.
  : java.lang.NumberFormatException: For input string: "100M"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Long.parseLong(Long.java:589)
      at java.lang.Long.parseLong(Long.java:631)
      at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1538)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
      at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
      at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
      at scala.collection.immutable.List.flatMap(List.scala:355)
      at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
      at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
      at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:391)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>I have read several other SO posts on this topic (like <a href="https://stackoverflow.com/questions/55119337/read-files-from-s3-pyspark">this one</a> or <a href="https://stackoverflow.com/questions/48149929/hive-1-2-metastore-service-doesnt-start-after-configuring-it-to-s3-storage-inst">this</a>) and have done all they have mentioned but nothing seems to fix my issue.</p>
  
  <p>I am using <code>spark-2.4.4-bin-without-hadoop</code> and <code>hadoop-3.1.2</code>. As for the jar files, I've got:</p>
  
  <ul>
  <li>aws-java-sdk-bundle-1.11.199.jar </li>
  <li>hadoop-aws-3.0.0.jar</li>
  <li>hadoop-common-3.0.0.jar</li>
  </ul>
  
  <p>Also, using the following <code>spark-submit</code> command to run the code:</p>
  
  <pre><code>/opt/spark-2.4.4-bin-without-hadoop/bin/spark-submit 
  --conf spark.app.name=read_json --master yarn --deploy-mode client --num-executors 2 
  --executor-cores 2 --executor-memory 2G --driver-cores 2 --driver-memory 1G 
  --jars /home/my_project/jars/aws-java-sdk-bundle-1.11.199.jar,
  /home/my_project/jars/hadoop-aws-3.0.0.jar,/home/my_project/jars/hadoop-common-3.0.0.jar 
  --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.rpc.askTimeout=600s" /home/my_project/read_json.py
  </code></pre>
  
  <p>Anything I might be missing here?</p>