title: Apache Spark to_json options parameter
body: <p>I either don't know what I'm looking for or the documentation is lacking. The latter seems to be the case, given this:</p>
  
  <p><a href="http://spark.apache.org/docs/2.2.2/api/java/org/apache/spark/sql/functions.html#to_json-org.apache.spark.sql.Column-java.util.Map-" rel="nofollow noreferrer">http://spark.apache.org/docs/2.2.2/api/java/org/apache/spark/sql/functions.html#to_json-org.apache.spark.sql.Column-java.util.Map-</a></p>
  
  <p>"options - options to control how the struct column is converted into a json string. accepts the same options and the json data source."</p>
  
  <p>Great! So, what are my options?</p>
  
  <p>I'm doing something like this:</p>
  
  <pre><code>Dataset&lt;Row&gt; formattedReader = reader
      .withColumn("id", lit(id))
      .withColumn("timestamp", lit(timestamp))
      .withColumn("data", to_json(struct("record_count")));
  </code></pre>
  
  <p>...and I get this result:</p>
  
  <pre><code>{
    "id": "ABC123",
    "timestamp": "2018-11-16 20:40:26.108",
    "data": "{\"record_count\": 989}"
  }
  </code></pre>
  
  <p>I'd like this (remove back-slashes and quotes from "data"):</p>
  
  <pre><code>{
    "id": "ABC123",
    "timestamp": "2018-11-16 20:40:26.108",
    "data": {"record_count": 989}
  }
  </code></pre>
  
  <p>Is this one of the options by chance? Is there a better guide out there for Spark? The most frustrating part about Spark hasn't been getting it to do what I want, it's been a lack of good information on what it <em>can</em> do.</p>