title: Logging in spark structured streaming
body: <p>I am able to develop  a pipeline  which reads from kafka does some transformations and write the output to kafka sink as well as parque sink. I would like adding effective logging to log the  intermediate results of the transformation  like in a regular streaming application.</p>
  
  <p>One option I see is to log the <code>queryExecutionstreams</code> via  </p>
  
  <pre><code>df.queryExecution.analyzed.numberedTreeString 
  </code></pre>
  
  <p>or </p>
  
  <pre><code>logger.info("Query progress"+ query.lastProgress)
  logger.info("Query status"+ query.status)
  </code></pre>
  
  <p>But this doesn't seem to have a way to see the business specific messages on which the stream is running on. </p>
  
  <p>Is there a way how I can add more logging info like the data which it's processing?</p>