title: Where does log4j writes the logs in cluster mode?
body: <p>Purpose - Store custom log from streaming app to HDFS or UNIX directory for streaming application<br></p>
  
  <p>I am running spark streaming program in cluster mode.But logs are not getting written to given log path. checked both HDFS and Local directory.By log4j debug property i can see files in action. Am i missing something? </p>
  
  <p>--files log4j_driver.properties <br> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j_driver.properties -Dlog4j.debug=true "<br>    --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j_driver.properties -Dlog4j.debug=true"  <br></p>
  
  <p><strong>Log4j-Property file <br></strong></p>
  
  <p>My Log4j properties file -</p>
  
  <p>log=/tmp/cc</p>
  
  <p>log4j.rootLogger=INFO,rolling<br>
  log4j.appender.rolling=org.apache.log4j.RollingFileAppender<br>
  log4j.appender.rolling.File=${log}/abc.log<br>
  log4j.appender.rolling.layout=org.apache.log4j.PatternLayout<br>
  log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n<br>
  log4j.appender.rolling.maxFileSize=2KB<br>
  log4j.appender.rolling.maxBackupIndex=10<br>
  log4j.appender.rolling.encoding=UTF-8<br>
  log4j.logger.org.apache.spark=INFO<br>
  log4j.appender.rolling.ImmediateFlush=true<br>
  log4j.appender.rolling.Threshold=debug<br>
  log4j.appender.rolling.Append=true<br>
  log4j.logger.org.eclipse.jetty=INFO<br></p>
  
  <p><strong>Cluster Driver Log</strong> <br></p>
  
  <p>log4j: Renaming file /tmp/cc/abc.log.2 to /tmp/cc/abc.log.3 <br>
  log4j: Renaming file /tmp/cc/abc.log.1 to /tmp/cc/abc.log.2 <br>
  log4j: Renaming file /tmp/cc/abc.log to /tmp/cc/abc.log.1<br>
  log4j: setFile called: /tmp/cc/abc.log, false<br>
  log4j: setFile ended<br>
  log4j: rolling over count=5141<br>
  log4j: maxBackupIndex=10<br>
  log4j: Renaming file /tmp/cc/abc.log.9 to /tmp/cc/abc.log.10<br>
  log4j: Renaming file /tmp/cc/abc.log.8 to /tmp/cc/abc.log.9<br>
  log4j: Renaming file /tmp/cc/abc.log.7 to /tmp/cc/abc.log.8<br>
  log4j: Renaming file /tmp/cc/abc.log.6 to /tmp/cc/abc.log.7<br></p>
  
  <p>I read- We can specify - ${spark.yarn.app.container.log.dir}/app.log in log4j but not sure what is the default path for this property or if we need to set manually then as well . When i was runninng this application in client mode - logs are perfectly getting logged to local directory.</p>