title: Unable to see Spark UI when I submit in yarn-cluster of Hadoop-2.6
body: <p>I'm using Apache Spark 1.6 with Cluster of 8 nodes remotely. I'm submitting job using spark-submit like below on master node:</p>
  
  <pre><code>hastimal@hadoop-8:/usr/local/spark$ ./bin/spark-submit --class  umkc.graph.SparkRdfCcCount  --master yarn-cluster  --num-executors 7 --executor-memory 52g --executor-cores 7 --driver-memory 52g --conf spark.default.parallelism=49 --conf spark.driver.maxResultSize=4g  --conf spark.yarn.executor.memoryOverhead=4608 --conf spark.yarn.driver.memoryOverhead=4608 --conf spark.akka.frameSize=1200 --conf spark.network.timeout=300  --conf spark.io.compression.codec=lz4 --conf spark.rdd.compress=true --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://128.110.152.54:9000/SparkHistory --conf spark.broadcast.compress=true --conf spark.shuffle.spill.compress=true --conf spark.shuffle.compress=true --conf spark.shuffle.manager=sort /users/hastimal/SparkProcessing.jar /inputRDF/data-793-805.nt
  </code></pre>
  
  <p>Everything is fine. I'm getting output without any error but when I go to see Spark UI it doesn't show. In my Spark Scala code I have written like this:</p>
  
  <pre><code>val conf = new SparkConf().setAppName("Spark Processing").set("spark.ui.port","4041")
  </code></pre>
  
  <p>After following couple of things including <a href="https://stackoverflow.com/questions/31233830/apache-spark-setting-spark-eventlog-enabled-and-spark-eventlog-dir-at-submit-or">this</a> and <a href="https://stackoverflow.com/questions/22676593/why-does-hadoop-fs-mkdir-fail-with-permission-denied">this</a> I resolved my issues related to permission and writing in HDFS. When I run <code>spark-submit</code> and I see logs in Yarn it shows like this:</p>
  
  <pre><code>16/04/25 16:34:23 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
  16/04/25 16:34:23 INFO util.Utils: Successfully started service 'SparkUI' on port 4041.
  16/04/25 16:34:23 INFO ui.SparkUI: Started SparkUI at http://128.110.152.131:4041
  16/04/25 16:34:23 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
  16/04/25 16:34:24 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41216.
  16/04/25 16:34:24 INFO netty.NettyBlockTransferService: Server created on 41216
  16/04/25 16:34:24 INFO storage.BlockManagerMaster: Trying to register BlockManager
  16/04/25 16:34:24 INFO storage.BlockManagerMasterEndpoint: Registering block manager 128.110.152.131:41216 with 34.4 GB RAM, BlockManagerId(driver, 128.110.152.131, 41216)
  16/04/25 16:34:24 INFO storage.BlockManagerMaster: Registered BlockManager
  </code></pre>
  
  <p>Which means Spark UI has been started on <code>http://128.110.152.131:4041</code>which is one of the data node again and when I go to that URL it shows refuse error like below:</p>
  
  <p><a href="https://i.stack.imgur.com/u3y4q.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/u3y4q.jpg" alt="This is the refuse error I&#39;m getting"></a></p>
  
  <p>FYI: All used ports and opened in all machines. Please help me. I want to see DAG of my Spark Job. I'm able to see all yarn applications through Yarn UI. I can see Application UI like below using port 8088:
  <a href="https://i.stack.imgur.com/c5lLp.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/c5lLp.jpg" alt="Application UI"></a>. I want to see Spark UI with DAG like we see in Standalone or using IntelliJ IDE.</p>