title: Execute spark-submit programmatically from java
body: <p>I am trying to execute it via:</p>
  
  <pre><code>Process process = Runtime.getRuntime().exec(spark_cmd);
  </code></pre>
  
  <p>with no luck. The command ran via shell starts my application which succeeds. Running it via exec start a process which dies shortly after and does nothing.
  When I try</p>
  
  <pre><code>process.waitFor();
  </code></pre>
  
  <p>it hangs and waits forever. Real magic begins when I try to read something from the process:</p>
  
  <pre><code> InputStreamReader isr = new InputStreamReader(process.getErrorStream());
   BufferedReader br = new BufferedReader(isr);
  </code></pre>
  
  <p>To do so I start a thread that reads from the stream in a while loop:</p>
  
  <pre><code>class ReadingThread extends Thread {
  
     BufferedReader reader;
     Wontekk(BufferedReader reader) {
        this.reader = reader;
  
     }
  
     @Override
     public void run() {
        String line;
        try {
           while ((line = reader.readLine()) != null) {
              System.out.println(line);
           }
        } catch (IOException e) {
           // TODO Auto-generated catch block
           e.printStackTrace();
        }
     }
  }
  </code></pre>
  
  <p>Application starts, does some stuff, and hangs. When I abort my application, spark application wakes up (??????????) and completes remaining work. Does anyone have reasonable explanation of what is happening?</p>
  
  <p>thanks</p>