title: Java - Problem with multiple, concurrent runtime.exec() InputStreams
body: <p>I have no choice but to retrieve some external data by means of several <code>Runtime.exec()</code> calls to a VBScript.  I truly hate this implementation, as I lose my cross-platform flexibility, but I may eventually develop similar *nix scripts to at least mitigate the problem.  Before anyone asks, I <em>cannot</em> work around the need to call an external script to gather my data.  I will live with the problems that causes.</p>
  
  <p>The <code>exec()</code> processes are run in a custom class that extends <code>Runnable</code>.  It uses a <code>BufferedReader</code> to read in the data from <code>getInputStream()</code>.</p>
  
  <p><em><strong>Edit</strong>: more code added as requested, but I don't see how the extra code is relevant :) I hope it helps, because it took a while to format! Oh, and go easy on my code style if it's ugly, but constructive criticism is encouraged...</em></p>
  
  <pre><code>public class X extends JFrame implements Runnable {
  
     ...
     static final int THREADS_MAX = 4;
     ExecutorService  exec;
     ...
     public static void main(String[] args) {
        ...
        SwingUtilities.invokeLater(new X("X"));
     } // End main(String[])
  
     public X (String title) {
        ...
        exec = Executors.newFixedThreadPool(THREADS_MAX);
        ...
  
        // Create all needed instances of Y
        for (int i = 0; i &lt; objects.length; i++) {
           Y[i] = new Y(i);
        } // End for(i)
  
        // Initialization moved here for easy single-thread testing
        // Undesired, of course
        for (int i = 0; i &lt; objects.length; i++) {
           Y[i].initialize(parent);
        } // End for(i)
  
     } // End X
  
     class Y implements Runnable {
        // Define variables/arrays used to capture data here
        String computerName = "";
        ...
  
        public Y(int rowIndex) {
           row          = rowIndex;
           ...
           computerName = (String)JTable.getValueAt(row, 0);
           ...
           exec.execute(this);
        } // End Y(int)
  
        public void run() {
           // Initialize variables/arrays used to capture data here
           ...
  
           // Initialization should be done here for proper threading
           //initialize(parent);
        } // End run()
  
        public void initialize(Z obj) {
           runTime = Runtime.getRuntime();
           ...
  
           try {
              process = runTime.exec("cscript.exe query.vbs " + computerName);
              stdErr  = process.getErrorStream();
              stdIn   = process.getInputStream();
              isrErr  = new InputStreamReader(stdErr);
              isrIn   = new InputStreamReader(stdIn);
              brErr   = new BufferedReader(isrErr);
              brIn    = new BufferedReader(isrIn);
  
              while ((line = brIn.readLine()) != null) {
                 // Capture, parse, and store data here
                 ...
              } // End while
  
           } catch (IOException e) {
              System.out.println("Unable to run script");
           } catch (Exception e) {
              e.printStackTrace();
           } finally {
              try {
                 stdErr.close();
                 stdIn. close();
                 isrErr.close();
                 isrIn. close();
                 brErr. close();
                 brIn.  close();
              } catch (IOException e) {
                 System.out.println("Unable to close streams.");
              } // End try
           } // End try
        } // End initialize(Z)
        ...
     } // End class Y
  } // End class X
  </code></pre>
  
  <p>If I execute the commands individually, I gather the data as I expect.  However, if I execute the commands in the <code>run()</code> block of the class (meaning the calls are concurrent, as I am hoping for), it appears as if only one input stream is generated, which all <code>BufferedReaders</code> consume concurrently.</p>
  
  <p>To debug the issue, I output each consumed line on the console prefixed by which instance of my class was reading the input stream.  I expect something like the following, understanding that they may be out of order from instance-to-instance, but the line order of a single instance would be intact:</p>
  
  <pre><code>exec 0: Line1
  exec 1: Line1
  exec 2: Line1
  exec 0: Line2
  exec 1: Line2
  exec 2: Line2
  exec 0: Line3
  exec 1: Line3
  exec 2: Line3
  ...
  </code></pre>
  
  <p>What's strange is I get the expected number of instances of the very first line of the output (<code>Microsoft (R) Windows Script Host Version 5.7</code>), but after this line, only one process continues to produce data in the input stream, and all readers randomly-consume this one stream, such as this example:</p>
  
  <pre><code>exec 2: Microsoft (R) Windows Script Host Version 5.7
  exec 0: Microsoft (R) Windows Script Host Version 5.7
  exec 1: Microsoft (R) Windows Script Host Version 5.7
  exec 0: line2
  exec 1: line3
  exec 2: line4
  ...
  </code></pre>
  
  <p>To make matters worse, the readers stall and <code>readLine()</code> never returns null.  I read that this type of behavior might have something to do with the buffer size, but when I run only two concurrent threads, even with a short output, it still exhibits the same behavior.  Nothing is captured in <code>stdErr</code> to indicate there is a problem.</p>
  
  <p>To see if this was a limitation of the script host, I created a batch file that <code>START</code>s multiple instances of the script concurrently.  <strong>I should state this was run <em>outside</em> of Java, in a cmd shell, and launches several of its own shells.</strong>  However, each concurrent instance fully returned the expected results and behaved well.</p>
  
  <p><strong>Edit:</strong> As another troubleshooting idea, I decided to re-enable concurrency, but stagger my initialization method by inserting the following into my <code>Y.run()</code> block:</p>
  
  <pre><code>try {
     Thread.sleep((int)(Math.random() * 1200));
  } catch (InterruptedException e) {
     System.out.println("Can't sleep!");
  } // End try
  initialize(monitor);
  </code></pre>
  
  <p>into my code.  I begin to see multiple outputs for the first few lines, but it quickly reverts to multiple consumers consuming the same producer, and as soon as the first completed stream closes, the rest of the consumers fire exceptions.  The next consumer fires an <code>IOException: Read error</code>, and the rest fire <code>IOException: Stream closed</code>!</p>
  
  <p>According to maaartinus, it IS possible to run multiple, concurrent <code>InputStreams</code>, so now the question becomes what is causing the undesired behavior?  How can I independently grab their input streams?  I don't want to have to write to a temporary file just to process the data back in, if I can avoid it.</p>