title: Understanding Jetty's "Closed while Pending/Unready" warning
body: <p>We have an asynchronous servlet which produces the following warning log from Jetty:</p>
  
  <pre><code>java.io.IOException: Closed while Pending/Unready
  </code></pre>
  
  <p>After enabling debug logs I got the following stacktrace:</p>
  
  <pre><code>WARN [jetty-25948] (HttpOutput.java:278)   - java.io.IOException: Closed while Pending/Unready
  DEBUG [jetty-25948] (HttpOutput.java:279)   - 
  java.io.IOException: Closed while Pending/Unready
          at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:277) ~[jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.Response.closeOutput(Response.java:1044) [jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:488) [jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:293) [jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) [jetty-util.jar:9.4.8.v20171121]
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) [jetty-util.jar:9.4.8.v20171121]
          at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
  </code></pre>
  
  <p>It did not help too much.</p>
  
  <p>The warning comes only after Jetty calls the <code>onTimeout()</code> method of our <code>AsyncListener</code>. QA sometimes could reproduce it by using <code>kill -9</code> on the client side application.</p>
  
  <p>How can I reproduce this warning with a sample servlet-client code? I would like to understand this issue in a simpler environment than our production code to be able to fix the production one afterwards. How should a sample servlet behave? Is it possible to reproduce that with an Apache Commons HttpClient client side in the same JUnit test? (That would be great for writing an integration test without complicated network hacking, like <code>kill -9</code>.)</p>
  
  <p>I have tried a few things to implement a sample async servlet and client without success. I don't think that attaching this code would help too much but I can do that if anyone interested.</p>
  
  <p>Jetty version: 9.4.8.v20171121</p>
  
  <p><strong>update (2018-06-27):</strong></p>
  
  <p>Reflecting to @Joakim Erdfelt's helpful answer, I have not found any <code>close()</code> call in our code, but found a suspicious missing synchronization. Here is the base of our async poll servlet:</p>
  
  <pre class="lang-java prettyprint-override"><code>public class QueuePollServlet extends HttpServlet {
  
      public QueuePollServlet() {
      }
  
      @Override
      protected void doPost(final HttpServletRequest req, final HttpServletResponse resp)
              throws ServletException, IOException {
          resp.setContentType(MediaType.OCTET_STREAM.type());
          resp.setStatus(HttpServletResponse.SC_OK);
          resp.flushBuffer();
          final AsyncContext async = req.startAsync();
          async.setTimeout(30_000);
          final ServletOutputStream output = resp.getOutputStream();
          final QueueWriteListener writeListener = new QueueWriteListener(async, output);
          async.addListener(writeListener);
          output.setWriteListener(writeListener);
      }
  
      private static class QueueWriteListener implements AsyncListener, WriteListener {
  
          private final AsyncContext asyncContext;
          private final ServletOutputStream output;
  
          public QueueWriteListener(final AsyncContext asyncContext, final ServletOutputStream output) {
              this.asyncContext = checkNotNull(asyncContext, "asyncContext cannot be null");
              this.output = checkNotNull(output, "output cannot be null");
          }
  
          @Override
          public void onWritePossible() throws IOException {
              writeImpl();
          }
  
          private synchronized void writeImpl() throws IOException {
              while (output.isReady()) {
                  final byte[] message = getNextMessage();
                  if (message == null) {
                      output.flush();
                      return;
                  }
                  output.write(message);
              }
          }
  
          private void completeImpl() {
              asyncContext.complete();
          }
  
          public void dataArrived() {
              try {
                  writeImpl();
              } catch (IOException e) {
                  ...
              }
          }
  
          public void noMoreBuffers() {
              completeImpl();
          }
  
          @Override
          public void onTimeout(final AsyncEvent event) throws IOException {
              completeImpl();
          }
  
          @Override
          public void onError(final Throwable t) {
              logger.error("Writer.onError", t);
              completeImpl();
          }
  
  
          ...
      }
  }
  </code></pre>
  
  <p>A probable race condition:</p>
  
  <ol>
  <li>DataFeederThread: calls <code>dataArrived()</code> -> <code>writeImpl()</code>, then it gets that <code>output.isReady()</code> is <code>true</code>.</li>
  <li>Jetty calls <code>onTimeout()</code> which completes the context.</li>
  <li>DataFeederThread: calls <code>output.write()</code> in the while loop but found the completed context.</li>
  </ol>
  
  <p>Could this scenario cause the <code>Closed while Pending/Unready</code> warning or is it another issue?
  I am right that making <code>completeImpl()</code> <code>synchronized</code> solves the problem or there is something else to care about?</p>
  
  <p><strong>update (2018-06-28):</strong></p>
  
  <p>We also have a similar <code>onError</code> implementation in <code>QueueWriteListener</code> as the following snippet:</p>
  
  <pre class="lang-java prettyprint-override"><code>@Override
  public void onError(final Throwable t) {
      logger.error("Writer.onError", t);
      completeImpl();
  }
  </code></pre>
  
  <p>Anyway, there is no <code>onError</code> error log around the <code>Closed while Pending/Unready</code> log message (looking at a two hour timeframe for each), just EOFs like the following ones from our <code>DataFeederThread</code>:</p>
  
  <pre class="lang-none prettyprint-override"><code>DEBUG [DataFeederThread] (HttpOutput.java:224) - 
  org.eclipse.jetty.io.EofException: null
          at org.eclipse.jetty.server.HttpConnection$SendCallback.reset(HttpConnection.java:704) ~[jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpConnection$SendCallback.access$300(HttpConnection.java:668) ~[jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:526) ~[jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:778) ~[jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:834) ~[jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:234) [jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:218) [jetty-server.jar:9.4.8.v20171121]
          at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:392) [jetty-server.jar:9.4.8.v20171121]
          at com.example.QueuePollServlet$QueueWriteListener.writeImpl()
          at com.example.QueuePollServlet$QueueWriteListener.dataArrived()
  
  
  DEBUG [DataFeederThread] (QueuePollServlet.java:217) - messageArrived exception
  org.eclipse.jetty.io.EofException: Closed
          at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:476) ~[jetty-server.jar:9.4.8.v20171121]
          at com.example.QueuePollServlet$QueueWriteListener.writeImpl()
          at com.example.QueuePollServlet$QueueWriteListener.dataArrived()
  </code></pre>