title: Java REST API call fails with exception "Async operation timed out" when payload exceeds certain size
body: <p>I am setting up an automated Cron job on a bunch of Linux servers which requires invoking some PUT/POST APIs provided by an IBM WebSphere Application Server(8.5.5). I wrote my own REST client in Java, but strangely sometimes the API execution hangs with below exception. </p>
  
  <pre><code>com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[JAX-RS Servlet]: java.net.SocketTimeoutException: Async operation timed out
      at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:191)
      at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
      at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.fillABuffer(HttpServiceContextImpl.java:4212)
      at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readSingleBlock(HttpServiceContextImpl.java:3440)
      at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readBodyBuffer(HttpServiceContextImpl.java:3546)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.getRequestBodyBuffer(HttpInboundServiceContextImpl.java:1822)
      at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.bufferIsGood(WCCByteBufferInputStream.java:371)
      at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.read(WCCByteBufferInputStream.java:265)
      at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java:322)
      at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
      at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
      at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
      at java.io.InputStreamReader.read(InputStreamReader.java:188)
      at com.ibm.json.java.JSONArray.parse(JSONArray.java:150)
      at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:44)
      at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:25)
      at org.apache.wink.server.internal.registry.ServerInjectableFactory$EntityParam.getValue(ServerInjectableFactory.java:197)
      at org.apache.wink.common.internal.registry.InjectableFactory.instantiate(InjectableFactory.java:68)
      at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:44)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)
      at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.handlers.OptionsMethodWADLHandler.handleRequest(OptionsMethodWADLHandler.java:51)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
      at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
      at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
      at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:60)
      at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)
      at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)
      at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
      at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
      at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
      at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
      at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
      at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:909)
      at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
      at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
      at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
      at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
      at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
      at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
      at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
      at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
      at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
      at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
      at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
      at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
      at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
  Caused by: com.ibm.io.async.AsyncTimeoutException(Async operation timed out, [Timeout, rc=0])
      at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:359)
      at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
      at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
      at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:184)
      ... 78 more
  </code></pre>
  
  <p>This only occurred on some of the Linux servers, and it worked perfectly fine when I developed it in my Windows workstation. So it shouldn't be a problem with time out settings in the WAS server. I tried to debug using curl as below </p>
  
  <pre><code>curl -vX PUT http://&lt;was_server_ip&gt;:9081/RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen -d @test.json --header "Content-Type: application/json"
  </code></pre>
  
  <p>and found out when the payload exceeds 1306 bytes (see Content-Length in curl output below), the API starts hanging and gives above-mentioned exception. </p>
  
  <pre><code>&gt; PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen
  HTTP/1.1
  &gt; User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
  &gt; Host: 9.51.163.190:9081
  &gt; Accept: */*
  &gt; Content-Type: application/json
  &gt; Content-Length: 1307
  &gt; Expect: 100-continue
  &gt;
  &lt; HTTP/1.1 100 Continue
  &lt; Content-Length: 0
  &lt; Date: Wed, 24 Aug 2016 07:35:23 GMT
  &lt; HTTP/1.1 500 Internal Server Error
  &lt; Date: Wed, 24 Aug 2016 07:35:23 GMT
  &lt; X-Powered-By: Servlet/3.0
  &lt; Content-Type: text/html;charset=ISO-8859-1
  &lt; $WSEP:
  &lt; Content-Language: en-US
  &lt; Content-Length: 102
  &lt; Connection: Close
  &lt;
  Error 500: javax.servlet.ServletException: java.net.SocketTimeoutException: Async operation timed out
  * Closing connection #0
  </code></pre>
  
  <p>When the payload is less than 1307 bytes, the API gets executed properly. </p>
  
  <pre><code>&gt; PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen HTTP/1.1
  &gt; User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
  &gt; Host: 9.51.163.190:9081
  &gt; Accept: */*
  &gt; Content-Type: application/json
  &gt; Content-Length: 1306
  &gt; Expect: 100-continue
  &gt;
  &lt; HTTP/1.1 100 Continue
  &lt; Content-Length: 0
  &lt; Date: Wed, 24 Aug 2016 07:37:51 GMT
  &lt; HTTP/1.1 200 OK
  &lt; Date: Wed, 24 Aug 2016 07:37:51 GMT
  &lt; X-Powered-By: Servlet/3.0
  &lt; Content-Type: application/json
  &lt; Access-Control-Allow-Headers: application/json
  &lt; Access-Control-Allow-Methods: GET, POST, PUT, DELETE
  &lt; Access-Control-Allow-Origin: *
  &lt; Content-Language: en-US
  &lt; Transfer-Encoding: chunked
  &lt;
  * Connection #0 to host 9.51.163.190 left intact
  * Closing connection #0
  {"Identifier":"syncTestSuites completed","Result":{"Newly Added Test Suites":[]}}
  </code></pre>
  
  <p>I need to check with their administrators on what settings might cause this on those problematic Linux machines. But also I am posting it here to see if anyone could give some thoughtful insight on the issue. </p>