title: CXF, Camel and Spring thread blocking issue
body: <p>Issue is with Camel and CXF (endpoints and client) and multiple requests firing at once.</p>
  
  <p>Here's the exception we are getting:</p>
  
  <pre><code>java.net.SocketException: Unexpected end of file from server
      at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
      at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
      at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1543)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1513)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1318)
      at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
      at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
      at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
      at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:570)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:479)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)
      at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
      at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)
  </code></pre>
  
  <p>I have tried hitting the endpoint directly with SoapUI load tests and can not reproduce the problem. It only happens going through the spring jaxws:client bean.</p>
  
  <p>One request at a time from the client works fine and we never see the error, only when we try to do multiple.</p>
  
  <p>Also performed a thread dump while the threads are waiting, and all of the requests (5 of them) look like the following:</p>
  
  <pre><code>http-listener-1(1)" daemon prio=5 tid=0x00007fa2843e8800 nid=0x9d07 in Object.wait() [0x000000018a266000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on &lt;0x00000001708b27e8&gt; (a org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream)
      at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.getHttpResponse(AsyncHTTPConduit.java:562)
      - locked &lt;0x00000001708b27e8&gt; (a org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream)
      at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.getResponseCode(AsyncHTTPConduit.java:674)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1543)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1513)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1318)
      at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:383)
      at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
      at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
      at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
      - locked &lt;0x00000001704175f8&gt; (a org.apache.cxf.phase.PhaseInterceptorChain)
      at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:570)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:479)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)
      at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
      at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)
  </code></pre>
  
  <p>I also tried using the ASync client, while the stacktrace and thread dump looked different, the result was the same.</p>
  
  <p>This also seems to be happening only on our dev instances (Mac laptops) and we are running this inside Glassfish 4.0 with JDK 1.7.x (1.7u65 on my machine, slightly different version on others)</p>
  
  <p>One of the camel routes, although it seems to affect all of them (and they are setup the sameway)</p>
  
  <pre><code>from("cxf:/structure?serviceClass=" + StructureEndpoint.class.getName() + "&amp;loggingFeatureEnabled=true")
              .routeId("structure")
              .to("log:com.test.camel")
              .recipientList(simple("direct:structure:${header.operationName}"));
  </code></pre>
  
  <p>When this happens I never see the camel/cxf log output enabled by loggingFeatureEnabled, and so we don't see our .to("log:*") output either. It just never gets here.</p>
  
  <h3>Reproducable Test:</h3>
  
  <p>SoapUI, simple load test. Start/end threads at 5 
  Start test, get errors for first 5 requests. 
  The test goes through our frontend, which uses the cxf client to hit the backend camel/cxf services.</p>
  
  <p>Hitting camel/cxf services directly does not cause any issues.</p>
  
  <h3>Envirnoment Test Update</h3>
  
  <p>We have 2 other environments running on Linux, both on JDK 1.7.0u60, same version of glassfish that do not exhibit this behavior. All machines affected are OS X only. (will be testing a windows box shortly)</p>