title: ActiveMQ Unexpected failure from FailoverTransport
body: <p>I have a (fairly) complicated Java app that uses Camel (2.11.0) and ActiveMQ (5.8.0) to send messages between a Camel route and various ActiveMQ queues. The Camel-ActiveMQ endpoint that I am using defines a failover like so:</p>
  
  <pre><code>failover:(tcp://myBroker01.example.com:61616)?jms.prefetchPolicy.queuePrefetch=1&amp;randomize=false&amp;maxReconnectDelay=250
  </code></pre>
  
  <p>You'll notice I only have 1 broker defined; for now I only have 1 broker (and so that kind of cancels out the usefulness of the failover transport), but down the road I may have several brokers as part of this failover URI.</p>
  
  <p>When I run my app, I see the following error in my app logs:</p>
  
  <pre><code>org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: &lt;com.me.myapp.model.WidgetVO&gt;
      ...middle portion of stack trace omitted for brevity; probably irrelevant
  Caused by: javax.jms.JMSException: Unexpected failure.
      at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
      at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1409)
      at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1496)
      at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:325)
      at org.apache.activemq.pool.ConnectionPool$2.makeObject(ConnectionPool.java:105)
      at org.apache.activemq.pool.ConnectionPool$2.makeObject(ConnectionPool.java:90)
      at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
      at org.apache.activemq.pool.ConnectionPool.createSession(ConnectionPool.java:142)
      at org.apache.activemq.pool.PooledConnection.createSession(PooledConnection.java:174)
      at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
      at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:457)
      ... 76 more
  Caused by: java.io.IOException: Unexpected failure.
      at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:620)
      at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
      at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
      at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
      at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1380)
      ... 85 more
  </code></pre>
  
  <p>So it looks like Camel is correctly reading my failover transport configuration and is attempting to asynchronously send (<code>oneway(...)</code>) a serialized <code>WidgetVO</code> message to my queue. I searched <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.activemq/activemq-client/5.8.0/org/apache/activemq/transport/failover/FailoverTransport.java" rel="nofollow">GrepCode for line 620 of <code>FailoverTransport</code></a> and here's the code block that the exception is being thrown from:</p>
  
  <pre><code>if (transport == null) {
      // Previous loop may have exited due to use being
      // disposed.
      if (disposed) {
          error = new IOException("Transport disposed.");
      } else if (connectionFailure != null) {
          error = connectionFailure;
      } else if (timedout == true) {
          error = new IOException("Failover timeout of " + timeout + " ms reached.");
      } else {
          error = new IOException("Unexpected failure.");
      }
  
      break;
  }
  </code></pre>
  
  <p>So the <code>Transport</code> is NULL, however the message has not been marked for disposal, nor is there a connection failure, nor has our connection to the broker timed out.</p>
  
  <p>Does any of this mean anything to anyone? Can anyone give me some pointers as to how to begin debugging this? I'm happy to post more code if it's needed.</p>
  
  <p><strong>Update</strong>: Here is another WARN message that I see in my app log <em>prior</em> to the exception being thrown:</p>
  
  <pre><code>07:53:00,306 WARN  [DefaultJmsMessageListenerContainer] Rejecting received message because of
  the listener container having been stopped in the meantime: ActiveMQTextMessage {commandId
  = 7, responseRequired = true, messageId = ID:&lt;myserver.example.com&gt;, originalDestination = null,
  originalTransactionId = null, producerId = ID:&lt;myserver.example.com&gt;, destination = queue://myQueue,
  transactionId = null, expiration = 0, timestamp = 1392295980294, arrival = 0,
  brokerInTime = 1392295980297, brokerOutTime = 1392295980298, correlationId = null, replyTo = null,
  persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0,
  targetConsumerId = null, compressed = false, userID = null, content = null,
  marshalledProperties = org.apache.activemq.util.ByteSequence@2d0b20f5, dataStructure = null,
  redeliveryCounter = 0, size = 0, properties = {breadcrumbId=ID-&lt;myserver.example.com&gt;},
  readOnlyProperties = true, readOnlyBody = true, droppable = false, text = &lt;com.me.myapp.WidgetVO&gt;
   ...WidgetVO&gt;}
  </code></pre>
  
  <p>I had to obviously change all the server names to <code>&lt;myserver.example.com&gt;</code> for security purposes. I thought it was interesting that the <code>transactionId</code> is NULL...</p>