title: Netty 4, Java 7 JVM SIGSEGV crash under load
body: <p>I have a binary protocol implemented with Netty that is being performance tested, and the JVM is crashing with the below report.  I do not know how to repeat the crash, but it does happen regularly and only under heavy load.  I have the following dependencies:</p>
  
  <ul>
  <li>java 7.0_51-b13</li>
  <li>netty 4.0.18_Final</li>
  <li>fedora 20</li>
  </ul>
  
  <p>It appears that the array copy is occurring in the nioEventLoopGroup thread.  The performance test I am running is sending a large number of small messages over ~50 TCP connections.  Where a large number is about 1 million 200 byte messages per connection. Each message has 2 response messages sent back.  </p>
  
  <p>This is what I am doing to create Netty:</p>
  
  <p>Bootstrap:</p>
  
  <pre><code>m_serverBootstrap.group(m_eventLoopGroup)
   .channel(NioServerSocketChannel.class)
   .localAddress(m_config.getSmppPort())
   .childAttr(InternalAttributeKeys.METRICS, m_metricRegistry)
   .childHandler(new CustomServerChannelInitializer());
  
  m_serverBindChannelFuture = m_serverBootstrap.bind().sync();
  </code></pre>
  
  <p>CustomerServerChannelInitializer</p>
  
  <pre><code>protected void initChannel(SocketChannel ch) throws Exception {
      log.info("initChannel(SocketChannel ch) {} {} ",ch,this);
      ch.pipeline()
      .addLast(new IpFilterHandler())
      .addLast(new ProtocolEncoder())
      .addLast(new LengthFieldBasedFrameDecoder(4 * 1024, 0, 4, -4, 0))
      .addLast(new ProtocolDecoder())
      .addLast(new WindowingHandler())
      .addLast(new SequenceNumberAssignmentHandler())
      .addLast("idleState", new IdleStateHandler(idleTime, idleTime, idleTime))
      .addLast("idleDisconnect", m_idleDisconnectHandler)
      .addLast("auth", m_authHandler)
      .addLast("catchall", new CatchallHandler(false));
      ch.config().setAllocator(PooledByteBufAllocator.DEFAULT);
      ch.config().setAutoRead(true);
      log.info("finished initChannel(SocketChannel ch) {} {} ",ch,this);
  }
  </code></pre>
  
  <p>After initial connection the pipeline is altered again in the authHandler</p>
  
  <pre><code>@Override
  protected void channelRead0(ChannelHandlerContext ctx, CustomMessage msg) throws Exception {
      ResponseMessage response = auth(msg,ctx);
      ctx.pipeline().replace("auth", "msghandler", new MessageHandler());
      ctx.pipeline().replace("idleState", "inactivityPeriod", new IdleStateHandler());
      ctx.pipeline().addAfter("msghandler", "responsehandler", new ResponseHandler());
      ctx.pipeline().addAfter("responsehandler", "heartbeat", new HeartbeatHandler());
      ctx.pipeline().addAfter("heartbeat", "disconnect", new DisconnectHandler());
      ctx.channel().closeFuture().addListener(new CleanupChannelFutureListener(ctx));
      ctx.writeAndFlush(response);
  }
  </code></pre>
  
  <p>jvm report.  I have a detailed report if it helps <a href="http://pastebin.com/RV0KqPMf" rel="nofollow">http://pastebin.com/RV0KqPMf</a> 
  If the JMX threads in the detailed report are bothering you, I can and have reproduced the issue without them.  </p>
  
  <pre><code>#
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGSEGV (0xb) at pc=0x00007ffa9eb18eaa, pid=1731, tid=140710808540928
  #
  # JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)
  # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode linux-amd64 compressed oops)
  # Problematic frame:
  # v  ~StubRoutines::jbyte_disjoint_arraycopy
  #
  # Core dump written. Default location: /home/user/dir/core or core.1731
  #
  # If you would like to submit a bug report, please visit:
  #   http://bugreport.sun.com/bugreport/crash.jsp
  #
  
  ---------------  T H R E A D  ---------------
  
  Current thread (0x00007ff9fc06f800):  JavaThread "nioEventLoopGroup-2-12" [_thread_in_Java, id=1912, stack(0x00007ff9c9b25000,0x00007ff9c9c26000)]
  
  siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00007ff987df7715
  </code></pre>
  
  <p>What is the best way to find out what is causing this SIGSEGV in the JVM?</p>