title: [KAFKA][GRAYLOG] Direct Buffer OOM
body: <p>I have a simple setup with several producers writing to one kafka with multiple topics. My one and only consumer is Graylog instance writing to ES. After a week or two of activity kafka started to throw:</p>
  
  <pre><code>[2017-10-03 13:17:56,018] ERROR Processor got uncaught exception. (kafka.network.Processor)
  java.lang.OutOfMemoryError: Direct buffer memory
          at java.nio.Bits.reserveMemory(Bits.java:694)
          at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)
          at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
          at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
          at sun.nio.ch.IOUtil.read(IOUtil.java:195)
          at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
          at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
          at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)
          at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
          at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
          at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
          at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
          at kafka.network.Processor.run(SocketServer.scala:413)
          at java.lang.Thread.run(Thread.java:748)
  [2017-10-03 13:17:56,530] ERROR Processor got uncaught exception. (kafka.network.Processor)
  java.lang.OutOfMemoryError: Direct buffer memory
          at java.nio.Bits.reserveMemory(Bits.java:694)
          at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)
          at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
          at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
          at sun.nio.ch.IOUtil.read(IOUtil.java:195)
          at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
          at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
          at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)
          at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
          at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
          at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
          at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
          at kafka.network.Processor.run(SocketServer.scala:413)
          at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>Everything seems to work fine so far but I expect this kafka will go down soom. Is there anything I can do with this?</p>
  
  <p>Graylog machine: 10 GB RAM, 6 CPU 2,7 MHz 
  server JVM:</p>
  
  <pre><code>\_ /usr/bin/java -Xms5g -Xmx5g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -jar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm /usr/share/graylog-server/graylog.jar server -f /etc/graylog/server/server.conf -np
  </code></pre>
  
  <p>kafka zookeeper:</p>
  
  <pre><code>java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/kafka/bin/../logs/zookeeper-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/kafka/bin/../logs -Dlog4j.configuration=file:./../config/log4j.properties -cp :/data/kafka/bin/../libs/* org.apache.zookeeper.server.quorum.QuorumPeerMain /data/kafka/config/zookeeper.properties
  </code></pre>
  
  <p>kafka:</p>
  
  <pre><code>java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/kafka/bin/../logs -Dlog4j.configuration=file:./../config/log4j.properties -cp :/data/kafka/bin/../libs/* kafka.Kafka /data/kafka/config/server.properties
  </code></pre>
  
  <p>Memory stats on Kafka/Graylog machine:</p>
  
  <pre><code>              total        used        free      shared  buff/cache   available
  Mem:           9831        7337         133           6        2360        2130
  Swap:          2047        2047           0
  </code></pre>
  
  <p>3xElasticsearch: 10GB RAM, 8 CPU 2,7 MHz</p>
  
  <pre><code>/bin/java -Xms5g -Xmx5g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.5.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
  </code></pre>