title: Cassandra crashes with Out Of Memory within minutes after starting
body: <p>We have a Cassandra cluster with 3 nodes and replication factor 3 on AWS using EC2Snitch.</p>
  <p>Instance type is c5.2xlarge (8 core and 16GB RAM).</p>
  <p>The cluster had been working fine but suddenly since yesterday evening, the cassandra process on all the nodes started crashing. They are set to restart automatically but then they crash with Out of Memory Heap Space error in 1 or 2 or 3 minutes after start.</p>
  <p>Heap configs:</p>
  <pre><code>MAX_HEAP_SIZE=&quot;4G&quot;
  HEAP_NEWSIZE=&quot;800M&quot;
  </code></pre>
  <p>After this, we tried increasing the node size to r5.4x or 128 GB memory and assigned 64GB Heap but still the same thing happens, irrespective of all 3 nodes being started or only one node being started at a time. We could note that first garbage collection happens after some time and then consecutively within seconds, failing to free any further memory and eventually crashing.</p>
  <p>We are not sure what is being pulled to memory immediately after starting.</p>
  <p>Other parameters:</p>
  <ul>
  <li>Cassandra version : 2.2.13</li>
  <li>Database size is 250GB</li>
  <li><code>hinted_handoff_enabled: true</code></li>
  <li><code>commitlog_segment_size_in_mb: 64</code></li>
  <li><code>memtable_allocation_type: offheap_buffers</code></li>
  </ul>
  <p>Any help here, would be appreciated.</p>
  <p><strong>Edit:</strong>
  We found that there is particular table when queried, it causes the casssandra node to crash.</p>
  <pre><code>cqlsh:my_keyspace&gt; select count(*) from my_table ;
  ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message=&quot;Operation timed out - received only 0 responses.&quot; info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
  </code></pre>
  <p>So we think, it is related to the data being corrupt/huge in this particular table.
  Thanks.</p>