title: Ways to improve Cassandra read performance in my scenario
body: <p>We have recently started using Cassandra database in production. We have a <code>single cross colo cluster of 24 nodes</code> meaning <code>12 nodes in PHX</code> and <code>12 nodes in SLC colo</code>. We have a <code>replication factor of 4</code> which means <code>2 copies will be there in each datacenter</code>. </p>
  
  <p>Below is the way by which <code>keyspace</code> and <code>column families</code> have been created by our <code>Production DBA's</code>.</p>
  
  <blockquote>
    <p>create keyspace profile with placement_strategy =
    'org.apache.cassandra.locator.NetworkTopologyStrategy' and
    strategy_options = {slc:2,phx:2};</p>
  
  <pre><code>create column family PROFILE_USER
  with key_validation_class = 'UTF8Type'
  and comparator = 'UTF8Type'
  and default_validation_class = 'UTF8Type'
  and gc_grace = 86400;
  </code></pre>
  </blockquote>
  
  <p>We are running <code>Cassandra 1.2.2</code> and it has <code>org.apache.cassandra.dht.Murmur3Partitioner</code>, with <code>KeyCaching</code>, <code>SizeTieredCompactionStrategy</code> and <code>Virtual Nodes</code> enabled as well. </p>
  
  <p><strong>Machine Specifications for Cassandra production nodes-</strong></p>
  
  <pre><code>16 cores, 32 threads
  128GB RAM
  4 x 600GB SAS in Raid 10, 1.1TB usable
  2 x 10GbaseT NIC, one usable
  </code></pre>
  
  <p>Below is the result I am getting.</p>
  
  <pre><code>Read Latency(95th Percentile)      Number of Threads    Duration the program was running(in minutes)    Throughput(requests/seconds)    Total number of id's requested    Total number of columns requested
      9 milliseconds                         10                      30                                               1977                              3558701                        65815867
  </code></pre>
  
  <p>I am not sure what other things I should try it out with Cassandra to get much better <code>read performance</code>. I am assuming it is hitting the disk in my case. Should I try increasing the Replication Factor to some higher number? Any other suggestion?</p>
  
  <p>I believe reading the data from HDD is around 6-12ms as compared to SSD's? In my case it is hitting the disk everytime I guess and enabling key cache is not working fine here. I cannot enable RowCache becasue it’s more efficient to use OS page cache. Maintaining row cache in JVM is very expensive, thus row cache is recommended for smaller number of rows, like &lt;100K rows, only.</p>
  
  <p>Is there any way I can verify whether keycaching is working fine in my case or not?</p>
  
  <p>This is what I get when I do show schema for column family-</p>
  
  <pre><code>create column PROFILE
    with column_type = 'Standard'
    and comparator = 'UTF8Type'
    and default_validation_class = 'UTF8Type'
    and key_validation_class = 'UTF8Type'
    and read_repair_chance = 0.1
    and dclocal_read_repair_chance = 0.0
    and populate_io_cache_on_flush = false
    and gc_grace = 86400
    and min_compaction_threshold = 4
    and max_compaction_threshold = 32
    and replicate_on_write = true
    and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
    and caching = 'KEYS_ONLY'
    and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
  </code></pre>
  
  <p>Is there anything I should make a change to get good read performance?</p>