title: why the rpc timeout occur in cassandra
body: <p>I tried using cqlsh -3 version on my keyspace and used select query on a column family.
  It's return data in some causes and throws RPC time out in some other causes,I don't know the exact root cause.
  I used select query with single where condition</p>
  
  <pre><code>select * FROM date where date='2013-10-11 00:00:00+0000'; 
  </code></pre>
  
  <p>In this date column has secondary index with datatype text in UTF8 format</p>
  
  <pre><code>Request did not complete within rpc_timeout.
  </code></pre>
  
  <p>I checked with cassandra log.it throws</p>
  
  <pre><code>ERROR [ReadStage:117] 2013-12-03 19:21:46,813 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:117,5,main]
  at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
  at org.apache.cassandra.db.columniterator.SSTableNamesIterator.&lt;init&gt;(SSTableNamesIterator.java:60)
  at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
  at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
  at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:132)
  at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
  at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1390)
  at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213)
  at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1125)
  at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:191)
  at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:109)
  at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
  at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
  at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1499)
  at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:82)
  at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:548)
  at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1487)
  at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:44)
  at org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1055)
  at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1547)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:724)
  </code></pre>
  
  <p>why this happening?
  I am checking in my local with single seed?</p>
  
  <p>Update 1:</p>
  
  <p>my date table Structure</p>
  
  <pre><code>CREATE TABLE date (
    key text PRIMARY KEY,
    date text,
    date_id text,
    day bigint,
    day_name text
  ) WITH COMPACT STORAGE AND
    bloom_filter_fp_chance=0.010000 AND
    caching='KEYS_ONLY' AND
    comment='' AND
    dclocal_read_repair_chance=0.000000 AND
    gc_grace_seconds=864000 AND
    read_repair_chance=0.100000 AND
    replicate_on_write='true' AND
    populate_io_cache_on_flush='false' AND
    compaction={'min_sstable_size': '52428800', 'class': 'SizeTieredCompactionStrategy'} AND
    compression={'chunk_length_kb': '64', 'sstable_compression': 'SnappyCompressor'};
  </code></pre>
  
  <p>I checked with cassandra log,Its shows</p>
  
  <pre><code>ERROR [ReadStage:94] 2013-12-03 22:07:17,116 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:94,5,main]
  java.lang.AssertionError: DecoratedKey(-8665312888645846270,.......................&lt;!--some bytes of numbers-------&gt;
  /var/lib/cassandra/data/keyspace/columnfamily/keyspace-columnfamily-ic-1-Data.db
          at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
          at org.apache.cassandra.db.columniterator.SSTableNamesIterator.&lt;init&gt;(SSTableNamesIterator.java:60)
          at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
          at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
          at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:132)
          at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
          at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1390)
          at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213)
          at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1125)
          at org.apache.cassandra.db.Table.getRow(Table.java:347)
          at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
          at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1033)
          at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1547)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:724)
  </code></pre>
  
  <p>currently i am using cassandra-1.2.6.
  I check with this link,is this cassandra issue?
   <a href="https://issues.apache.org/jira/browse/CASSANDRA-4687" rel="nofollow">https://issues.apache.org/jira/browse/CASSANDRA-4687</a></p>