title: ElasticSearch setup for a large cluster with heavy aggregations
body: <h1>Context and current state</h1>
  
  <p>We are migrating our cluster from Cassandra to a full ElasticSearch cluster. We are indexing documents at average of <strong>~250-300 docs per seconds</strong>. In ElasticSearch 1.2.0 it represents ~8Go per day.</p>
  
  <pre class="lang-javascript prettyprint-override"><code>{
   "generic":
      {
        "id": "twi471943355505459200",
        "type": "twitter",
        "title": "RT @YukBerhijabb: The Life is Choice - https://m.facebook.com/story.php?story_fbid=637864496306297&amp;id=100002482564531&amp;refid=17",
        "content": "RT @YukBerhijabb: The Life is Choice - https://m.facebook.com/story.php?story_fbid=637864496306297&amp;id=100002482564531&amp;refid=17",
        "source": "&lt;a href=\"https://twitter.com/download/android\" rel=\"nofollow\"&gt;Twitter for  Android&lt;/a&gt;",
        "geo": null,
        "link": "http://twitter.com/rosi_sifah/status/471943355505459200",
        "lang": "en",
        "created_at": 1401355038000,
        "author": {
          "username": "rosi_sifah",
          "name": "Rosifah",
          "id": 537798506,
          "avatar": "http://pbs.twimg.com/profile_images/458917673456238592/Im22zoIV_normal.jpeg",
          "link": "http://twitter.com/rosi_sifah"
        }
      },
   "twitter": {
     // a tweet JSON
   }
  }
  </code></pre>
  
  <p>Our users save requests in our SQL database and when they ask for their dashboard we would like to request our ES cluster with their query (retrieved from database) and <strong>do some aggregation on top of it</strong> using the new ES aggregation framework.</p>
  
  <p>Each dashboard is displayed with an explicit, user selected, date range so we always use </p>
  
  <pre class="lang-javascript prettyprint-override"><code>"range": {
   "generic.created_at": {
     "from": 1401000000000,
     "to": 1401029019706
    }
  }
  </code></pre>
  
  <p>along with the ES query.</p>
  
  <p>We specified <strong>_routing</strong> that way:</p>
  
  <pre class="lang-javascript prettyprint-override"><code>"_routing":{
   "required":true,
   "path":"generic.id"
  },
  </code></pre>
  
  <p>and the <strong>_id</strong> with:</p>
  
  <pre class="lang-javascript prettyprint-override"><code>"_id": {
    "index": "not_analyzed",
    "store": "false",
    "path": "generic.id"
  }
  </code></pre>
  
  <p>For approximately 5 days we've stored <strong>67 millions documents (about 40Go)</strong> inside one index. We've learn about the good practice of spliting the index by day. So now our indices are splitted by day ([index-name]-[YYYY-MM-DD]).</p>
  
  <p>Currently each index has 5 shards and 1 replica, we have a cluster composed of 3 machines each with 8 cores, 16Go of RAM and 8To of HDD. We plan to use another machine as a gateway (8 cores, 16Go of RAM, 1To of HDD).</p>
  
  <p>We've leaved ES configuration by default besides the cluster configuration.</p>
  
  <h1>Questions</h1>
  
  <ol>
  <li>For each document we want to index, we say explicitly what index to
  use. Currently we use the date of the day. Should we use the date of
  the document in order to prevent hot spot? Because currently it
  means that documents from various days (specified inside their
  created_at) can live in the same index of the current day.</li>
  <li>Are 5 shards enough (or too much) for 21 600 000 documents by day? </li>
  <li>If we want all our aggregate queries to be processed in less than 1 second how many replica should we setup up?</li>
  <li>Should we change our routing? Since we don't know ahead of time which documents will be processed before the aggregation for each request we make to the cluster (since the query is user defined)</li>
  <li>What kind of hardware (how many machines, what configuration) should we put inside this cluster to support 6 month of documents?</li>
  </ol>
  
  <p><strong>[Update]</strong></p>
  
  <p>Here is some example of queries:</p>
  
  <p>A word cloud</p>
  
  <pre class="lang-javascript prettyprint-override"><code>GET idx-2014-05-01/stream/_search?search_type=count
  {
   "query":{
     "bool": {
       "must": [{
         "query_string" : {
           "query" : "(generic.lang:fr OR generic.lang:en) AND (generic.content:javascript)"
          }},{
          "range": {
            "generic.created_at": {
              "from": 1401000000000,
              "to": 1401029019706
            }
          }}
       ]
     }
   },
    "aggs":{
      "words":{
        "terms":{
          "field": "generic.content",
          "size": 40
        }
      }
    }
  }
  </code></pre>
  
  <p>An histogram</p>
  
  <pre class="lang-javascript prettyprint-override"><code>GET idx-2014-05-01/stream/_search?search_type=count
  {
   "query":{
     "bool": {
       "must": [{
         "query_string" : {
           "query" : "generic.content:apple"
          }},{
          "range": {
            "generic.created_at": {
              "from": 1401000000000,
              "to": 1401029019706
            }
          }}
       ]
     }
   },
    "aggs":{
      "volume":{
        "date_histogram":{
          "field": "generic.created_at",
          "interval":"minute"
        }
      }
    }
  }
  </code></pre>
  
  <p>Must used language</p>
  
  <pre class="lang-javascript prettyprint-override"><code>GET idx-2014-05-01/stream/_search?search_type=count
  {
   "query":{
     "bool": {
       "must": [{
         "query_string" : {
           "query" : "(generic.lang:fr OR generic.lang:en) AND (generic.content:javascript)"
          }},{
          "range": {
            "generic.created_at": {
              "from": 1401000000000,
              "to": 1401029019706
            }
          }}
       ]
     }
   },
    "aggs":{
      "top_source":{
        "terms":{
          "field": "generic.lang"
        }
      }
    }
  }
  </code></pre>
  
  <p><img src="https://www.linkedin.com/profile/view?id=AAIAAAD-U-%0AYBJN9lpixm6nxrnAaHQY5DoZ0FcNQ&amp;trk=nav_responsive_tab_profile" width="1" height="1"></p>