title: elasticsearch - there is no copy of the shard available?
body: <p>I have a few indices in red, after a failure of the system, caused by disk full.
  But I cannot reallocate the lost shard. It says "there is no copy of the shard available"</p>
  
  <pre><code> curl -XGET 'localhost:9200/_cluster/allocation/explain?pretty'
  {
    "shard" : {
      "index" : "my_index",
      "index_uuid" : "iNY9t81wQf6wJc-KqufUrg",
      "id" : 0,
      "primary" : true
    },
    "assigned" : false,
    "shard_state_fetch_pending" : false,
    "unassigned_info" : {
      "reason" : "ALLOCATION_FAILED",
      "at" : "2017-05-30T07:33:04.192Z",
      "failed_attempts" : 5,
      "delayed" : false,
      "details" : "failed to create shard, failure FileSystemException[/data/es/storage/nodes/0/indices/iNY9t81wQf6wJc-KqufUrg/0/_state/state-13.st.tmp: Read-only file system]",
      "allocation_status" : "deciders_no"
    },
    "allocation_delay_in_millis" : 60000,
    "remaining_delay_in_millis" : 0,
    "nodes" : {
      "KvOd2vSQTOSgjgqyEnOKpA" : {
        "node_name" : "node1",
        "node_attributes" : { },
        "store" : {
          "shard_copy" : "NONE"
        },
        "final_decision" : "NO",
        "final_explanation" : "there is no copy of the shard available",
        "weight" : -3.683333,
        "decisions" : [
          {
            "decider" : "max_retry",
            "decision" : "NO",
            "explanation" : "shard has already failed allocating [5] times vs. [5] retries allowed unassigned_info[[reason=ALLOCATION_FAILED], at[2017-05-30T07:33:04.192Z], failed_attempts[5], delayed=false, details[failed to create shard, failure FileSystemException[/data/es/storage/nodes/0/indices/iNY9t81wQf6wJc-KqufUrg/0/_state/state-13.st.tmp: Read-only file system]], allocation_status[deciders_no]] - manually call [/_cluster/reroute?retry_failed=true] to retry"
          }
        ]
      },
      "pC9fL41xRgeZDAEYvNR1eQ" : {
        "node_name" : "node2",
        "node_attributes" : { },
        "store" : {
          "shard_copy" : "AVAILABLE"
        },
        "final_decision" : "NO",
        "final_explanation" : "the shard cannot be assigned because one or more allocation decider returns a 'NO' decision",
        "weight" : -2.333333,
        "decisions" : [
          {
            "decider" : "max_retry",
            "decision" : "NO",
            "explanation" : "shard has already failed allocating [5] times vs. [5] retries allowed unassigned_info[[reason=ALLOCATION_FAILED], at[2017-05-30T07:33:04.192Z], failed_attempts[5], delayed=false, details[failed to create shard, failure FileSystemException[/data/es/storage/nodes/0/indices/iNY9t81wQf6wJc-KqufUrg/0/_state/state-13.st.tmp: Read-only file system]], allocation_status[deciders_no]] - manually call [/_cluster/reroute?retry_failed=true] to retry"
          }
        ]
      },
      "1g7eCfEQS9u868lFSoo7FQ" : {
        "node_name" : "node3",
        "node_attributes" : { },
        "store" : {
          "shard_copy" : "AVAILABLE"
        },
        "final_decision" : "NO",
        "final_explanation" : "the shard cannot be assigned because one or more allocation decider returns a 'NO' decision",
        "weight" : 40.866665,
        "decisions" : [
          {
            "decider" : "max_retry",
            "decision" : "NO",
            "explanation" : "shard has already failed allocating [5] times vs. [5] retries allowed unassigned_info[[reason=ALLOCATION_FAILED], at[2017-05-30T07:33:04.192Z], failed_attempts[5], delayed=false, details[failed to create shard, failure FileSystemException[/data/es/storage/nodes/0/indices/iNY9t81wQf6wJc-KqufUrg/0/_state/state-13.st.tmp: Read-only file system]], allocation_status[deciders_no]] - manually call [/_cluster/reroute?retry_failed=true] to retry"
          }
        ]
      }
    }
  }
  </code></pre>
  
  <p>I tried basically every option of the <code>reroute</code> command (documentation <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html" rel="nofollow noreferrer">here</a>). but it gives me 400 error.. like this:</p>
  
  <pre><code>curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -H 'Content-Type: application/json' -d'
  {
      "commands" : [
  
          {
            "allocate_replica" : {
                  "index" : "myindex", "shard" : 0,
                  "node" : "node2"
            }
          }
      ]
  }'
  </code></pre>
  
  <p>response:</p>
  
  <pre><code>{
    "error" : {
      "root_cause" : [
        {
          "type" : "illegal_argument_exception",
          "reason" : "[allocate_replica] trying to allocate a replica shard [myindex][0], while corresponding primary shard is still unassigned"
        }
      ],
      "type" : "illegal_argument_exception",
      "reason" : "[allocate_replica] trying to allocate a replica shard [myindex][0], while corresponding primary shard is still unassigned"
    },
    "status" : 400
  }
  </code></pre>