title: Initiated state change for partition from OfflinePartition to OnlinePartition failed
body: <p>I reconfigured my kafka cluster, changing:</p>
  
  <ul>
  <li>default replication factor from 1 to 3 and also</li>
  <li>changing the location of the kafka data dir on disk</li>
  </ul>
  
  <p>So after restarting all nodes, the cluster seemed ok  but then I noticed all the topics are failing to come online. In the logs there are messages like this for each topic:</p>
  
  <pre><code>state-change.log: [2018-02-01 12:41:42,176] ERROR Controller 826437096    epoch 19 initiated state change for partition [filedrop,0] from   OfflinePartition to OnlinePartition failed (state.change.logger)
  </code></pre>
  
  <p>So none of the topics are usable; Listing topics with kafkacat -L -b shows leaders not available.</p>
  
  <pre><code>Metadata for all topics (from broker -1: lol-045:9092/bootstrap):
  7 brokers:
  broker 826437096 at lol-044:9092
  broker 746155422 at lol-047:9092
  broker 651737161 at lol-046:9092
  broker 728512596 at lol-048:9092
  broker 213763378 at lol-045:9092
  broker 622553932 at lol-049:9092
  broker 746727274 at lol-050:9092
  14 topics:
  topic "lol.stripped" with 3 partitions:
   partition 2, leader -1, replicas:, isrs:, Broker: Leader not available
   partition 1, leader -1, replicas:, isrs:, Broker: Leader notavailable
   partition 0, leader -1, replicas:, isrs:, Broker: Leader not available
  </code></pre>
  
  <p>However, newly created topics are correctly replicated and healthy</p>
  
  <pre><code>topic "lol-kafka-health" with 3 partitions:
  partition 2, leader 622553932, replicas: 622553932,213763378,651737161, isrs: 622553932,213763378,651737161
  partition 1, leader 213763378, replicas: 622553932,213763378,826437096, isrs: 213763378,826437096,622553932
  partition 0, leader 826437096, replicas: 213763378,746727274,826437096, isrs: 826437096,746727274,213763378
  </code></pre>
  
  <p>So I think some kind of metadata corruption happened during the reconfigure</p>
  
  <p>My question is:</p>
  
  <ul>
  <li>Is there any way I can get these topic partitions online again ?</li>
  </ul>
  
  <p>Given that:</p>
  
  <ul>
  <li>the broker ids were changed during the reconfigure</li>
  <li>the zookeeper cluster for kafka went down temporarily during the reconfig </li>
  </ul>
  
  <p>In addition, are there some procedures  I can use to investigate how recoverable these topics are?</p>
  
  <p>Many thanks in advance!</p>