{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/27536","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/27536/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/27536/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/27536/events","html_url":"https://github.com/elastic/elasticsearch/issues/27536","id":277018985,"number":27536,"title":"Translog Operations with no sequence numbers fail recoveries after a rolling upgrading to 6.0","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars1.githubusercontent.com/u/1006375?v=4","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"labels":[{"id":152510590,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800","default":false},{"id":288393673,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Sequence%20IDs","name":":Sequence IDs","color":"009800","default":false},{"id":92913658,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/blocker","name":"blocker","color":"e11d21","default":false},{"id":23173,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c","default":true},{"id":700184271,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v6.0.1","name":"v6.0.1","color":"DDDDDD","default":false},{"id":654494999,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v6.1.0","name":"v6.1.0","color":"dddddd","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-11-27T13:31:24Z","updated_at":"2017-12-05T14:43:56Z","closed_at":"2017-12-05T14:43:56Z","author_association":"MEMBER","body":"During a rolling upgrade shards on 6.0 nodes should be able to work with operations with no sequence numbers when the primary is still on a 5.6 node. After the primary is on a 6.0 node, it will start generating sequence numbers for all shard copies. To simplify the number of edge cases we have to deal with we have designed the code to only make the transition in one direction - i.e., once a shard moves to the sequence numbers universe, it will never go back to operated in a non sequence numbers universe. We also added protections in place to verify it doesn't happen.\r\n\r\nSadly, a rolling upgrade can lead to a situation that triggers those protections and fail the recoveries. Here is a typical exception line:\r\n\r\n> Recovery failed from {bos1-es3}{JR9qqMjEQF6c22eGZqIAcw}{izoNnZpWTsyt5_xxymVDbg}{192.168.8.152}{192.168.8.152:9300}{ml.max_open_jobs=10, ml.enabled=true} into {bos1-es1}{tBZdi1W_TRa16getb8eNlA}{picaUSZyS7K34rTVpJvu1Q}{192.168.8.150}{192.168.8.150:9300}{ml.max_open_jobs=10, ml.enabled=true}]; nested: RemoteTransportException[[bos1-es3][192.168.8.152:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[2] phase2 failed]; nested: RemoteTransportException[[bos1-es1][192.168.8.150:9300][internal:index/shard/recovery/translog_ops]]; nested: TranslogException[Failed to write operation [Index{id='AV_HS9TBPU9Vhf7shs_U', type='px-web-server', seqNo=-2, primaryTerm=0}]]; nested: IllegalArgumentException[sequence number must be assigned];\r\n\r\nThis can happen in the following scenario:\r\n1) A replica is on a 6.0 node while the primary is on 5.6 while indexing is going on.\r\n2) The replica inserts operations with no seq# to it's translog\r\n3) The primary goes down (planned or not) and the replica becomes a new primary.\r\n4)  The primary comes back (or another shard) and starts recovery.  While recovery is ongoing, indexing operations come in.\r\n5)  The recovering replica opens up its engine and processes the new indexing operations, switching to seq# mode.\r\n6) The old operations from the translogs are replayed and since they have no seq #s in them the assumptions are violated and the recovery failed. \r\n\r\nThis problem is made worse by the fact that we now ship all the translogs to create a replica that adheres to the translog retention policy.\r\n\r\nTo work around it, people can reduce their translog retention policy to 0 (`index.translog.retention.size`) and then flush the primary shard. This should clean up the old ops from the translog. After that (and setting `index.translog.retention.size` back to null) you can run a reroute command with retry_failed flag : `POST /_cluster/reroute?retry_failed=true`. \r\n\r\nI'm still evaluating possible solutions. Based on the complexity we can decide which version the solution should go to. It's also an open question why our tests didn't catch this. "}}
{"actor":{"display_login":"bleskes"}}