{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17186","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17186/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17186/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17186/events","html_url":"https://github.com/cockroachdb/cockroach/issues/17186","id":245073798,"number":17186,"title":"stability: investigate recovery from permanent unavailability","user":{"login":"hxiaodon","id":10990210,"avatar_url":"https://avatars2.githubusercontent.com/u/10990210?v=4","gravatar_id":"","url":"https://api.github.com/users/hxiaodon","html_url":"https://github.com/hxiaodon","followers_url":"https://api.github.com/users/hxiaodon/followers","following_url":"https://api.github.com/users/hxiaodon/following{/other_user}","gists_url":"https://api.github.com/users/hxiaodon/gists{/gist_id}","starred_url":"https://api.github.com/users/hxiaodon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hxiaodon/subscriptions","organizations_url":"https://api.github.com/users/hxiaodon/orgs","repos_url":"https://api.github.com/users/hxiaodon/repos","events_url":"https://api.github.com/users/hxiaodon/events{/privacy}","received_events_url":"https://api.github.com/users/hxiaodon/received_events","type":"User","site_admin":false},"labels":[{"id":560660618,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/community-request","name":"community-request","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/3","html_url":"https://github.com/cockroachdb/cockroach/milestone/3","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/3/labels","id":1298443,"number":3,"title":"Later","description":"","creator":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars1.githubusercontent.com/u/1614265?v=4","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"open_issues":409,"closed_issues":134,"state":"open","created_at":"2015-09-09T21:24:17Z","updated_at":"2017-12-05T14:14:11Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2017-07-24T13:06:46Z","updated_at":"2017-12-05T20:10:46Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\nI proposed this issue in gitter at last weekend. It should be helpful if the disaster recovery tool is provided:)\r\n\r\nHi,All. I started a local cockroachdb cluster with 3 instances(node1,node2,node3), node2 and node3 join node1, all works fine even one node is crashed. When I crash node1 and node2(clean data folder and restart), node3 could not provide service(It's expected), but I could not dump data from node3.Suppose we have this scenario, if 2 nodes are just permanently dead and the daily backup operation was triggered long time ago while the rest healthy node keep the latest data(until service unavailable) compared with the backup one, at this time, I could not use the healthy node's data(dump/backup) with adding other 2 new nodes to provide the service. Does it make sense? Thanks!\r\nWhat I mean is that cluster do not provide service when minority nodes left make sense, but it does not make sense to block dumping data(raft committed one) when minority nodes keep the complete copy(replication number equal to cluster nodes number)\r\n\r\nAndrei Matei's reply\r\n@hxiaodon indeed, we want to provide some type of data recovery tools for this situation, but we haven’t gotten to it yet"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/349426137","html_url":"https://github.com/cockroachdb/cockroach/issues/17186#issuecomment-349426137","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17186","id":349426137,"user":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars1.githubusercontent.com/u/160562?v=4","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"created_at":"2017-12-05T20:10:46Z","updated_at":"2017-12-05T20:10:46Z","author_association":"OWNER","body":"That is more or less what I had in mind for an \"in place lossy recovery\". I still wouldn't prioritize working on this until after we have \"read from follower in the past without quorum\" (which would enable an after-the-fact backup/restore-based recovery), though. There's a lot of tricky logic here that will be rarely exercised, and I'd rather work on things that bring us closer to other things we need than a lot of special cases for this kind of recovery.\r\n\r\n> restart all nodes in the cluster in \"recovery mode\". I think this could just be initializing mode but it may be worth disabling other moving parts (for example, we certainly don't want to be GC'ing replicas now).\r\n\r\nMaybe it's better if this is an offline process: stop all the nodes and run a `cockroach recover` command on each one with a list of node ids that will be purged from all range descriptors. \r\n\r\n> overwrite all copies of the old range descriptor with that one\r\n\r\nAt what timestamp? Is this rewriting history or inserting a new value? What about the uncommitted tail of the raft log? \r\n\r\n> note that this process is restartable, i.e. a crash half-way through doesn't prevent the next attempt from making progress.\r\n\r\nIt's restartable, but it's not reversible. Once you start this process, you're committed (so you can't abort this recovery process if you manage to revive the \"dead\" nodes). \r\n\r\n> SCRUB the data (to see whether the status quo is safe to use) or roll back to a safe AS OF SYSTEM TIME or backup.\r\n\r\nIf SCRUB finds errors it can't fix, then you're back to the \"rewrite all data\" option (or perhaps a more limited version, like dropping and recreating an index). So the viability of this option will depend on how likely this kind of inconsistency will be. I'm worried that errors will be common, and it'll be a while before SCRUB is able to fix a lot of them. "}}
{"actor":{"display_login":"bdarnell"}}