{"action":"created","comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/120204815","pull_request_review_id":42148519,"id":120204815,"diff_hunk":"@@ -470,6 +483,39 @@ func (r *Replica) leasePostApply(\n \t}\n }\n \n+var addMu syncutil.Mutex\n+\n+func (r *Replica) addSSTablePostApply(ctx context.Context, data []byte) {\n+\tvar checksum []byte\n+\t{\n+\t\thash := sha512.New()","path":"pkg/storage/replica_proposal.go","position":47,"original_position":47,"commit_id":"f93686ae4d7e5371caab4c95345210cb787a5318","original_commit_id":"f93686ae4d7e5371caab4c95345210cb787a5318","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars2.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"body":"I think we settled on CRC32.","created_at":"2017-06-05T20:44:16Z","updated_at":"2017-06-05T20:44:16Z","html_url":"https://github.com/cockroachdb/cockroach/pull/16158#discussion_r120204815","pull_request_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158","_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/120204815"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/16158#discussion_r120204815"},"pull_request":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158"}}},"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158","id":122549299,"html_url":"https://github.com/cockroachdb/cockroach/pull/16158","diff_url":"https://github.com/cockroachdb/cockroach/pull/16158.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/16158.patch","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/16158","number":16158,"state":"open","locked":false,"title":"[WIP] storage: add AddSSTable command to link a preconstructed file into RocksDB","user":{"login":"danhhz","id":52528,"avatar_url":"https://avatars2.githubusercontent.com/u/52528?v=3","gravatar_id":"","url":"https://api.github.com/users/danhhz","html_url":"https://github.com/danhhz","followers_url":"https://api.github.com/users/danhhz/followers","following_url":"https://api.github.com/users/danhhz/following{/other_user}","gists_url":"https://api.github.com/users/danhhz/gists{/gist_id}","starred_url":"https://api.github.com/users/danhhz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danhhz/subscriptions","organizations_url":"https://api.github.com/users/danhhz/orgs","repos_url":"https://api.github.com/users/danhhz/repos","events_url":"https://api.github.com/users/danhhz/events{/privacy}","received_events_url":"https://api.github.com/users/danhhz/received_events","type":"User","site_admin":false},"body":"This is the implementation of https://github.com/cockroachdb/cockroach/pull/15895.\r\n\r\nA command is added that takes the contents of an sstable as constructed\r\nby a RocksDB SstFileWriter (or our RocksDBSstFileWriter wrapper).\r\n\r\nThe propser verifies that the keys in the sstable are within the range\r\nspecified by the request header, verifies the key-value checksums, and\r\ncomputes the new MVCCStats.\r\n\r\nOn each replica, as a side effect of application, the file is linked\r\ndirectly into RocksDB using its IngestExternalFile call.\r\n\r\n---\r\n\r\nA new cluster setting 'kv.import.addsstable.enabled' is added to\r\noptionally use the new AddSSTable command to implement Import (and thus\r\nRestore). This should lead to reduced disk usage and eventually\r\ndramatically increased throughput.\r\n\r\nThe setting defaults to false for now while we finalize the\r\nimplementation of AddSSTable (we're looking to avoid writing the file\r\ncontents in the normal RocksDB raft log to save even more disk usage)\r\nand while we run it through production testing.\r\n\r\nA test is added with runs the usual backupAndRestore test but with the\r\nsetting turned on.\r\n\r\nThis PR also introduces a hook to disable the use of RocksDB range\r\ndeletiongs. This works around a limitation in RocksDB that currently\r\ndoes lot allow ingesting a file while a snapshot is outstanding. There\r\nis an upcoming fix in RocksDB which will let us remove this workaround.\r\n\r\n---\r\n\r\nThis is still waiting on a fix from RocksDB and the AddSSTable command wants some tests (it is tested a bit in TestBackupRestoreAddSSTable), but I wanted to get it out for early review. Note that this still stores the sstable data in RocksDB on behalf of the raft log, @tschottdorf has an upcoming rfc to address that","created_at":"2017-05-25T23:55:10Z","updated_at":"2017-06-05T20:44:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8352f3385c6ec138113aef64a289d1e7eb28df15","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158/commits","review_comments_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158/comments","review_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/16158/comments","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/f93686ae4d7e5371caab4c95345210cb787a5318","head":{"label":"danhhz:add_sstable","ref":"add_sstable","sha":"f93686ae4d7e5371caab4c95345210cb787a5318","user":{"login":"danhhz","id":52528,"avatar_url":"https://avatars2.githubusercontent.com/u/52528?v=3","gravatar_id":"","url":"https://api.github.com/users/danhhz","html_url":"https://github.com/danhhz","followers_url":"https://api.github.com/users/danhhz/followers","following_url":"https://api.github.com/users/danhhz/following{/other_user}","gists_url":"https://api.github.com/users/danhhz/gists{/gist_id}","starred_url":"https://api.github.com/users/danhhz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danhhz/subscriptions","organizations_url":"https://api.github.com/users/danhhz/orgs","repos_url":"https://api.github.com/users/danhhz/repos","events_url":"https://api.github.com/users/danhhz/events{/privacy}","received_events_url":"https://api.github.com/users/danhhz/received_events","type":"User","site_admin":false},"repo":{"id":47640635,"name":"cockroach","full_name":"danhhz/cockroach","owner":{"login":"danhhz","id":52528,"avatar_url":"https://avatars2.githubusercontent.com/u/52528?v=3","gravatar_id":"","url":"https://api.github.com/users/danhhz","html_url":"https://github.com/danhhz","followers_url":"https://api.github.com/users/danhhz/followers","following_url":"https://api.github.com/users/danhhz/following{/other_user}","gists_url":"https://api.github.com/users/danhhz/gists{/gist_id}","starred_url":"https://api.github.com/users/danhhz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danhhz/subscriptions","organizations_url":"https://api.github.com/users/danhhz/orgs","repos_url":"https://api.github.com/users/danhhz/repos","events_url":"https://api.github.com/users/danhhz/events{/privacy}","received_events_url":"https://api.github.com/users/danhhz/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/danhhz/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":true,"url":"https://api.github.com/repos/danhhz/cockroach","forks_url":"https://api.github.com/repos/danhhz/cockroach/forks","keys_url":"https://api.github.com/repos/danhhz/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/danhhz/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/danhhz/cockroach/teams","hooks_url":"https://api.github.com/repos/danhhz/cockroach/hooks","issue_events_url":"https://api.github.com/repos/danhhz/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/danhhz/cockroach/events","assignees_url":"https://api.github.com/repos/danhhz/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/danhhz/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/danhhz/cockroach/tags","blobs_url":"https://api.github.com/repos/danhhz/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/danhhz/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/danhhz/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/danhhz/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/danhhz/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/danhhz/cockroach/languages","stargazers_url":"https://api.github.com/repos/danhhz/cockroach/stargazers","contributors_url":"https://api.github.com/repos/danhhz/cockroach/contributors","subscribers_url":"https://api.github.com/repos/danhhz/cockroach/subscribers","subscription_url":"https://api.github.com/repos/danhhz/cockroach/subscription","commits_url":"https://api.github.com/repos/danhhz/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/danhhz/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/danhhz/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/danhhz/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/danhhz/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/danhhz/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/danhhz/cockroach/merges","archive_url":"https://api.github.com/repos/danhhz/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/danhhz/cockroach/downloads","issues_url":"https://api.github.com/repos/danhhz/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/danhhz/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/danhhz/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/danhhz/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/danhhz/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/danhhz/cockroach/releases{/id}","deployments_url":"https://api.github.com/repos/danhhz/cockroach/deployments","created_at":"2015-12-08T18:19:02Z","updated_at":"2016-03-09T21:31:14Z","pushed_at":"2017-05-31T21:45:27Z","git_url":"git://github.com/danhhz/cockroach.git","ssh_url":"git@github.com:danhhz/cockroach.git","clone_url":"https://github.com/danhhz/cockroach.git","svn_url":"https://github.com/danhhz/cockroach","homepage":null,"size":325969,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"cockroachdb:master","ref":"master","sha":"7fb6d983651d77cbd200f3553ce842a82fcf30ff","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars0.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars0.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"CockroachDB - the open source, cloud-native SQL database.","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","deployments_url":"https://api.github.com/repos/cockroachdb/cockroach/deployments","created_at":"2014-02-06T00:18:47Z","updated_at":"2017-06-05T20:24:05Z","pushed_at":"2017-06-05T20:40:12Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":"https://www.cockroachlabs.com","size":591978,"stargazers_count":10411,"watchers_count":10411,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":991,"mirror_url":null,"open_issues_count":955,"forks":991,"open_issues":955,"watchers":10411,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/16158"},"issue":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/16158"},"comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/16158/comments"},"review_comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158/comments"},"review_comment":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/16158/commits"},"statuses":{"href":"https://api.github.com/repos/cockroachdb/cockroach/statuses/f93686ae4d7e5371caab4c95345210cb787a5318"}}}}
{"actor":{"display_login":"tschottdorf"}}