{"action":"created","comment":{"url":"https://api.github.com/repos/palantir/atlasdb/pulls/comments/116599443","pull_request_review_id":38232993,"id":116599443,"diff_hunk":"@@ -0,0 +1,57 @@\n+/*\n+ * Copyright 2017 Palantir Technologies, Inc. All rights reserved.\n+ *\n+ * Licensed under the BSD-3 License (the \"License\");\n+ * you may not use this file except in compliance with the License.\n+ * You may obtain a copy of the License at\n+ *\n+ * http://opensource.org/licenses/BSD-3-Clause\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+package com.palantir.atlasdb.keyvalue.api;\n+\n+import javax.annotation.Nullable;\n+\n+import org.immutables.value.Value;\n+\n+@Value.Immutable\n+public interface CandidateCellForSweeping {\n+\n+    Cell cell();\n+\n+    /**\n+     * All start timestamps for the cell that are strictly less than\n+     * {@link CandidateCellForSweepingRequest#sweepTimestamp()} and are not in\n+     * {@link CandidateCellForSweepingRequest#timestampsToIgnore()}, in ascending order.\n+     *\n+     * Note that this will be null if and only if {@link #isCandidate()} is false.\n+     */\n+    @Nullable\n+    long[] sortedTimestamps();\n+\n+    /**\n+     * If {@link CandidateCellForSweepingRequest#shouldCheckIfLatestValueIsEmpty()} was set to true,\n+     * then this method returns true if and only if the value corresponding to the last entry of\n+     * {@link #sortedTimestamps()} is empty.\n+     * <p>\n+     * Otherwise, the return value is undefined and depends on the implementation.\n+     */\n+    boolean isLatestValueEmpty();","path":"atlasdb-api/src/main/java/com/palantir/atlasdb/keyvalue/api/CandidateCellForSweeping.java","position":44,"original_position":44,"commit_id":"71d2e69018708387322effd17a633f770923ccd8","original_commit_id":"71d2e69018708387322effd17a633f770923ccd8","user":{"login":"gbonik","id":994786,"avatar_url":"https://avatars0.githubusercontent.com/u/994786?v=3","gravatar_id":"","url":"https://api.github.com/users/gbonik","html_url":"https://github.com/gbonik","followers_url":"https://api.github.com/users/gbonik/followers","following_url":"https://api.github.com/users/gbonik/following{/other_user}","gists_url":"https://api.github.com/users/gbonik/gists{/gist_id}","starred_url":"https://api.github.com/users/gbonik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gbonik/subscriptions","organizations_url":"https://api.github.com/users/gbonik/orgs","repos_url":"https://api.github.com/users/gbonik/repos","events_url":"https://api.github.com/users/gbonik/events{/privacy}","received_events_url":"https://api.github.com/users/gbonik/received_events","type":"User","site_admin":false},"body":"Having a non-empty value (or any kind of value) at a later timestamp doesn't matter -- we can entirely ignore everything above the sweep timestamp.\r\n\r\nRegarding docs -- take a looks at the precise definition of a \"candidate cell\" in GetCandidateCellsForSweepingShim.\r\n","created_at":"2017-05-15T20:58:58Z","updated_at":"2017-05-15T20:58:58Z","html_url":"https://github.com/palantir/atlasdb/pull/1911#discussion_r116599443","pull_request_url":"https://api.github.com/repos/palantir/atlasdb/pulls/1911","_links":{"self":{"href":"https://api.github.com/repos/palantir/atlasdb/pulls/comments/116599443"},"html":{"href":"https://github.com/palantir/atlasdb/pull/1911#discussion_r116599443"},"pull_request":{"href":"https://api.github.com/repos/palantir/atlasdb/pulls/1911"}}},"pull_request":{"url":"https://api.github.com/repos/palantir/atlasdb/pulls/1911","id":120690338,"html_url":"https://github.com/palantir/atlasdb/pull/1911","diff_url":"https://github.com/palantir/atlasdb/pull/1911.diff","patch_url":"https://github.com/palantir/atlasdb/pull/1911.patch","issue_url":"https://api.github.com/repos/palantir/atlasdb/issues/1911","number":1911,"state":"open","locked":false,"title":"The big sweep rewrite, part 1","user":{"login":"gbonik","id":994786,"avatar_url":"https://avatars0.githubusercontent.com/u/994786?v=3","gravatar_id":"","url":"https://api.github.com/users/gbonik","html_url":"https://github.com/gbonik","followers_url":"https://api.github.com/users/gbonik/followers","following_url":"https://api.github.com/users/gbonik/following{/other_user}","gists_url":"https://api.github.com/users/gbonik/gists{/gist_id}","starred_url":"https://api.github.com/users/gbonik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gbonik/subscriptions","organizations_url":"https://api.github.com/users/gbonik/orgs","repos_url":"https://api.github.com/users/gbonik/repos","events_url":"https://api.github.com/users/gbonik/events{/privacy}","received_events_url":"https://api.github.com/users/gbonik/received_events","type":"User","site_admin":false},"body":"We want to replace KVS.getRangeOfTimestamps() with a new call\r\ngetCandidateCellsForSweeping(). In this commit, to keep the change\r\nsmaller, we temporarily implement the new call using two existing\r\ncalls (getRangeOfTimestamps() and getRange()) without changing the\r\nKeyValueService interface. See GetCandidateCellsForSweepingShim.java.\r\nThis is more or less equivalent to the current implementation, just\r\nwritten differently.\r\n\r\nWhat's different about the new call:\r\n    - It returns individual cells rather than RowResults, so it's more\r\n      clear that large rows can be split into several batches;\r\n    - Its behavior is very sweep-specific and allows only loading\r\n      cells that can potentially be swept. This pushes more logic\r\n      into the underlying database and allows sending less data back\r\n      over the network;\r\n    - The implementation for DbKvs will be much more efficient by using\r\n      the same idea as the recent DbKvs.getRange() rewrite -\r\n      performing a single query to get a single page of results;\r\n    - If the THOROUGH strategy is used, the new call will return the\r\n      flag that indicates whether the value corresponding to the latest\r\n      timestamp is empty. Thus we can avoid calling getRange() to load\r\n      values.\r\n\r\nAlso, we now re-partition the cells before we delete them. There are\r\nnow three parameters that define how batching is performed. See\r\nSweepBatchConfig for the definition of these parameters. We also get rid\r\nof the dynamic batch size multiplier in the background sweeper. It\r\ndidn't really make much sense: if you OOM, you crash hard anyway; if\r\nsome other error occurs, it's either a bug in the code or an\r\nintermittent problem (such as network outage), and decreasing the batch\r\nsize would help neither.\r\n\r\nHere is the plan for the next few commits:\r\n    - Make the new call a part of the KVS interface, but still\r\n      using the \"shim\" as the implementation;\r\n    - Implement the call for DbKvs;\r\n    - Implement the call for CassandraKvs;\r\n    - Implement the call for InMemoryKvs;\r\n    - (Optional) Implement the call for RocksDbKvs & JdbcKvs?\r\n    - Delete getRangeOfTimestamps() because it shouldn't be used anywhere\r\n      at that point.\r\n\r\n-------------\r\nSorry for a huge PR again, but there is no meaningful way to split it.\n\n<!-- Reviewable:start -->\n---\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"34\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/palantir/atlasdb/1911)\n<!-- Reviewable:end -->\n","created_at":"2017-05-15T20:44:04Z","updated_at":"2017-05-15T20:58:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e17ae5709c6fb53e9516f027af0d3e10974c972e","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/palantir/atlasdb/pulls/1911/commits","review_comments_url":"https://api.github.com/repos/palantir/atlasdb/pulls/1911/comments","review_comment_url":"https://api.github.com/repos/palantir/atlasdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/palantir/atlasdb/issues/1911/comments","statuses_url":"https://api.github.com/repos/palantir/atlasdb/statuses/71d2e69018708387322effd17a633f770923ccd8","head":{"label":"palantir:feature/sweep-part1","ref":"feature/sweep-part1","sha":"71d2e69018708387322effd17a633f770923ccd8","user":{"login":"palantir","id":303157,"avatar_url":"https://avatars3.githubusercontent.com/u/303157?v=3","gravatar_id":"","url":"https://api.github.com/users/palantir","html_url":"https://github.com/palantir","followers_url":"https://api.github.com/users/palantir/followers","following_url":"https://api.github.com/users/palantir/following{/other_user}","gists_url":"https://api.github.com/users/palantir/gists{/gist_id}","starred_url":"https://api.github.com/users/palantir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palantir/subscriptions","organizations_url":"https://api.github.com/users/palantir/orgs","repos_url":"https://api.github.com/users/palantir/repos","events_url":"https://api.github.com/users/palantir/events{/privacy}","received_events_url":"https://api.github.com/users/palantir/received_events","type":"Organization","site_admin":false},"repo":{"id":36960637,"name":"atlasdb","full_name":"palantir/atlasdb","owner":{"login":"palantir","id":303157,"avatar_url":"https://avatars3.githubusercontent.com/u/303157?v=3","gravatar_id":"","url":"https://api.github.com/users/palantir","html_url":"https://github.com/palantir","followers_url":"https://api.github.com/users/palantir/followers","following_url":"https://api.github.com/users/palantir/following{/other_user}","gists_url":"https://api.github.com/users/palantir/gists{/gist_id}","starred_url":"https://api.github.com/users/palantir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palantir/subscriptions","organizations_url":"https://api.github.com/users/palantir/orgs","repos_url":"https://api.github.com/users/palantir/repos","events_url":"https://api.github.com/users/palantir/events{/privacy}","received_events_url":"https://api.github.com/users/palantir/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/palantir/atlasdb","description":"Transactional Distributed Database Layer","fork":false,"url":"https://api.github.com/repos/palantir/atlasdb","forks_url":"https://api.github.com/repos/palantir/atlasdb/forks","keys_url":"https://api.github.com/repos/palantir/atlasdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/palantir/atlasdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/palantir/atlasdb/teams","hooks_url":"https://api.github.com/repos/palantir/atlasdb/hooks","issue_events_url":"https://api.github.com/repos/palantir/atlasdb/issues/events{/number}","events_url":"https://api.github.com/repos/palantir/atlasdb/events","assignees_url":"https://api.github.com/repos/palantir/atlasdb/assignees{/user}","branches_url":"https://api.github.com/repos/palantir/atlasdb/branches{/branch}","tags_url":"https://api.github.com/repos/palantir/atlasdb/tags","blobs_url":"https://api.github.com/repos/palantir/atlasdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/palantir/atlasdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/palantir/atlasdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/palantir/atlasdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/palantir/atlasdb/statuses/{sha}","languages_url":"https://api.github.com/repos/palantir/atlasdb/languages","stargazers_url":"https://api.github.com/repos/palantir/atlasdb/stargazers","contributors_url":"https://api.github.com/repos/palantir/atlasdb/contributors","subscribers_url":"https://api.github.com/repos/palantir/atlasdb/subscribers","subscription_url":"https://api.github.com/repos/palantir/atlasdb/subscription","commits_url":"https://api.github.com/repos/palantir/atlasdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/palantir/atlasdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/palantir/atlasdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/palantir/atlasdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/palantir/atlasdb/contents/{+path}","compare_url":"https://api.github.com/repos/palantir/atlasdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/palantir/atlasdb/merges","archive_url":"https://api.github.com/repos/palantir/atlasdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/palantir/atlasdb/downloads","issues_url":"https://api.github.com/repos/palantir/atlasdb/issues{/number}","pulls_url":"https://api.github.com/repos/palantir/atlasdb/pulls{/number}","milestones_url":"https://api.github.com/repos/palantir/atlasdb/milestones{/number}","notifications_url":"https://api.github.com/repos/palantir/atlasdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/palantir/atlasdb/labels{/name}","releases_url":"https://api.github.com/repos/palantir/atlasdb/releases{/id}","deployments_url":"https://api.github.com/repos/palantir/atlasdb/deployments","created_at":"2015-06-05T23:42:44Z","updated_at":"2017-05-13T10:15:02Z","pushed_at":"2017-05-15T20:44:12Z","git_url":"git://github.com/palantir/atlasdb.git","ssh_url":"git@github.com:palantir/atlasdb.git","clone_url":"https://github.com/palantir/atlasdb.git","svn_url":"https://github.com/palantir/atlasdb","homepage":"http://palantir.github.io/atlasdb/","size":36617,"stargazers_count":469,"watchers_count":469,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":76,"mirror_url":null,"open_issues_count":150,"forks":76,"open_issues":150,"watchers":469,"default_branch":"develop"}},"base":{"label":"palantir:develop","ref":"develop","sha":"27e38e2f8f4cc5fada7510b090550e74850f13d3","user":{"login":"palantir","id":303157,"avatar_url":"https://avatars3.githubusercontent.com/u/303157?v=3","gravatar_id":"","url":"https://api.github.com/users/palantir","html_url":"https://github.com/palantir","followers_url":"https://api.github.com/users/palantir/followers","following_url":"https://api.github.com/users/palantir/following{/other_user}","gists_url":"https://api.github.com/users/palantir/gists{/gist_id}","starred_url":"https://api.github.com/users/palantir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palantir/subscriptions","organizations_url":"https://api.github.com/users/palantir/orgs","repos_url":"https://api.github.com/users/palantir/repos","events_url":"https://api.github.com/users/palantir/events{/privacy}","received_events_url":"https://api.github.com/users/palantir/received_events","type":"Organization","site_admin":false},"repo":{"id":36960637,"name":"atlasdb","full_name":"palantir/atlasdb","owner":{"login":"palantir","id":303157,"avatar_url":"https://avatars3.githubusercontent.com/u/303157?v=3","gravatar_id":"","url":"https://api.github.com/users/palantir","html_url":"https://github.com/palantir","followers_url":"https://api.github.com/users/palantir/followers","following_url":"https://api.github.com/users/palantir/following{/other_user}","gists_url":"https://api.github.com/users/palantir/gists{/gist_id}","starred_url":"https://api.github.com/users/palantir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palantir/subscriptions","organizations_url":"https://api.github.com/users/palantir/orgs","repos_url":"https://api.github.com/users/palantir/repos","events_url":"https://api.github.com/users/palantir/events{/privacy}","received_events_url":"https://api.github.com/users/palantir/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/palantir/atlasdb","description":"Transactional Distributed Database Layer","fork":false,"url":"https://api.github.com/repos/palantir/atlasdb","forks_url":"https://api.github.com/repos/palantir/atlasdb/forks","keys_url":"https://api.github.com/repos/palantir/atlasdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/palantir/atlasdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/palantir/atlasdb/teams","hooks_url":"https://api.github.com/repos/palantir/atlasdb/hooks","issue_events_url":"https://api.github.com/repos/palantir/atlasdb/issues/events{/number}","events_url":"https://api.github.com/repos/palantir/atlasdb/events","assignees_url":"https://api.github.com/repos/palantir/atlasdb/assignees{/user}","branches_url":"https://api.github.com/repos/palantir/atlasdb/branches{/branch}","tags_url":"https://api.github.com/repos/palantir/atlasdb/tags","blobs_url":"https://api.github.com/repos/palantir/atlasdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/palantir/atlasdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/palantir/atlasdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/palantir/atlasdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/palantir/atlasdb/statuses/{sha}","languages_url":"https://api.github.com/repos/palantir/atlasdb/languages","stargazers_url":"https://api.github.com/repos/palantir/atlasdb/stargazers","contributors_url":"https://api.github.com/repos/palantir/atlasdb/contributors","subscribers_url":"https://api.github.com/repos/palantir/atlasdb/subscribers","subscription_url":"https://api.github.com/repos/palantir/atlasdb/subscription","commits_url":"https://api.github.com/repos/palantir/atlasdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/palantir/atlasdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/palantir/atlasdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/palantir/atlasdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/palantir/atlasdb/contents/{+path}","compare_url":"https://api.github.com/repos/palantir/atlasdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/palantir/atlasdb/merges","archive_url":"https://api.github.com/repos/palantir/atlasdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/palantir/atlasdb/downloads","issues_url":"https://api.github.com/repos/palantir/atlasdb/issues{/number}","pulls_url":"https://api.github.com/repos/palantir/atlasdb/pulls{/number}","milestones_url":"https://api.github.com/repos/palantir/atlasdb/milestones{/number}","notifications_url":"https://api.github.com/repos/palantir/atlasdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/palantir/atlasdb/labels{/name}","releases_url":"https://api.github.com/repos/palantir/atlasdb/releases{/id}","deployments_url":"https://api.github.com/repos/palantir/atlasdb/deployments","created_at":"2015-06-05T23:42:44Z","updated_at":"2017-05-13T10:15:02Z","pushed_at":"2017-05-15T20:44:12Z","git_url":"git://github.com/palantir/atlasdb.git","ssh_url":"git@github.com:palantir/atlasdb.git","clone_url":"https://github.com/palantir/atlasdb.git","svn_url":"https://github.com/palantir/atlasdb","homepage":"http://palantir.github.io/atlasdb/","size":36617,"stargazers_count":469,"watchers_count":469,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":76,"mirror_url":null,"open_issues_count":150,"forks":76,"open_issues":150,"watchers":469,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/palantir/atlasdb/pulls/1911"},"html":{"href":"https://github.com/palantir/atlasdb/pull/1911"},"issue":{"href":"https://api.github.com/repos/palantir/atlasdb/issues/1911"},"comments":{"href":"https://api.github.com/repos/palantir/atlasdb/issues/1911/comments"},"review_comments":{"href":"https://api.github.com/repos/palantir/atlasdb/pulls/1911/comments"},"review_comment":{"href":"https://api.github.com/repos/palantir/atlasdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/palantir/atlasdb/pulls/1911/commits"},"statuses":{"href":"https://api.github.com/repos/palantir/atlasdb/statuses/71d2e69018708387322effd17a633f770923ccd8"}}}}
{"actor":{"display_login":"gbonik"}}