{"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/picard/issues/893","repository_url":"https://api.github.com/repos/broadinstitute/picard","labels_url":"https://api.github.com/repos/broadinstitute/picard/issues/893/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/picard/issues/893/comments","events_url":"https://api.github.com/repos/broadinstitute/picard/issues/893/events","html_url":"https://github.com/broadinstitute/picard/pull/893","id":247865117,"number":893,"title":"Adding the ability to set a maximum size of duplicate sets for checking optical dups","user":{"login":"eitanbanks","id":2751498,"avatar_url":"https://avatars0.githubusercontent.com/u/2751498?v=4","gravatar_id":"","url":"https://api.github.com/users/eitanbanks","html_url":"https://github.com/eitanbanks","followers_url":"https://api.github.com/users/eitanbanks/followers","following_url":"https://api.github.com/users/eitanbanks/following{/other_user}","gists_url":"https://api.github.com/users/eitanbanks/gists{/gist_id}","starred_url":"https://api.github.com/users/eitanbanks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eitanbanks/subscriptions","organizations_url":"https://api.github.com/users/eitanbanks/orgs","repos_url":"https://api.github.com/users/eitanbanks/repos","events_url":"https://api.github.com/users/eitanbanks/events{/privacy}","received_events_url":"https://api.github.com/users/eitanbanks/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-08-03T23:40:19Z","updated_at":"2017-08-03T23:41:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/broadinstitute/picard/pulls/893","html_url":"https://github.com/broadinstitute/picard/pull/893","diff_url":"https://github.com/broadinstitute/picard/pull/893.diff","patch_url":"https://github.com/broadinstitute/picard/pull/893.patch"},"body":"### Description\r\n\r\nThe algorithm for determining optical duplicates is n^2, which is fine when handling normal samples.  But when you hit a sample that has one or more duplicate sets with over 1 million read pairs, it's really bad (yup, that's over a trillion comparisons made in that algorithm; we're seeing that take over a week to run!).  While it may be worth making the algorithm more efficient one day, really once we have a duplicate set with that many reads we're okay just skipping the optical dup check.\r\n\r\nIn this PR I've added a max set size for the optical dup check.  I've set it to 300K because that's already ~10 billion comparisons.  A user can always disable this maximum (at his/her own peril) on the command-line.\r\n\r\n### Checklist (never delete this)\r\n\r\nNever delete this, it is our record that procedure was followed. If you find that for whatever reason one of the checklist points doesn't apply to your PR, you can leave it unchecked but please add an explanation below.\r\n\r\n#### Content\r\n- [ ] Added or modified tests to cover changes and any new functionality\r\n- [ ] Edited the README / documentation (if applicable)\r\n- [ ] All tests passing on Travis\r\n\r\n#### Review\r\n- [ ] Final thumbs-up from reviewer\r\n- [ ] Rebase, squash and reword as applicable\r\n\r\nFor more detailed guidelines, see https://github.com/broadinstitute/picard/wiki/Guidelines-for-pull-requests\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/broadinstitute/picard/issues/comments/320118451","html_url":"https://github.com/broadinstitute/picard/pull/893#issuecomment-320118451","issue_url":"https://api.github.com/repos/broadinstitute/picard/issues/893","id":320118451,"user":{"login":"eitanbanks","id":2751498,"avatar_url":"https://avatars0.githubusercontent.com/u/2751498?v=4","gravatar_id":"","url":"https://api.github.com/users/eitanbanks","html_url":"https://github.com/eitanbanks","followers_url":"https://api.github.com/users/eitanbanks/followers","following_url":"https://api.github.com/users/eitanbanks/following{/other_user}","gists_url":"https://api.github.com/users/eitanbanks/gists{/gist_id}","starred_url":"https://api.github.com/users/eitanbanks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eitanbanks/subscriptions","organizations_url":"https://api.github.com/users/eitanbanks/orgs","repos_url":"https://api.github.com/users/eitanbanks/repos","events_url":"https://api.github.com/users/eitanbanks/events{/privacy}","received_events_url":"https://api.github.com/users/eitanbanks/received_events","type":"User","site_admin":false},"created_at":"2017-08-03T23:41:39Z","updated_at":"2017-08-03T23:41:39Z","body":"@tfenne would love your review here if you have time (I know that Nils just had a baby so don't want to bother him).  Otherwise, I'll just wait until Yossi is back from vacation."}}
{"actor":{"display_login":"eitanbanks"}}