{"action":"created","issue":{"url":"https://api.github.com/repos/cfe-lab/MiCall/issues/393","repository_url":"https://api.github.com/repos/cfe-lab/MiCall","labels_url":"https://api.github.com/repos/cfe-lab/MiCall/issues/393/labels{/name}","comments_url":"https://api.github.com/repos/cfe-lab/MiCall/issues/393/comments","events_url":"https://api.github.com/repos/cfe-lab/MiCall/issues/393/events","html_url":"https://github.com/cfe-lab/MiCall/issues/393","id":228876336,"number":393,"title":"Reduce memory consumption in prelim_map","user":{"login":"donkirkby","id":1639148,"avatar_url":"https://avatars2.githubusercontent.com/u/1639148?v=3","gravatar_id":"","url":"https://api.github.com/users/donkirkby","html_url":"https://github.com/donkirkby","followers_url":"https://api.github.com/users/donkirkby/followers","following_url":"https://api.github.com/users/donkirkby/following{/other_user}","gists_url":"https://api.github.com/users/donkirkby/gists{/gist_id}","starred_url":"https://api.github.com/users/donkirkby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/donkirkby/subscriptions","organizations_url":"https://api.github.com/users/donkirkby/orgs","repos_url":"https://api.github.com/users/donkirkby/repos","events_url":"https://api.github.com/users/donkirkby/events{/privacy}","received_events_url":"https://api.github.com/users/donkirkby/received_events","type":"User","site_admin":false},"labels":[{"id":93313964,"url":"https://api.github.com/repos/cfe-lab/MiCall/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":130308520,"url":"https://api.github.com/repos/cfe-lab/MiCall/labels/in%20progress","name":"in progress","color":"ededed","default":false}],"state":"open","locked":false,"assignee":{"login":"donkirkby","id":1639148,"avatar_url":"https://avatars2.githubusercontent.com/u/1639148?v=3","gravatar_id":"","url":"https://api.github.com/users/donkirkby","html_url":"https://github.com/donkirkby","followers_url":"https://api.github.com/users/donkirkby/followers","following_url":"https://api.github.com/users/donkirkby/following{/other_user}","gists_url":"https://api.github.com/users/donkirkby/gists{/gist_id}","starred_url":"https://api.github.com/users/donkirkby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/donkirkby/subscriptions","organizations_url":"https://api.github.com/users/donkirkby/orgs","repos_url":"https://api.github.com/users/donkirkby/repos","events_url":"https://api.github.com/users/donkirkby/events{/privacy}","received_events_url":"https://api.github.com/users/donkirkby/received_events","type":"User","site_admin":false},"assignees":[{"login":"donkirkby","id":1639148,"avatar_url":"https://avatars2.githubusercontent.com/u/1639148?v=3","gravatar_id":"","url":"https://api.github.com/users/donkirkby","html_url":"https://github.com/donkirkby","followers_url":"https://api.github.com/users/donkirkby/followers","following_url":"https://api.github.com/users/donkirkby/following{/other_user}","gists_url":"https://api.github.com/users/donkirkby/gists{/gist_id}","starred_url":"https://api.github.com/users/donkirkby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/donkirkby/subscriptions","organizations_url":"https://api.github.com/users/donkirkby/orgs","repos_url":"https://api.github.com/users/donkirkby/repos","events_url":"https://api.github.com/users/donkirkby/events{/privacy}","received_events_url":"https://api.github.com/users/donkirkby/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cfe-lab/MiCall/milestones/5","html_url":"https://github.com/cfe-lab/MiCall/milestone/5","labels_url":"https://api.github.com/repos/cfe-lab/MiCall/milestones/5/labels","id":766561,"number":5,"title":"near future","description":"Issues that will be in the next few releases","creator":{"login":"donkirkby","id":1639148,"avatar_url":"https://avatars2.githubusercontent.com/u/1639148?v=3","gravatar_id":"","url":"https://api.github.com/users/donkirkby","html_url":"https://github.com/donkirkby","followers_url":"https://api.github.com/users/donkirkby/followers","following_url":"https://api.github.com/users/donkirkby/following{/other_user}","gists_url":"https://api.github.com/users/donkirkby/gists{/gist_id}","starred_url":"https://api.github.com/users/donkirkby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/donkirkby/subscriptions","organizations_url":"https://api.github.com/users/donkirkby/orgs","repos_url":"https://api.github.com/users/donkirkby/repos","events_url":"https://api.github.com/users/donkirkby/events{/privacy}","received_events_url":"https://api.github.com/users/donkirkby/received_events","type":"User","site_admin":false},"open_issues":63,"closed_issues":1,"state":"open","created_at":"2014-08-26T20:06:05Z","updated_at":"2017-06-27T21:47:50Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2017-05-15T23:42:01Z","updated_at":"2017-07-11T21:22:35Z","closed_at":null,"body":"We're trying to allocate enough memory for the pipeline steps when we run them under Slurm, and some of them failed prelim_map because it used more than 3GB on a large sample. I ran prelim_map on a small sample and tracked the memory usage reported by `top`. It reported 100MB used by `bowtie2`, and 350MB used by the python process. The `bowtie2` memory was stable, but the python memory steadily climbed.\n\n* [ ] See what is being held in memory, and decide if it can be avoided.\n* [ ] Look at other steps and create separate issues if needed.\n* [ ] Look at steps in the Mixed HCV pipeline."},"comment":{"url":"https://api.github.com/repos/cfe-lab/MiCall/issues/comments/314576107","html_url":"https://github.com/cfe-lab/MiCall/issues/393#issuecomment-314576107","issue_url":"https://api.github.com/repos/cfe-lab/MiCall/issues/393","id":314576107,"user":{"login":"donkirkby","id":1639148,"avatar_url":"https://avatars2.githubusercontent.com/u/1639148?v=3","gravatar_id":"","url":"https://api.github.com/users/donkirkby","html_url":"https://github.com/donkirkby","followers_url":"https://api.github.com/users/donkirkby/followers","following_url":"https://api.github.com/users/donkirkby/following{/other_user}","gists_url":"https://api.github.com/users/donkirkby/gists{/gist_id}","starred_url":"https://api.github.com/users/donkirkby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/donkirkby/subscriptions","organizations_url":"https://api.github.com/users/donkirkby/orgs","repos_url":"https://api.github.com/users/donkirkby/repos","events_url":"https://api.github.com/users/donkirkby/events{/privacy}","received_events_url":"https://api.github.com/users/donkirkby/received_events","type":"User","site_admin":false},"created_at":"2017-07-11T21:22:35Z","updated_at":"2017-07-11T21:22:35Z","body":"Reduced memory usage in `aln2counts.py` by writing work in progress to file cache.\n\nHere's the memory that each sample uses, after the changes:\n* 88160A_HCV (87MB, 912s)\n* 88160AMIDI_MidHCV (91MB, 60s)\n* 69268A-V3-1_V3LOOP (66MB, 9s)\n* 69258A-V3_V3LOOP (51MB, 4s)\n\nUnsurprisingly, writing to disk is slower by up to a minute, but the memory usage now stays below 100MB."}}
{"actor":{"display_login":"donkirkby"}}