{"action":"created","issue":{"url":"https://api.github.com/repos/dg13/rasqual/issues/9","repository_url":"https://api.github.com/repos/dg13/rasqual","labels_url":"https://api.github.com/repos/dg13/rasqual/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/dg13/rasqual/issues/9/comments","events_url":"https://api.github.com/repos/dg13/rasqual/issues/9/events","html_url":"https://github.com/dg13/rasqual/pull/9","id":279211816,"number":9,"title":"Improve create asvcf","user":{"login":"ghuls","id":1299177,"avatar_url":"https://avatars2.githubusercontent.com/u/1299177?v=4","gravatar_id":"","url":"https://api.github.com/users/ghuls","html_url":"https://github.com/ghuls","followers_url":"https://api.github.com/users/ghuls/followers","following_url":"https://api.github.com/users/ghuls/following{/other_user}","gists_url":"https://api.github.com/users/ghuls/gists{/gist_id}","starred_url":"https://api.github.com/users/ghuls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghuls/subscriptions","organizations_url":"https://api.github.com/users/ghuls/orgs","repos_url":"https://api.github.com/users/ghuls/repos","events_url":"https://api.github.com/users/ghuls/events{/privacy}","received_events_url":"https://api.github.com/users/ghuls/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-12-05T01:23:43Z","updated_at":"2017-12-06T14:40:14Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/dg13/rasqual/pulls/9","html_url":"https://github.com/dg13/rasqual/pull/9","diff_url":"https://github.com/dg13/rasqual/pull/9.diff","patch_url":"https://github.com/dg13/rasqual/pull/9.patch"},"body":"This pull request fixes some small issues (source code files with executable bits set, improved usage and error messages) and a big issue with createASVCF.sh (spawning parseCigar milion times).\r\n\r\nThe most important commit of this pull request:\r\n```\r\n    Rewrite createASVCF.sh and make it more than 1000 times faster .\r\n\r\n    Rewrite createASVCF.sh:\r\n      - Make createASVCF.sh more than 1000 times faster, by rewriting:\r\n\r\n            samtools view -F 0x0100 $I $CHR:$START-$END \\\r\n                | awk -v RASQUALDIR=$RASQUALDIR '$7==\"=\"{\r\n                            cmd = RASQUALDIR\"/src/ASVCF/parseCigar \"$6;\r\n                            cmd | getline N; print $3\"\\t\"$4\"\\t\"$4+N-1\"\\t\"$6\"\\t\"$10;\r\n                            close(cmd);}' \\\r\n                | $RASQUALDIR/src/ASVCF/countAS $TMP.gz \\\r\n                | awk '{print $5\",\"$6}' \\\r\n                | gzip >> ${I%.bam}.$ID.gz\r\n\r\n        The problem with this code is that awk will spawn the\r\n        parseCigar command for each line. This is very slow\r\n        (can take a week or more) when the BAM files contain\r\n        milions of mapped reads as parseCigar will be spawned\r\n        milions of times to parse a very small string.\r\n\r\n        The new code implements the function of parseCigar directly in awk\r\n        and does not spawn any child processes and speeds up createASVCF.sh\r\n        tremendously.\r\n\r\n        The new code also handles BAM files with single-end reads instead of\r\n        only BAM files with paired-end reads. If \"single_end\" is specified,\r\n        reads are not filtered out if their mate chromosome is unknown (as\r\n        they don't have a mate).\r\n\r\n      - Use \"mktemp\" instead of deprecated \"tempfile\" and\r\n        write all temp files to a temp directory instead of\r\n        to the directory the BAM files are located.\r\n        \"tempfile\" is also not always available, while \"mktemp\"\r\n        is part of the \"coreutils\" package.\r\n\r\n      - If ${TMPDIR} is set from outside the script, it will use\r\n        that directory as base directory for the tempory directory.\r\n\r\n      - Add usage information and error messages when run with not\r\n        enough arguments or invalid arguments.\r\n\r\n      - Use indexed bash arrays for storing BAM and AS filenames,\r\n        so the BAM list filename does not need to be reread every\r\n        time you want to loop over the files.\r\n\r\n        It also allows to write some code more cleanly:\r\n\r\n            \"${RASQUALDIR}/src/ASVCF/zpaste\" \"${AS_FILES[@]}\" > \"${ALL_ASC_FILENAME}\";\r\n\r\n        instead of:\r\n\r\n            echo -n $RASQUALDIR/src/ASVCF/zpaste > $PASTE\r\n            for I in `cat $BAMLIST`\r\n            do\r\n            echo -n \" \"${I%.bam}.$ID.gz >> $PASTE\r\n            done\r\n            sh $PASTE > $ALLASC\r\n```\r\n\r\nWe let **createASVCF.sh** run for 5 days (30 BAM files) and it was still not finished.\r\nAfter this rewrite, it took less than 20 minutes."},"comment":{"url":"https://api.github.com/repos/dg13/rasqual/issues/comments/349658791","html_url":"https://github.com/dg13/rasqual/pull/9#issuecomment-349658791","issue_url":"https://api.github.com/repos/dg13/rasqual/issues/9","id":349658791,"user":{"login":"ghuls","id":1299177,"avatar_url":"https://avatars2.githubusercontent.com/u/1299177?v=4","gravatar_id":"","url":"https://api.github.com/users/ghuls","html_url":"https://github.com/ghuls","followers_url":"https://api.github.com/users/ghuls/followers","following_url":"https://api.github.com/users/ghuls/following{/other_user}","gists_url":"https://api.github.com/users/ghuls/gists{/gist_id}","starred_url":"https://api.github.com/users/ghuls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghuls/subscriptions","organizations_url":"https://api.github.com/users/ghuls/orgs","repos_url":"https://api.github.com/users/ghuls/repos","events_url":"https://api.github.com/users/ghuls/events{/privacy}","received_events_url":"https://api.github.com/users/ghuls/received_events","type":"User","site_admin":false},"created_at":"2017-12-06T14:40:14Z","updated_at":"2017-12-06T14:40:14Z","author_association":"NONE","body":"@natsuhiko \r\nDo you have time to take a look."}}
{"actor":{"display_login":"ghuls"}}