{"push_id":2184331567,"size":23,"distinct_size":23,"ref":"refs/heads/dev","head":"e198dd75782ce5b2a9619a5283e77a026393e149","before":"5ad4f57a63b00cf6f27a6852e13d87d7902cac4a","commits":[{"sha":"3fa608f54850d9a93e910cdc55fa1f87c4c87c7c","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Use pysam.merge instead of samtools merge","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/3fa608f54850d9a93e910cdc55fa1f87c4c87c7c"},{"sha":"16d0d3ecafb47ca8f112e8fa339f492446f9e80c","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Add unsorted test bam","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/16d0d3ecafb47ca8f112e8fa339f492446f9e80c"},{"sha":"3c9999d89acfdd7aa1f56969ca7e03da3d83e655","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Use pysam for metadata setting and grooming\n\nThere are also some noteworthy changes here:\n  - We do always respect the sort-order specified in the header\n  - If the sort order is not mentioned in the header or no header\n    exists we coordinate-sort the file.\n  - We do not use indexing to determine if a file is coordinate sorted,\n    because this does not work reliably with samtools/pysam > 1.X,\n    since arbitrarily sorted files can be indexed now.\n\nThis also fixes advanced metadata setting (sort_order, bam_version and more),\nwhich appears to have been broken. This probably went by unnoticed because of the catch-all\ntry-except-pass.\nThe downside to fixing this is that I had to (temporarilly, hopefully)\ncomment out the reference_names, reference_lengths, bam_header and readgroups attributes,\nbecause they led to the following error:\n\n```\ngalaxy.model.metadata DEBUG 2017-11-19 14:47:30,540 loading metadata from file for: HistoryDatasetAssociation 582\ngalaxy.jobs.runners.local ERROR 2017-11-19 14:47:30,636 Job wrapper finish method failed\nTraceback (most recent call last):\n  File \"/Users/mvandenb/src/galaxy/lib/galaxy/jobs/runners/local.py\", line 130, in queue_job\n    job_wrapper.finish(stdout, stderr, exit_code)\n  File \"/Users/mvandenb/src/galaxy/lib/galaxy/jobs/__init__.py\", line 1357, in finish\n    self.sa_session.flush()\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py\", line 157, in do\n    return getattr(self.registry(), name)(*args, **kwargs)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2019, in flush\n    self._flush(objects)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2137, in _flush\n    transaction.rollback(_capture_exception=True)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py\", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2101, in _flush\n    flush_context.execute()\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 373, in execute\n    rec.execute(self)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 532, in execute\n    uow\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 170, in save_obj\n    mapper, table, update)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 706, in _emit_update_statements\n    execute(statement, multiparams)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 914, in execute\n    return meth(self, multiparams, params)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py\", line 323, in _execute_on_connection\n    return connection._execute_clauseelement(self, multiparams, params)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1010, in _execute_clauseelement\n    compiled_sql, distilled_params\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1146, in _execute_context\n    context)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1341, in _handle_dbapi_exception\n    exc_info\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py\", line 202, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb, cause=cause)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1139, in _execute_context\n    context)\n  File \"/Users/mvandenb/src/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py\", line 450, in do_execute\n    cursor.execute(statement, parameters)\nOperationalError: (psycopg2.OperationalError) index row size 7208 exceeds maximum 2712 for index \"ix_history_dataset_association_metadata\"\nHINT:  Values larger than 1/3 of a buffer page cannot be indexed.\nConsider a function index of an MD5 hash of the value, or use full text indexing.\n [SQL: 'UPDATE history_dataset_association SET update_time=%(update_time)s, blurb=%(blurb)s, peek=%(peek)s, metadata=%(_metadata)s WHERE history_dataset_association.id = %(history_dataset_association_id)s'] [parameters: {'_metadata': <psycopg2.extensions.Binary object at 0x119ed8990>, 'update_time': datetime.datetime(2017, 11, 19, 13, 47, 30, 598058), 'history_dataset_association_id': 582, 'blurb': '3.5 KB', 'peek': 'Binary bam alignments file'}]\n```","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/3c9999d89acfdd7aa1f56969ca7e03da3d83e655"},{"sha":"369485a43ad50a877dc68324c1a2b6ab876030ea","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Move Bam doctests to unittests","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/369485a43ad50a877dc68324c1a2b6ab876030ea"},{"sha":"ae72e5668364f6f72861c21a62a641f9f65636e1","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Use subprocess to check if pysam.index succeeds\n\nChecking if pysam.index succeeds tests whether a file is coordinate sorted.\nIf pysam.index fails to index htslib writes to stderr and this fails set meta\ntool. To prevent this we run this in a subprocess and discard stderr.","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/ae72e5668364f6f72861c21a62a641f9f65636e1"},{"sha":"836aea072e84411fb4a24e37fb306b5ec72044c6","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Add test file with random order","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/836aea072e84411fb4a24e37fb306b5ec72044c6"},{"sha":"bac56d6b27068d3744e84eb9756bc28bf392f718","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Drop samtools from metadata and upload tools\n\nWe only need samtools for the dataproviders, which shouldn't\nbe used by these tools.","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/bac56d6b27068d3744e84eb9756bc28bf392f718"},{"sha":"df35d089e3ea8c15330afaeedfa24e9c2f0fa7b2","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Add more unittests for CRAM, bcf and vcf","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/df35d089e3ea8c15330afaeedfa24e9c2f0fa7b2"},{"sha":"3f6a59e85ff8acf2fb0b39670f450a523ea11f38","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Replace bcftools index with pysam.bcftools.index or pysam.tabix_index","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/3f6a59e85ff8acf2fb0b39670f450a523ea11f38"},{"sha":"8e580ec7ac599993ae082732b5c80f2b9bcce493","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Metadata fixes for VcfGz","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/8e580ec7ac599993ae082732b5c80f2b9bcce493"},{"sha":"0ac77ad7fd5c6d6660521d69f6010cb61d4947ac","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Fix Vcf index generation and test","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/0ac77ad7fd5c6d6660521d69f6010cb61d4947ac"},{"sha":"6ca896ff9b199ac097f408e35e37a3365ff2ec46","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Fix index file syntax and fix linting problems","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/6ca896ff9b199ac097f408e35e37a3365ff2ec46"},{"sha":"9c4a2f70ea6a62f35b3c736730fcc71bddf6dce9","author":{"name":"Dannon Baker","email":"ae8fa834c2ffc5729255b1f9cfd68047d7325e2e@gmail.com"},"message":"Tabix indexing fixes.  Upstream used 'index' instead of 'index_filename', and 'force' is now required since we precreate the destination location.","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/9c4a2f70ea6a62f35b3c736730fcc71bddf6dce9"},{"sha":"2a6e00dd001ea4f27cd7d9a12cddfcf9649da7ef","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"one more linting fix","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/2a6e00dd001ea4f27cd7d9a12cddfcf9649da7ef"},{"sha":"07158e03308e026ee013bd6a50708167918c2d27","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Need force=True in pysam.tabix_index because the index file path exists in the object store","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/07158e03308e026ee013bd6a50708167918c2d27"},{"sha":"64deb4f935ac7969a0e81024297d84d9831a8cfd","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"We can use multiple threads when converting cram to bam files","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/64deb4f935ac7969a0e81024297d84d9831a8cfd"},{"sha":"90a56aea3f19df6943dfc648474d161e5c8f6d66","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Use more threads when sorting BAM files and GALAXY_SLOTS is set","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/90a56aea3f19df6943dfc648474d161e5c8f6d66"},{"sha":"eb636c11361ef8f6cf85c8750f0a37608dcbee1a","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Make sure files don't change in-place","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/eb636c11361ef8f6cf85c8750f0a37608dcbee1a"},{"sha":"1195183ea63c3916e1b8e163b2cc0839bd18d8c4","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Symlink tbi index to work around pysam limitation\n\nBefore https://github.com/pysam-developers/pysam/pull/586 is merged and a new release is out\nwe create a symlink to the tbi file, which is required for creating TabixFile\ninstances. Since we want to cleanup the symlinks I turned `get_data_file` into a contextmanager.\nAlong the way I also changed many open()/close() calls to `with` statements.","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/1195183ea63c3916e1b8e163b2cc0839bd18d8c4"},{"sha":"4795e490636e9a9234cb2d61c0a14508472a297d","author":{"name":"mvdbeek","email":"956c6368fd22c4d37e64fb8545c7be9ca0132de1@gmail.com"},"message":"Detect if index symlink hack is necessary","distinct":true,"url":"https://api.github.com/repos/galaxyproject/galaxy/commits/4795e490636e9a9234cb2d61c0a14508472a297d"}]}
{"actor":{"display_login":"nsoranzo"}}