{"action":"created","issue":{"url":"https://api.github.com/repos/MrOlm/drep/issues/8","repository_url":"https://api.github.com/repos/MrOlm/drep","labels_url":"https://api.github.com/repos/MrOlm/drep/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/MrOlm/drep/issues/8/comments","events_url":"https://api.github.com/repos/MrOlm/drep/issues/8/events","html_url":"https://github.com/MrOlm/drep/issues/8","id":221802091,"number":8,"title":"Pair wise mash clustering issue","user":{"login":"shaman-narayanasamy","id":4169854,"avatar_url":"https://avatars1.githubusercontent.com/u/4169854?v=3","gravatar_id":"","url":"https://api.github.com/users/shaman-narayanasamy","html_url":"https://github.com/shaman-narayanasamy","followers_url":"https://api.github.com/users/shaman-narayanasamy/followers","following_url":"https://api.github.com/users/shaman-narayanasamy/following{/other_user}","gists_url":"https://api.github.com/users/shaman-narayanasamy/gists{/gist_id}","starred_url":"https://api.github.com/users/shaman-narayanasamy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shaman-narayanasamy/subscriptions","organizations_url":"https://api.github.com/users/shaman-narayanasamy/orgs","repos_url":"https://api.github.com/users/shaman-narayanasamy/repos","events_url":"https://api.github.com/users/shaman-narayanasamy/events{/privacy}","received_events_url":"https://api.github.com/users/shaman-narayanasamy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-04-14T12:40:00Z","updated_at":"2017-04-18T13:02:11Z","closed_at":null,"body":"Dear Matt,\r\n\r\nMe again :) \r\n\r\nI was trying to see if dRep was behaving weirdly due to the datasets/environment. Therefore, I copied the genomes provided within dRep for testing purposes into my input folder and proceeded to run the following command:\r\n\r\n`$ dRep dereplicate_wf -p 4 --completeness 0.6 --strain_htr 101 --P_ani 0.6 --S_ani 0.965 --overwrite         --run_tax /scratch/users/snarayanasamy/LAO_TS_IMP-v1.3/TemporalBinning/RepresentativeBins -g /scratch/users/snarayanasamy/LAO_TS_IMP-v1.3/TemporalBinning/AllBins/*.fa`\r\n\r\nIt fails on the clustering step for some reason:\r\n\r\n```\r\nStep 1. Parse Arguments\r\nStep 2. Perform MASH (primary) clustering\r\n2a. Run pair-wise MASH clustering\r\n[====================] 100.00%\r\n2b. Cluster pair-wise MASH clustering\r\nTraceback (most recent call last):\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/env3.4.3/bin/dRep\", line 26, in <module>\r\n    controller.parseArguments(args)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/controller.py\", line 144, in parseArguments\r\n    self.dereplicate_wf_operation(**vars(args))\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/controller.py\", line 86, in dereplicate_wf_operation\r\n    drep.d_workflows.dereplicate_wrapper(kwargs['work_directory'],**kwargs)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/d_workflows.py\", line 36, in dereplicate_wrapper\r\n    drep.d_cluster.d_cluster_wrapper(wd, **kwargs)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/d_cluster.py\", line 287, in d_cluster_wrapper\r\n    Cdb, Mdb, Ndb = cluster_genomes(Bdb, data_folder, **kwargs)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/d_cluster.py\", line 106, in cluster_genomes\r\n    Cdb = cluster_mash_database(Mdb, data_folder= data_folder, **kwargs)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/d_cluster.py\", line 662, in cluster_mash_database\r\n    linkage_cutoff= P_Lcutoff)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/drep/d_cluster.py\", line 355, in cluster_hierarchical\r\n    linkage = scipy.cluster.hierarchy.linkage(arr, method= linkage_method)\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/scipy/cluster/hierarchy.py\", line 660, in linkage\r\n    n = int(distance.num_obs_y(y))\r\n  File \"/mnt/nfs/projects/ecosystem_biology/local_tools/pyenv/versions/3.4.3/envs/env3.4.3/lib/python3.4/site-packages/scipy/spatial/distance.py\", line 1718, in num_obs_y\r\n    raise ValueError(\"The number of observations cannot be determined on \"\r\nValueError: The number of observations cannot be determined on an empty distance matrix.\r\n```\r\n\r\nAny idea if this is an issue with my environment. \r\n\r\nPlease know that dRep is working very well for our data sets. It is just that we are finding some inconsistencies from time to time and would like to report them to be fixed. We are looking to keep using dRep on more of our data. Looking forward to hear from you. \r\n\r\nCheers,\r\nShaman"},"comment":{"url":"https://api.github.com/repos/MrOlm/drep/issues/comments/294833302","html_url":"https://github.com/MrOlm/drep/issues/8#issuecomment-294833302","issue_url":"https://api.github.com/repos/MrOlm/drep/issues/8","id":294833302,"user":{"login":"brooksomics","id":8507447,"avatar_url":"https://avatars1.githubusercontent.com/u/8507447?v=3","gravatar_id":"","url":"https://api.github.com/users/brooksomics","html_url":"https://github.com/brooksomics","followers_url":"https://api.github.com/users/brooksomics/followers","following_url":"https://api.github.com/users/brooksomics/following{/other_user}","gists_url":"https://api.github.com/users/brooksomics/gists{/gist_id}","starred_url":"https://api.github.com/users/brooksomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brooksomics/subscriptions","organizations_url":"https://api.github.com/users/brooksomics/orgs","repos_url":"https://api.github.com/users/brooksomics/repos","events_url":"https://api.github.com/users/brooksomics/events{/privacy}","received_events_url":"https://api.github.com/users/brooksomics/received_events","type":"User","site_admin":false},"created_at":"2017-04-18T13:02:11Z","updated_at":"2017-04-18T13:02:11Z","body":"Hey Matt,\r\n\r\nI'm having some trouble as well running dRep in my new environment.\r\n\r\nI tried the test suite as you suggested to Shaman.\r\n\r\n````\r\n$ mkdir test_backend\r\n$ ls ../tests/\r\ngenomes  test_backend  test_solutions  test_suite.py\r\n$ ./test_suite.py\r\nStep 1. Parse Arguments\r\nStep 2. Perform MASH (primary) clustering\r\n2a. Run pair-wise MASH clustering\r\n2b. Cluster pair-wise MASH clustering\r\n3 primary clusters made\r\nStep 3. Perform secondary clustering\r\nRunning 11 ANIn comparisons- should take ~ 0.6 min\r\nStep 4. Return output\r\nTraceback (most recent call last):\r\n  File \"./test_suite.py\", line 466, in <module>\r\n    test_quick()\r\n  File \"./test_suite.py\", line 462, in test_quick\r\n    rerun_test()\r\n  File \"./test_suite.py\", line 446, in rerun_test\r\n    QuickTests().run()\r\n  File \"./test_suite.py\", line 292, in run\r\n    self.unit_tests_1()\r\n  File \"./test_suite.py\", line 325, in unit_tests_1\r\n    assert db1.equals(db2), \"{0} is not the same!\".format(db)\r\nAssertionError: Ndb is not the same!\r\n````\r\nSo I see it complaining about Ndb. I'm using dRep v0.5.5 and according to tests/test_solutions/log/logger.log the most recent test_suite was run on v0.5.4. So maybe there are some old dataframes that need refreshing?\r\n\r\nSo I reran in a new directory with the provided genomes:\r\n```\r\ndRep dereplicate_wf . -g genomes/* -o -p 16 --run_tax --S_algorithm gANI -sa 0.99 --cent_index /home/ubuntu/scripts/github/centrifuge/indices/b+h+v &> drep.log &\r\n```\r\nEverything works! So perhaps updating your test_solutions/ dir is the answer here?\r\n\r\nMy problem is unrelated - so i'll start a new thread."}}
{"actor":{"display_login":"brooksomics"}}