{"action":"created","issue":{"url":"https://api.github.com/repos/datalad/datalad/issues/1169","repository_url":"https://api.github.com/repos/datalad/datalad","labels_url":"https://api.github.com/repos/datalad/datalad/issues/1169/labels{/name}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/1169/comments","events_url":"https://api.github.com/repos/datalad/datalad/issues/1169/events","html_url":"https://github.com/datalad/datalad/pull/1169","id":196663193,"number":1169,"title":"RF save/add/create","user":{"login":"mih","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false},"labels":[{"id":488743117,"url":"https://api.github.com/repos/datalad/datalad/labels/needs-review","name":"needs-review","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-12-20T12:40:49Z","updated_at":"2017-01-10T05:07:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/datalad/datalad/pulls/1169","html_url":"https://github.com/datalad/datalad/pull/1169","diff_url":"https://github.com/datalad/datalad/pull/1169.diff","patch_url":"https://github.com/datalad/datalad/pull/1169.patch"},"body":"High-level summary of the desired changes -- after staring two weeks at the code, trying to come up with something that makes more sense, is easier to test, and minimizes the places in the code where similar things happen:\r\n\r\n- `save` will no longer do half of the work of `add` -- it will only operate on content that was \"added\" by other means before. that being said, once the clean version is implemented we can and should reevaluate the scope again\r\n- `save` will no longer do half of the work of `uninstall` and `remove`. only paths to things that presently exist on the file system will be dealt with -- no convenience deinit of vanished submodules, no (possibly dangerous) `git rm` of vanished files anymore. maybe there will be a `fix-this-mess` command later on, and maybe it will triggered by `save`, but clean implementation first\r\n- \"surgical\" save: only path that we given explicitly (or implicitely via something like `all_changes=True` will be saved. Even if other changes were staged already.\r\n- three layers of functionality are envisioned:\r\n   1. toplevel API: throw it any number of paths in any number of datasets in any order and it will do the right thing with the respective datasets (flags will enable modes such as \"also save this one in the superdataset\", or \"record all changes to known components\")\r\n   2. intermediate: give it a dictionary with datasets to touch and the paths to safe in them, and it will figure out the order of processing, discover possibly required intermediate datasets to handle in order to fulfill a request and do the right thing\r\n   3. low: given a single dataset and a list of paths to save, it will save that content in this single dataset -- regardless of the type of content (file, directory, subdataset state)\r\n\r\nTo be addressed issues:\r\n- [x] #1175 (parts of)\r\n- [x] #1174 \r\n- [x] #1173 \r\n- [x] #1172 \r\n- [x] #1141 \r\n- [x] #1140 \r\n- [x] #1109 \r\n- [x] #1106 (50% of it)\r\n- [x] #1096 \r\n- [x] #1004 \r\n"},"comment":{"url":"https://api.github.com/repos/datalad/datalad/issues/comments/271487843","html_url":"https://github.com/datalad/datalad/pull/1169#issuecomment-271487843","issue_url":"https://api.github.com/repos/datalad/datalad/issues/1169","id":271487843,"user":{"login":"bpoldrack","id":10498301,"avatar_url":"https://avatars.githubusercontent.com/u/10498301?v=3","gravatar_id":"","url":"https://api.github.com/users/bpoldrack","html_url":"https://github.com/bpoldrack","followers_url":"https://api.github.com/users/bpoldrack/followers","following_url":"https://api.github.com/users/bpoldrack/following{/other_user}","gists_url":"https://api.github.com/users/bpoldrack/gists{/gist_id}","starred_url":"https://api.github.com/users/bpoldrack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bpoldrack/subscriptions","organizations_url":"https://api.github.com/users/bpoldrack/orgs","repos_url":"https://api.github.com/users/bpoldrack/repos","events_url":"https://api.github.com/users/bpoldrack/events{/privacy}","received_events_url":"https://api.github.com/users/bpoldrack/received_events","type":"User","site_admin":false},"created_at":"2017-01-10T05:07:46Z","updated_at":"2017-01-10T05:07:46Z","body":"Tracked down the issue to the following for now:\r\n\r\nIt's the `annex add` in interface/utils.py line 327, which doesn't work correctly in Travis' environment.\r\nThis leads to things not being staged, therefore not committed and that's why the tests are failing.\r\nAs @mih 's comment on that code states, it is supposed to work with new annex. Travis uses `git-annex-standlone` from 1/1/17, but git is 1.8.5.6 only. I guess, that's where things are going wrong.\r\n\r\nHere is an example snippet of the output running `nosetests -s -v datalad/distribution/tests/test_uninstall.py:test_uninstall_recursive` with extensive logging. First locally, second on Travis:\r\n\r\nlocal:\r\n```\r\ndatalad.interface.utils: DEBUG  : ...>case:151,393,329,197>utils:401>test_uninstall:295>dataset:518>uninstall:434>utils:267,325  staging files for commit: ['/tmp/datalad_temp_tree_test_uninstall_recursiveXwLE6A/deep']\r\ndatalad.cmd: DEBUG  : ...>annexrepo:683>gitrepo:290>annexrepo:676>gitrepo:290,660,290,1135>cmd:455,311,423  Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', 'deep']\r\ndatalad.cmd: DEBUG  : ...>uninstall:434>utils:267,329>gitrepo:290>annexrepo:707,1212,391>cmd:455,311,423  Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--debug', '--json', 'deep']\r\ndatalad.gitrepo: DEBUG  : ...>case:329,197>utils:401>test_uninstall:295>dataset:518>uninstall:434>utils:267,357>annexrepo:1470>gitrepo:791  Committing via direct call of git: ['git', 'commit', '-m', '[DATALAD] removed content', '/tmp/datalad_temp_tree_test_uninstall_recursiveXwLE6A/deep']\r\ndatalad.cmd: DEBUG  : ...>uninstall:434>utils:267,357>annexrepo:1470>gitrepo:792,290,1135>cmd:455,311,423  Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'commit', '-m', '[DATALAD] removed content', '/tmp/datalad_temp_tree_test_uninstall_recursiveXwLE6A/deep']\r\n```\r\n\r\nTravis:\r\n```\r\ndatalad.interface.utils: DEBUG: ...>utils:267,325>handlers:78  staging files for commit: ['/tmp/datalad_temp_tree_test_uninstall_recursiveAjpQj7/deep']\r\ndatalad.cmd: DEBUG: ...>utils:267,329>gitrepo:290>annexrepo:683>gitrepo:290>annexrepo:676>gitrepo:290,660,290,1135>cmd:455,311,423>handlers:78  Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', 'deep']\r\ndatalad.cmd: Level 8: ...>cmd:455,373,423>handlers:78  Finished running ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', 'deep'] with status 0\r\ndatalad.cmd: DEBUG: ...>annexrepo:707,1212,391>cmd:455,311,423>handlers:78  Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--debug', '--json', 'deep']\r\ndatalad.cmd: DEBUG: ...>cmd:455,364,152,423>handlers:78  stderr| [2017-01-10 02:29:27.550042962] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--others\",\"--exclude-standard\",\"-z\",\"--\",\"deep\"]\r\n| [2017-01-10 02:29:27.551940515] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--modified\",\"-z\",\"--\",\"deep\"]\r\n| [2017-01-10 02:29:27.553641879] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"diff\",\"--name-only\",\"--diff-filter=T\",\"-z\",\"--\",\"deep\"]\r\ndatalad.cmd: Level 8: ...>cmd:455,373,423>handlers:78  Finished running ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--debug', '--json', 'deep'] with status 0\r\ndatalad.gitrepo: DEBUG: ...>utils:267,357>annexrepo:1470>gitrepo:791>handlers:78  Committing via direct call of git: ['git', 'commit', '-m', '[DATALAD] removed content', '/tmp/datalad_temp_tree_test_uninstall_recursiveAjpQj7/deep']\r\ndatalad.cmd: DEBUG: ...>gitrepo:792,290,1135>cmd:455,311,423>handlers:78  Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'commit', '-m', '[DATALAD] removed content', '/tmp/datalad_temp_tree_test_uninstall_recursiveAjpQj7/deep']\r\ndatalad.cmd: DEBUG: ...>cmd:455,359,146,423>handlers:78  stdout| # On branch master\r\n| # Changes not staged for commit:\r\n| #\tmodified:   deep (modified content)\r\n| #\r\n| no changes added to commit\r\ndatalad.cmd: ERROR: ...>cmd:455,369>handlers:78  Failed to run ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'commit', '-m', '[DATALAD] removed content', '/tmp/datalad_temp_tree_test_uninstall_recursiveAjpQj7/deep'] under '/tmp/datalad_temp_tree_test_uninstall_recursiveAjpQj7'. Exit code=1. out=# On branch master\r\n| # Changes not staged for commit:\r\n| #\tmodified:   deep (modified content)\r\n| #\r\n| no changes added to commit\r\n|  err=\r\ndatalad.interface.utils: INFO: ...>utils:267,367>handlers:78  Nothing to save\r\n\r\n```\r\n\r\n"}}
{"actor":{"display_login":"bpoldrack"}}