{"action":"created","issue":{"url":"https://api.github.com/repos/dask/hdfs3/issues/124","repository_url":"https://api.github.com/repos/dask/hdfs3","labels_url":"https://api.github.com/repos/dask/hdfs3/issues/124/labels{/name}","comments_url":"https://api.github.com/repos/dask/hdfs3/issues/124/comments","events_url":"https://api.github.com/repos/dask/hdfs3/issues/124/events","html_url":"https://github.com/dask/hdfs3/issues/124","id":243952621,"number":124,"title":"Provide a function to recursively put a directory with subdirectories to HDFS","user":{"login":"FlorianWilhelm","id":3184072,"avatar_url":"https://avatars2.githubusercontent.com/u/3184072?v=4","gravatar_id":"","url":"https://api.github.com/users/FlorianWilhelm","html_url":"https://github.com/FlorianWilhelm","followers_url":"https://api.github.com/users/FlorianWilhelm/followers","following_url":"https://api.github.com/users/FlorianWilhelm/following{/other_user}","gists_url":"https://api.github.com/users/FlorianWilhelm/gists{/gist_id}","starred_url":"https://api.github.com/users/FlorianWilhelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FlorianWilhelm/subscriptions","organizations_url":"https://api.github.com/users/FlorianWilhelm/orgs","repos_url":"https://api.github.com/users/FlorianWilhelm/repos","events_url":"https://api.github.com/users/FlorianWilhelm/events{/privacy}","received_events_url":"https://api.github.com/users/FlorianWilhelm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-07-19T07:54:49Z","updated_at":"2017-07-19T13:09:56Z","closed_at":null,"body":"Right now it is only possible to recursively delete a directory. It would be nice if hdfs3 came with a function out of the box that allows recursively pushing a directory to HDFS.\r\n\r\nA naive implementation would be something like:\r\n```python\r\nclass cd(object):\r\n    \"\"\"Context manager for changing the current working directory\"\"\"\r\n    def __init__(self, new_path):\r\n        self.new_path = os.path.expanduser(new_path)\r\n\r\n    def __enter__(self):\r\n        self.old_path = os.getcwd()\r\n        os.chdir(self.new_path)\r\n\r\n    def __exit__(self, etype, value, traceback):\r\n        os.chdir(self.old_path)\r\n\r\ndef put_dir(hdfs_client, origin_path, destination_path):\r\n    \"\"\"Recursively push a directory to HDFS\r\n\r\n    Args:\r\n        hdfs_client: hdfs client\r\n        origin_path: origin path\r\n        destination_path: destination path on HDFS\r\n    \"\"\"\r\n    hdfs_client.mkdir(destination_path)\r\n    with cd(origin_path):\r\n        for root, dirs, files in os.walk('./'):\r\n            dest_path = os.path.join(destination_path, root)\r\n            for dir in dirs:\r\n                hdfs_client.mkdir(os.path.join(dest_path, dir))\r\n            for file in files:\r\n                file_path = os.path.join(root, file)\r\n                hdfs_client.put(file_path, os.path.join(dest_path, file))\r\n```\r\nBut this is pretty slow in practice. "},"comment":{"url":"https://api.github.com/repos/dask/hdfs3/issues/comments/316380913","html_url":"https://github.com/dask/hdfs3/issues/124#issuecomment-316380913","issue_url":"https://api.github.com/repos/dask/hdfs3/issues/124","id":316380913,"user":{"login":"martindurant","id":6042212,"avatar_url":"https://avatars1.githubusercontent.com/u/6042212?v=4","gravatar_id":"","url":"https://api.github.com/users/martindurant","html_url":"https://github.com/martindurant","followers_url":"https://api.github.com/users/martindurant/followers","following_url":"https://api.github.com/users/martindurant/following{/other_user}","gists_url":"https://api.github.com/users/martindurant/gists{/gist_id}","starred_url":"https://api.github.com/users/martindurant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martindurant/subscriptions","organizations_url":"https://api.github.com/users/martindurant/orgs","repos_url":"https://api.github.com/users/martindurant/repos","events_url":"https://api.github.com/users/martindurant/events{/privacy}","received_events_url":"https://api.github.com/users/martindurant/received_events","type":"User","site_admin":false},"created_at":"2017-07-19T13:09:56Z","updated_at":"2017-07-19T13:09:56Z","body":"This seems like a thing we can certainly do.\r\nI would use `os.walk` with the `origin_path` rather than changing the working directory, and I would make this a method on the HDFileSystem rather than a separate function. Would you care to make these changes and submit a PR, with tests? It would be nice to have the reverse method for fetching a set of directories to the local disc.\r\n\r\nFor further thought: the code would be essentially the same with s3fs or gcsfs, and could be implemented in parallel with dask to make a full `distcp`-like clone."}}
{"actor":{"display_login":"martindurant"}}