{"action":"created","issue":{"url":"https://api.github.com/repos/pangeo-data/pangeo/issues/48","repository_url":"https://api.github.com/repos/pangeo-data/pangeo","labels_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/48/comments","events_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/48/events","html_url":"https://github.com/pangeo-data/pangeo/issues/48","id":281146046,"number":48,"title":"workflow for moving data to cloud","user":{"login":"rabernat","id":1197350,"avatar_url":"https://avatars1.githubusercontent.com/u/1197350?v=4","gravatar_id":"","url":"https://api.github.com/users/rabernat","html_url":"https://github.com/rabernat","followers_url":"https://api.github.com/users/rabernat/followers","following_url":"https://api.github.com/users/rabernat/following{/other_user}","gists_url":"https://api.github.com/users/rabernat/gists{/gist_id}","starred_url":"https://api.github.com/users/rabernat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rabernat/subscriptions","organizations_url":"https://api.github.com/users/rabernat/orgs","repos_url":"https://api.github.com/users/rabernat/repos","events_url":"https://api.github.com/users/rabernat/events{/privacy}","received_events_url":"https://api.github.com/users/rabernat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2017-12-11T19:52:01Z","updated_at":"2017-12-20T18:56:52Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I am currently transferring a pretty large dataset (~11 TB) from a local server to gcs. \r\nHere is an abridged version basic workflow:\r\n```python\r\n# open dataset (about 80 x 133 GB netCDF files)\r\nds = xr.open_mfsdataset('*.nc', chunks={'time': 1, 'depth':1}) \r\n\r\n# configure gfs\r\nimport gcsfs\r\nconfig_json = '~/.config/gcloud/legacy_credentials/ryan.abernathey@gmail.com/adc.json'\r\nfs = gcsfs.GCSFileSystem(project='pangeo-181919', token=config_json)\r\nbucket = 'pangeo-data-private/path/to/data'\r\ngcsmap = gcsfs.mapping.GCSMap(bucket, gcs=fs, check=True, create=True)\r\n\r\n# set recommended compression\r\nimport zarr\r\ncompressor = zarr.Blosc(cname='zstd', clevel=5, shuffle=zarr.Blosc.AUTOSHUFFLE)\r\nencoding = {v: {'compressor': compressor} for v in ds.data_vars}\r\n\r\n# store\r\nds.to_zarr(store=gcsmap, mode='w', encoding=encoding)\r\n```\r\n\r\nEach chunk in the dataset has 2700 x 3600 elements (about 75 MB), and there are 292000 total chunks in the dataset.\r\n\r\nI am doing this through dask.distributed using a single, multi-threaded worker (24 threads). I am watching the progress through the dashboard.\r\n\r\nOnce I call `to_zarr`, it takes a long time before anything happens (about 1 hour). I can't figure out what dask is doing during this time. At some point the client errors with the following exception: `tornado.application - ERROR - Future <tornado.concurrent.Future object at 0x7fe371f58a58> exception was never retrieved`. Nevertheless, the computation eventually hits the scheduler, and I can watch its progress.\r\n\r\n![image](https://user-images.githubusercontent.com/1197350/33849791-abb76bee-de7f-11e7-96c1-2b21f709442c.png)\r\n\r\nI can see that there are over 1 million tasks. Most of the time is being spent in tasks called `open_dataset-concatenate` and `store-concatenate`. There are 315360 of each task, and each takes about ~20s. Doing the math, at this rate it will take a couple of days to upload the data, this is slower than scp by a factor of 2-5.\r\n\r\nI'm not sure if it's possible to do better. Just raising this issue to start a discussion.\r\n\r\nA command line utility to import netcdf directly to gcs/zarr would be a very useful tool to have."},"comment":{"url":"https://api.github.com/repos/pangeo-data/pangeo/issues/comments/353151480","html_url":"https://github.com/pangeo-data/pangeo/issues/48#issuecomment-353151480","issue_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/48","id":353151480,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2017-12-20T18:56:52Z","updated_at":"2017-12-20T18:56:52Z","author_association":"NONE","body":"Ah, I see that you're using a client.  We might want to avoid this when profiling.  Although the `dask.set_options` call *should* override that regardless"}}
{"actor":{"display_login":"mrocklin"}}