{"action":"created","issue":{"url":"https://api.github.com/repos/SciTools/iris/issues/2738","repository_url":"https://api.github.com/repos/SciTools/iris","labels_url":"https://api.github.com/repos/SciTools/iris/issues/2738/labels{/name}","comments_url":"https://api.github.com/repos/SciTools/iris/issues/2738/comments","events_url":"https://api.github.com/repos/SciTools/iris/issues/2738/events","html_url":"https://github.com/SciTools/iris/issues/2738","id":250627668,"number":2738,"title":"Fix lazy fill_value retrieval","user":{"login":"djkirkham","id":16189532,"avatar_url":"https://avatars0.githubusercontent.com/u/16189532?v=4","gravatar_id":"","url":"https://api.github.com/users/djkirkham","html_url":"https://github.com/djkirkham","followers_url":"https://api.github.com/users/djkirkham/followers","following_url":"https://api.github.com/users/djkirkham/following{/other_user}","gists_url":"https://api.github.com/users/djkirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/djkirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djkirkham/subscriptions","organizations_url":"https://api.github.com/users/djkirkham/orgs","repos_url":"https://api.github.com/users/djkirkham/repos","events_url":"https://api.github.com/users/djkirkham/events{/privacy}","received_events_url":"https://api.github.com/users/djkirkham/received_events","type":"User","site_admin":false},"labels":[{"id":658784970,"url":"https://api.github.com/repos/SciTools/iris/labels/dask-mask","name":"dask-mask","color":"5219e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-08-16T13:30:52Z","updated_at":"2017-08-18T07:51:43Z","closed_at":null,"body":"The \"smallest slice\" method of retrieving the fill value from a dask array doesn't always work. In particular it won't work if that array is formed from the concatenation/stacking of several arrays or proxies which have differing fill values, or combinations of ndarrays and masked arrays:\r\n\r\n```\r\n>>> import numpy as np\r\n>>> import dask.array as da\r\n>>> from iris._lazy_data import get_fill_value\r\n>>> a = da.from_array(np.array([1, 2, 3]), 1)\r\n>>> m = np.ma.masked_array([1, 2, 3], [0, 1, 0], fill_value=10)\r\n>>> m = da.from_array(m, 1, asarray=False)\r\n>>> am = da.concatenate([a, m])\r\n>>> print am.compute().fill_value\r\n10\r\n>>> print get_fill_value(am)\r\nNone\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/SciTools/iris/issues/comments/323285559","html_url":"https://github.com/SciTools/iris/issues/2738#issuecomment-323285559","issue_url":"https://api.github.com/repos/SciTools/iris/issues/2738","id":323285559,"user":{"login":"pelson","id":810663,"avatar_url":"https://avatars3.githubusercontent.com/u/810663?v=4","gravatar_id":"","url":"https://api.github.com/users/pelson","html_url":"https://github.com/pelson","followers_url":"https://api.github.com/users/pelson/followers","following_url":"https://api.github.com/users/pelson/following{/other_user}","gists_url":"https://api.github.com/users/pelson/gists{/gist_id}","starred_url":"https://api.github.com/users/pelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pelson/subscriptions","organizations_url":"https://api.github.com/users/pelson/orgs","repos_url":"https://api.github.com/users/pelson/repos","events_url":"https://api.github.com/users/pelson/events{/privacy}","received_events_url":"https://api.github.com/users/pelson/received_events","type":"User","site_admin":false},"created_at":"2017-08-18T07:51:43Z","updated_at":"2017-08-18T07:51:43Z","body":"Summary of key details:\r\n\r\n•\tnp.ma -> slice -> np.ma in all situations\r\n•\tda.ma -> slice -> da.ma in all situations\r\n•\tnp.ma -> save nc with fill value -> loaded array is ma only if there are masked points (fill value lost if no masked points)\r\n•\tThere is negligible performance difference between reading a variable with/without a fill_value if there are no actual masked points (this could certainly have *not* been the case)\r\n\r\n\r\n**My conclusion:** stream out our data with a default (overridable) fill value, warning if there are genuine data point collisions as we go. If we want round-trip fill_values, we need to add that to the data model (i.e. not use the fill_value in a ma, because of the previous comment). Adding fill_value to the data model is a possible thing - there are lots of details to ensure that we are consistent, and even then we still need to validate the arrays as they get written to ensure there are no data-point collisions. Implementing this has nothing to do with ```da.ma```, and I don't currently have a strong driver to actually *implement* round-trip ``fill_values`` (we don't round trip [other variable attributes](https://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/netcdf-f90/Attribute-Conventions.html), such as valid_range etc.)\r\n\r\n\r\n\r\n## Appendix\r\n```\r\nPerformance for reading 100,000,000 points from a NetCDF variable (no masked points):\r\n\r\nIn [7]: %timeit Dataset('test_w_fill_value.nc')['v'][:]\r\n1 loop, best of 3: 425 ms per loop\r\n\r\nIn [8]: %timeit Dataset('test_w_fill_value.nc')['v'][:]\r\n1 loop, best of 3: 425 ms per loop\r\n\r\nIn [9]: %timeit Dataset('test_no_fill_value.nc')['v'][:]\r\n1 loop, best of 3: 422 ms per loop\r\n\r\nIn [10]: %timeit Dataset('test_no_fill_value.nc')['v'][:]\r\n1 loop, best of 3: 422 ms per loop \r\n```"}}
{"actor":{"display_login":"pelson"}}