{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/2177","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/2177/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/2177/comments","events_url":"https://api.github.com/repos/dask/dask/issues/2177/events","html_url":"https://github.com/dask/dask/issues/2177","id":219663477,"number":2177,"title":"Removing the \"has_nulls\" option from to_parquet, or set default to True?","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars1.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-04-05T17:49:23Z","updated_at":"2017-04-07T18:26:26Z","closed_at":null,"body":"In dask.dataframe.io.to_parquet: \r\n\r\n```\r\n    has_nulls : bool, list or None\r\n        Specifies whether to write NULLs information for columns. If bools,\r\n        apply to all columns, if list, use for only the named columns, if None,\r\n        use only for columns which don't have a sentinel NULL marker (currently\r\n        object columns only).\r\n```\r\n\r\nI am looking in the logic in:\r\n\r\nhttps://github.com/dask/fastparquet/blob/master/fastparquet/writer.py#L626\r\n\r\nMy understand of this is that the default behavior is to _not_ write nulls in the Parquet definition levels (i.e. `RepetitionType:REQUIRED`) for float32/64 types. I'm concerned this is a dangerous default setting because the Parquet files written will not be readable in general by other systems that use Parquet files. Would you be open to removing this option altogether (respecting pandas's NULL semantics) or changing its default value to `True`?\r\n\r\ncc @martindurant "},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/292615053","html_url":"https://github.com/dask/dask/issues/2177#issuecomment-292615053","issue_url":"https://api.github.com/repos/dask/dask/issues/2177","id":292615053,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars0.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2017-04-07T18:26:26Z","updated_at":"2017-04-07T18:26:26Z","body":"I think that the following is a simple and motivational use case in favor of treating nans/nulls dilligently (what @wesm describes).  I haven't thought about this topic enough to have a strong opinion, but I think that this example might be useful.\r\n\r\nImagine you read the following CSV file\r\n\r\n```\r\na,b\r\n1,1\r\n2,2\r\n3,NA\r\n4,4\r\n```\r\n\r\nPandas treats the missing values here properly as missing values\r\n\r\n```python\r\n>>> df = pd.read_csv('myfile.csv')\r\n>>> df.b.sum()\r\n7\r\n```\r\n\r\nBut now we move the data to parquet and then read with Spark and try the same thing:\r\n\r\n```python\r\n>>> df.agg({'b': 'sum'})\r\nNaN\r\n```\r\n\r\nSo either Spark would need to interpret NaN's as Nulls (which I suspect is unlikely and probably incorrect from their perspective) or we are transforming the data so that it produces different results on different tools."}}
{"actor":{"display_login":"mrocklin"}}