{"action":"opened","issue":{"url":"https://api.github.com/repos/dask/dask/issues/2839","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/2839/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/2839/comments","events_url":"https://api.github.com/repos/dask/dask/issues/2839/events","html_url":"https://github.com/dask/dask/issues/2839","id":269407705,"number":2839,"title":"ddf read using read_csv report incorrect partition data","user":{"login":"littlegreenbean33","id":18648417,"avatar_url":"https://avatars2.githubusercontent.com/u/18648417?v=4","gravatar_id":"","url":"https://api.github.com/users/littlegreenbean33","html_url":"https://github.com/littlegreenbean33","followers_url":"https://api.github.com/users/littlegreenbean33/followers","following_url":"https://api.github.com/users/littlegreenbean33/following{/other_user}","gists_url":"https://api.github.com/users/littlegreenbean33/gists{/gist_id}","starred_url":"https://api.github.com/users/littlegreenbean33/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/littlegreenbean33/subscriptions","organizations_url":"https://api.github.com/users/littlegreenbean33/orgs","repos_url":"https://api.github.com/users/littlegreenbean33/repos","events_url":"https://api.github.com/users/littlegreenbean33/events{/privacy}","received_events_url":"https://api.github.com/users/littlegreenbean33/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-10-29T15:26:06Z","updated_at":"2017-10-29T15:26:06Z","closed_at":null,"author_association":"NONE","body":"\r\nPlease see summary of issue in this stackoveflow question [dask set_index, divisions. How to set up correctly?](https://stackoverflow.com/questions/47000069/dask-set-index-divisions-how-to-set-up-correctly)\r\n\r\nI was struggling trying to index a dataframe after reading a csv file.  Digging I realize that the result from get_partition was incorrect. As a consequence, I realized all mis problems with indexing were really related to partitions management.\r\n\r\n    ddf=dd.read_csv([\"tmpaa\"],names=[\"idx\",\"date\",\"a\",\"b\",\"c\",\"d\",\"e\",\"f\",\"g\"])\r\n\r\n    In [41]: ddf.head(3)\r\n    Out[41]: \r\n       idx        date   a  b   c  d   e   f             g\r\n    0    0  2015-04-27   1  1   0  0   0   2     97.411003\r\n    1    1  2016-04-20   2  0   0  0   0   1     56.868000\r\n    2    2  2016-09-09  42  5  11  1  54  58  16826.994141\r\n\r\n    In [42]: ddf.tail(3)\r\n    Out[42]: \r\n           idx        date   a  b  c  d   e   f             g\r\n    510116  146063  2016-02-27  78  5  8  5  38  86  13127.741211\r\n    510117  146063  2016-02-28  19  1  1  1  13  29   3844.364014\r\n    510118  146063  2016-02-29  12  3  0  1  21  29   5778.074219\r\n\r\nif I do a ddf.set_index(\"idx\", sorted=True), it will not work as divisions aren't apparently picked up.\r\n\r\n    In [43]: ddf=ddf.set_index(\"idx\",sorted=True)\r\n    In [44]: ddf.divisions\r\n    Out[44]: (0, 0, 0, 0, 0, 0, 0, 0, 146063)\r\n    \r\n    In [46]: ddf.head()\r\n    Out[46]: \r\n           date   a  b   c  d   e   f             g\r\n    idx                                                \r\n    0    2015-04-27   1  1   0  0   0   2     97.411003\r\n    1    2016-04-20   2  0   0  0   0   1     56.868000\r\n    2    2016-09-09  42  5  11  1  54  58  16826.994141\r\n\r\nThen I try to set up the divisions manually inspecting the first row of each dataframe in each partition.\r\n\r\n    In [47]: [ ddf.get_partition(x).compute().index.values[0] for x in range(8)]\r\n    Out[47]: [0, 0, 0, 0, 0, 0, 0, 0]\r\n\r\nApparently get_partition is including row[0] of the full dataframe, as the first element of each partition ?\r\n\r\n    In [50]: [ ddf.get_partition(x).compute().idx[0:3]  for x in range(8)]\r\n    Out[50]: \r\n    [0    0\r\n    1    1\r\n    2    2\r\n    Name: idx, dtype: int64,\r\n    0        0\r\n    1    20031\r\n    2    20031\r\n    Name: idx, dtype: int64,\r\n    0        0\r\n    1    40250\r\n    2    40250\r\n    Name: idx, dtype: int64,\r\n    0        0\r\n    1    60406\r\n    2    60406\r\n    Name: idx, dtype: int64,\r\n    0        0\r\n    1    80867\r\n    2    80867\r\n    Name: idx, dtype: int64,\r\n\r\nHowever, this works as expected.\r\n\r\n    df=pd.read_csv(\"tmpaa\",names=[\"idx\",\"date\",\"a\",\"b\",\"c\",\"d\",\"e\",\"f\",\"g\"],index_col=0)\r\n    ddf=dd.from_pandas(df,npartitions=8)\r\n    In [68]: ddf.divisions\r\n    Out[68]: (0, 18183, 36313, 55107, 73718, 92171, 109898, 127817, 146063)\r\n    In [69]: [ ddf.get_partition(x).compute().index.values[0]  for x in range(8)]\r\n\r\n\r\n    Out[69]: [0, 18183, 36313, 55107, 73718, 92171, 109898, 127817]\r\n\r\n\r\nAs per my understanding of reading the documentation, I should be getting same results one way or another.\r\n\r\nJC "}}
{"actor":{"display_login":"littlegreenbean33"}}