{"action":"created","issue":{"url":"https://api.github.com/repos/dask/fastparquet/issues/177","repository_url":"https://api.github.com/repos/dask/fastparquet","labels_url":"https://api.github.com/repos/dask/fastparquet/issues/177/labels{/name}","comments_url":"https://api.github.com/repos/dask/fastparquet/issues/177/comments","events_url":"https://api.github.com/repos/dask/fastparquet/issues/177/events","html_url":"https://github.com/dask/fastparquet/pull/177","id":239016439,"number":177,"title":"Flatten schema to enable loading of non-repeated columns","user":{"login":"martindurant","id":6042212,"avatar_url":"https://avatars2.githubusercontent.com/u/6042212?v=3","gravatar_id":"","url":"https://api.github.com/users/martindurant","html_url":"https://github.com/martindurant","followers_url":"https://api.github.com/users/martindurant/followers","following_url":"https://api.github.com/users/martindurant/following{/other_user}","gists_url":"https://api.github.com/users/martindurant/gists{/gist_id}","starred_url":"https://api.github.com/users/martindurant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martindurant/subscriptions","organizations_url":"https://api.github.com/users/martindurant/orgs","repos_url":"https://api.github.com/users/martindurant/repos","events_url":"https://api.github.com/users/martindurant/events{/privacy}","received_events_url":"https://api.github.com/users/martindurant/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-06-28T00:01:12Z","updated_at":"2017-06-29T16:55:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dask/fastparquet/pulls/177","html_url":"https://github.com/dask/fastparquet/pull/177","diff_url":"https://github.com/dask/fastparquet/pull/177.diff","patch_url":"https://github.com/dask/fastparquet/pull/177.patch"},"body":"Something like\r\n```\r\nroot\r\n| - visitor: OPTIONAL\r\n  | - ip: BYTE_ARRAY, UTF8, OPTIONAL\r\n  | - network_id: BYTE_ARRAY, UTF8, OPTIONAL\r\n  | - site_id: BYTE_ARRAY, UTF8, OPTIONAL\r\n```\r\nbecomes\r\n```\r\nroot\r\n| - visitor: OPTIONAL\r\n| - visitor.ip: BYTE_ARRAY, UTF8, OPTIONAL\r\n| - visitor.network_id: BYTE_ARRAY, UTF8, OPTIONAL\r\n| - visitor.site_id: BYTE_ARRAY, UTF8, OPTIONAL\r\n```\r\nNote that column 'visitor' contains no data at all, and there is no way\r\nto tell the difference between visitor=NULL and all of the fields of\r\nvisitor being NULL.\r\n\r\nPartially Fixes #35 "},"comment":{"url":"https://api.github.com/repos/dask/fastparquet/issues/comments/312027755","html_url":"https://github.com/dask/fastparquet/pull/177#issuecomment-312027755","issue_url":"https://api.github.com/repos/dask/fastparquet/issues/177","id":312027755,"user":{"login":"j-bennet","id":637013,"avatar_url":"https://avatars2.githubusercontent.com/u/637013?v=3","gravatar_id":"","url":"https://api.github.com/users/j-bennet","html_url":"https://github.com/j-bennet","followers_url":"https://api.github.com/users/j-bennet/followers","following_url":"https://api.github.com/users/j-bennet/following{/other_user}","gists_url":"https://api.github.com/users/j-bennet/gists{/gist_id}","starred_url":"https://api.github.com/users/j-bennet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/j-bennet/subscriptions","organizations_url":"https://api.github.com/users/j-bennet/orgs","repos_url":"https://api.github.com/users/j-bennet/repos","events_url":"https://api.github.com/users/j-bennet/events{/privacy}","received_events_url":"https://api.github.com/users/j-bennet/received_events","type":"User","site_admin":false},"created_at":"2017-06-29T16:55:53Z","updated_at":"2017-06-29T16:55:53Z","body":"Trying this out on a dataframe with metadata. Here is one row:\r\n\r\n```\r\nipdb> df.iloc[0]\r\nurl                                               http://blog.parsely.com/post/46/\r\n_type                                                                          url\r\napikey                                                            blog.parsely.com\r\naudience_segment_id                                                             11\r\nmetrics                          {u'$any/returning/engaged_seconds': 9, u'$any/...\r\nreferrers                                                                     None\r\nts_apikey                                                      2017-02-02 02:15:00\r\nvis_new                                                                         []\r\nvis_returning                               [908d02d6-df11-49a2-9a20-139e528ddb59]\r\nvisitors                                    [908d02d6-df11-49a2-9a20-139e528ddb59]\r\nts                                                             2017-02-02 07:15:00\r\n_index                                                   casterisk-5min-2017.02.02\r\n_id                              http://blog.parsely.com/post/46/|blog.parsely....\r\nindex_ts                                                2017-06-29 16:38:03.088000\r\nfreq                                                                          5min\r\ncampaign                                                                      None\r\nmetas.authors                                                    Andrew Montalenti\r\nmetas.canonical_hash                                                        QtBJro\r\nmetas.canonical_hash64                                                 FZyXxuRY9Os\r\nmetas.canonical_url                http://blog.parse.ly/post/46/whatever-it-takes/\r\nmetas.data_source                                                            crawl\r\nmetas.full_content_word_count                                                 1681\r\nmetas.image_url                  http://farm9.staticflickr.com/8051/8119717424_...\r\nmetas.page_type                                                               post\r\nmetas.post_id                      http://blog.parse.ly/post/46/whatever-it-takes/\r\nmetas.pub_date                                                 2012-11-02 04:00:00\r\nmetas.section                                                Analytics That Matter\r\nmetas.tags                                                            [None, None]\r\nmetas.thumb_url_medium           https://images.parsely.com/rr1hHDt1zufL7Gol-iN...\r\nmetas.title                                                      Whatever It Takes\r\nmetas.urls                              [None, None, None, None, None, None, None]\r\nName: 0, dtype: object\r\n```\r\n\r\nLooks like all urls became `None` in `metas.urls`, and all `metas.authors` became strings, instead of string arrays.\r\n\r\nurls:\r\n\r\n```\r\nipdb> df[\"metas.urls\"]\r\n0       [None, None, None, None, None, None, None]\r\n1       [None, None, None, None, None, None, None]\r\n2       [None, None, None, None, None, None, None]\r\n3       [None, None, None, None, None, None, None]\r\n4                         [None, None, None, None]\r\n5                         [None, None, None, None]\r\n6                         [None, None, None, None]\r\n7                         [None, None, None, None]\r\n8                         [None, None, None, None]\r\n9                         [None, None, None, None]\r\n10                        [None, None, None, None]\r\n11                        [None, None, None, None]\r\n12                        [None, None, None, None]\r\n13                        [None, None, None, None]\r\n14                        [None, None, None, None]\r\n15                        [None, None, None, None]\r\n16                        [None, None, None, None]\r\n17                        [None, None, None, None]\r\n18                        [None, None, None, None]\r\n19                        [None, None, None, None]\r\n20                        [None, None, None, None]\r\n21                        [None, None, None, None]\r\n22                        [None, None, None, None]\r\n23                        [None, None, None, None]\r\n24                        [None, None, None, None]\r\n25                        [None, None, None, None]\r\n26                        [None, None, None, None]\r\n27                        [None, None, None, None]\r\n28                        [None, None, None, None]\r\n29                        [None, None, None, None]\r\n```\r\n\r\nauthors:\r\n\r\n```\r\nipdb> df[\"metas.authors\"]\r\n0       Andrew Montalenti\r\n1       Andrew Montalenti\r\n2       Andrew Montalenti\r\n3       Andrew Montalenti\r\n4                Parse.ly\r\n5                Parse.ly\r\n6                Parse.ly\r\n7                Parse.ly\r\n8       Andrew Montalenti\r\n9       Andrew Montalenti\r\n10      Andrew Montalenti\r\n11      Andrew Montalenti\r\n12      Andrew Montalenti\r\n13      Andrew Montalenti\r\n14      Andrew Montalenti\r\n15      Andrew Montalenti\r\n16          Allie VanNest\r\n17          Allie VanNest\r\n18          Allie VanNest\r\n19          Allie VanNest\r\n20          Allie VanNest\r\n21          Allie VanNest\r\n22          Allie VanNest\r\n23          Allie VanNest\r\n24          Allie VanNest\r\n25          Allie VanNest\r\n26          Allie VanNest\r\n27          Allie VanNest\r\n28          Allie VanNest\r\n29          Allie VanNest\r\n```"}}
{"actor":{"display_login":"j-bennet"}}