{"action":"created","comment":{"url":"https://api.github.com/repos/apache/arrow/pulls/comments/111131909","pull_request_review_id":32352974,"id":111131909,"diff_hunk":"@@ -104,6 +111,360 @@ def _get_column_indices(self, column_names):\n                     for column in column_names]\n \n \n+# ----------------------------------------------------------------------\n+# Metadata container providing instructions about reading a single Parquet\n+# file, possibly part of a partitioned dataset\n+\n+\n+class ParquetDatasetPiece(object):\n+    \"\"\"\n+    A single chunk of a potentially larger Parquet dataset to read. The\n+    arguments will indicate to read either a single row group or all row\n+    groups, and whether to add partition keys to the resulting pyarrow.Table\n+\n+    Parameters\n+    ----------\n+    partition_keys : list of tuples\n+      [(column name, ordinal index)]\n+    \"\"\"\n+\n+    def __init__(self, path, row_group=None, partition_keys=None):\n+        self.path = path\n+        self.row_group = row_group\n+        self.partition_keys = partition_keys or []\n+\n+    def __eq__(self, other):\n+        if not isinstance(other, ParquetDatasetPiece):\n+            return False\n+        return (self.path == other.path and\n+                self.row_group == other.row_group and\n+                self.partition_keys == other.partition_keys)\n+\n+    def __ne__(self, other):\n+        return not (self == other)\n+\n+    def __repr__(self):\n+        return ('{0}({1!r}, row_group={2!r}, partition_keys={3!r})'\n+                .format(type(self).__name__, self.path,\n+                        self.row_group,\n+                        self.partition_keys))\n+\n+    def __str__(self):\n+        result = ''\n+\n+        if len(self.partition_keys) > 0:\n+            partition_str = ', '.join('{0}={1}'.format(name, index)\n+                                      for name, index in self.partition_keys)\n+            result += 'partition[{0}] '.format(partition_str)\n+\n+        result += self.path\n+\n+        if self.row_group is not None:\n+            result += ' | row_group={0}'.format(self.row_group)\n+\n+        return result\n+\n+    def open(self, open_file_func):\n+        return open_file_func(self.path)\n+\n+    def read(self, open_file_func, columns=None, nthreads=1,\n+             partitions=None):\n+        reader = self.open(open_file_func)","path":"python/pyarrow/parquet.py","position":109,"original_position":109,"commit_id":"deb6d82136ff3a804112f0015e3b52464ff723b1","original_commit_id":"deb6d82136ff3a804112f0015e3b52464ff723b1","user":{"login":"jreback","id":953992,"avatar_url":"https://avatars1.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"body":"maybe ``open_file_func=None`` convenient as a default here? (maybe also change the signature\r\n``columns=None, partitions=None, nthreads=1, open_file_func=None``?\r\n\r\nindifferent tweeen ``partitions`` and ``columns`` ordering in signature, but should be consistent (e.g. ``read_row_group``)","created_at":"2017-04-12T12:10:27Z","updated_at":"2017-04-12T12:37:23Z","html_url":"https://github.com/apache/arrow/pull/529#discussion_r111131909","pull_request_url":"https://api.github.com/repos/apache/arrow/pulls/529","_links":{"self":{"href":"https://api.github.com/repos/apache/arrow/pulls/comments/111131909"},"html":{"href":"https://github.com/apache/arrow/pull/529#discussion_r111131909"},"pull_request":{"href":"https://api.github.com/repos/apache/arrow/pulls/529"}}},"pull_request":{"url":"https://api.github.com/repos/apache/arrow/pulls/529","id":115430685,"html_url":"https://github.com/apache/arrow/pull/529","diff_url":"https://github.com/apache/arrow/pull/529.diff","patch_url":"https://github.com/apache/arrow/pull/529.patch","issue_url":"https://api.github.com/repos/apache/arrow/issues/529","number":529,"state":"open","locked":false,"title":"ARROW-539: [Python] Add support for reading partitioned Parquet files with Hive-like directory schemes","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars1.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"body":"cc @jreback @cpcloud for review. I probably didn't get all the use cases, but this should be a good start.\r\n\r\nFirst, the directory structure is walked to determine the distinct partition keys. These keys are later used as the dictionary for `arrow::DictionaryArray` objects which are constructed. \r\n\r\nI also created the `ParquetDatasetPiece` class to enable distributed processing of file components in frameworks like Dask. We may need to address pickling of the `ParquetPartitions` object (which must be passed to `ParquetDatasetPiece.read` so the right array metadata can be constructed. ","created_at":"2017-04-11T22:42:39Z","updated_at":"2017-04-12T12:37:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e4b70abd69cdb34c685c73916ad60afdbad75bf5","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/arrow/pulls/529/commits","review_comments_url":"https://api.github.com/repos/apache/arrow/pulls/529/comments","review_comment_url":"https://api.github.com/repos/apache/arrow/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/arrow/issues/529/comments","statuses_url":"https://api.github.com/repos/apache/arrow/statuses/deb6d82136ff3a804112f0015e3b52464ff723b1","head":{"label":"wesm:ARROW-539","ref":"ARROW-539","sha":"deb6d82136ff3a804112f0015e3b52464ff723b1","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars1.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"repo":{"id":52022289,"name":"arrow","full_name":"wesm/arrow","owner":{"login":"wesm","id":329591,"avatar_url":"https://avatars1.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/wesm/arrow","description":"Mirror of Apache Arrow","fork":true,"url":"https://api.github.com/repos/wesm/arrow","forks_url":"https://api.github.com/repos/wesm/arrow/forks","keys_url":"https://api.github.com/repos/wesm/arrow/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wesm/arrow/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wesm/arrow/teams","hooks_url":"https://api.github.com/repos/wesm/arrow/hooks","issue_events_url":"https://api.github.com/repos/wesm/arrow/issues/events{/number}","events_url":"https://api.github.com/repos/wesm/arrow/events","assignees_url":"https://api.github.com/repos/wesm/arrow/assignees{/user}","branches_url":"https://api.github.com/repos/wesm/arrow/branches{/branch}","tags_url":"https://api.github.com/repos/wesm/arrow/tags","blobs_url":"https://api.github.com/repos/wesm/arrow/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wesm/arrow/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wesm/arrow/git/refs{/sha}","trees_url":"https://api.github.com/repos/wesm/arrow/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wesm/arrow/statuses/{sha}","languages_url":"https://api.github.com/repos/wesm/arrow/languages","stargazers_url":"https://api.github.com/repos/wesm/arrow/stargazers","contributors_url":"https://api.github.com/repos/wesm/arrow/contributors","subscribers_url":"https://api.github.com/repos/wesm/arrow/subscribers","subscription_url":"https://api.github.com/repos/wesm/arrow/subscription","commits_url":"https://api.github.com/repos/wesm/arrow/commits{/sha}","git_commits_url":"https://api.github.com/repos/wesm/arrow/git/commits{/sha}","comments_url":"https://api.github.com/repos/wesm/arrow/comments{/number}","issue_comment_url":"https://api.github.com/repos/wesm/arrow/issues/comments{/number}","contents_url":"https://api.github.com/repos/wesm/arrow/contents/{+path}","compare_url":"https://api.github.com/repos/wesm/arrow/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wesm/arrow/merges","archive_url":"https://api.github.com/repos/wesm/arrow/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wesm/arrow/downloads","issues_url":"https://api.github.com/repos/wesm/arrow/issues{/number}","pulls_url":"https://api.github.com/repos/wesm/arrow/pulls{/number}","milestones_url":"https://api.github.com/repos/wesm/arrow/milestones{/number}","notifications_url":"https://api.github.com/repos/wesm/arrow/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wesm/arrow/labels{/name}","releases_url":"https://api.github.com/repos/wesm/arrow/releases{/id}","deployments_url":"https://api.github.com/repos/wesm/arrow/deployments","created_at":"2016-02-18T16:41:50Z","updated_at":"2017-03-26T23:01:52Z","pushed_at":"2017-04-11T22:49:34Z","git_url":"git://github.com/wesm/arrow.git","ssh_url":"git@github.com:wesm/arrow.git","clone_url":"https://github.com/wesm/arrow.git","svn_url":"https://github.com/wesm/arrow","homepage":null,"size":4077,"stargazers_count":1,"watchers_count":1,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"ab520cbc7a1e3fe14a2290322ca2e392af30d612","user":{"login":"apache","id":47359,"avatar_url":"https://avatars3.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":51905353,"name":"arrow","full_name":"apache/arrow","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars3.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/arrow","description":"Apache Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ","fork":false,"url":"https://api.github.com/repos/apache/arrow","forks_url":"https://api.github.com/repos/apache/arrow/forks","keys_url":"https://api.github.com/repos/apache/arrow/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/arrow/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/arrow/teams","hooks_url":"https://api.github.com/repos/apache/arrow/hooks","issue_events_url":"https://api.github.com/repos/apache/arrow/issues/events{/number}","events_url":"https://api.github.com/repos/apache/arrow/events","assignees_url":"https://api.github.com/repos/apache/arrow/assignees{/user}","branches_url":"https://api.github.com/repos/apache/arrow/branches{/branch}","tags_url":"https://api.github.com/repos/apache/arrow/tags","blobs_url":"https://api.github.com/repos/apache/arrow/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/arrow/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/arrow/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/arrow/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/arrow/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/arrow/languages","stargazers_url":"https://api.github.com/repos/apache/arrow/stargazers","contributors_url":"https://api.github.com/repos/apache/arrow/contributors","subscribers_url":"https://api.github.com/repos/apache/arrow/subscribers","subscription_url":"https://api.github.com/repos/apache/arrow/subscription","commits_url":"https://api.github.com/repos/apache/arrow/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/arrow/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/arrow/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/arrow/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/arrow/contents/{+path}","compare_url":"https://api.github.com/repos/apache/arrow/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/arrow/merges","archive_url":"https://api.github.com/repos/apache/arrow/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/arrow/downloads","issues_url":"https://api.github.com/repos/apache/arrow/issues{/number}","pulls_url":"https://api.github.com/repos/apache/arrow/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/arrow/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/arrow/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/arrow/labels{/name}","releases_url":"https://api.github.com/repos/apache/arrow/releases{/id}","deployments_url":"https://api.github.com/repos/apache/arrow/deployments","created_at":"2016-02-17T08:00:23Z","updated_at":"2017-04-11T13:25:38Z","pushed_at":"2017-04-11T23:04:36Z","git_url":"git://github.com/apache/arrow.git","ssh_url":"git@github.com:apache/arrow.git","clone_url":"https://github.com/apache/arrow.git","svn_url":"https://github.com/apache/arrow","homepage":"https://arrow.apache.org","size":3952,"stargazers_count":875,"watchers_count":875,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":214,"mirror_url":"git://git.apache.org/arrow.git","open_issues_count":9,"forks":214,"open_issues":9,"watchers":875,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/arrow/pulls/529"},"html":{"href":"https://github.com/apache/arrow/pull/529"},"issue":{"href":"https://api.github.com/repos/apache/arrow/issues/529"},"comments":{"href":"https://api.github.com/repos/apache/arrow/issues/529/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/arrow/pulls/529/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/arrow/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/arrow/pulls/529/commits"},"statuses":{"href":"https://api.github.com/repos/apache/arrow/statuses/deb6d82136ff3a804112f0015e3b52464ff723b1"}}}}
{"actor":{"display_login":"jreback"}}