{"action":"created","issue":{"url":"https://api.github.com/repos/apache/arrow/issues/1294","repository_url":"https://api.github.com/repos/apache/arrow","labels_url":"https://api.github.com/repos/apache/arrow/issues/1294/labels{/name}","comments_url":"https://api.github.com/repos/apache/arrow/issues/1294/comments","events_url":"https://api.github.com/repos/apache/arrow/issues/1294/events","html_url":"https://github.com/apache/arrow/pull/1294","id":272436150,"number":1294,"title":"ARROW-1693: [JS] Fix reading C++ dictionary-encoded vectors","user":{"login":"trxcllnt","id":178183,"avatar_url":"https://avatars0.githubusercontent.com/u/178183?v=4","gravatar_id":"","url":"https://api.github.com/users/trxcllnt","html_url":"https://github.com/trxcllnt","followers_url":"https://api.github.com/users/trxcllnt/followers","following_url":"https://api.github.com/users/trxcllnt/following{/other_user}","gists_url":"https://api.github.com/users/trxcllnt/gists{/gist_id}","starred_url":"https://api.github.com/users/trxcllnt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trxcllnt/subscriptions","organizations_url":"https://api.github.com/users/trxcllnt/orgs","repos_url":"https://api.github.com/users/trxcllnt/repos","events_url":"https://api.github.com/users/trxcllnt/events{/privacy}","received_events_url":"https://api.github.com/users/trxcllnt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2017-11-09T04:20:17Z","updated_at":"2017-11-16T21:53:20Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/arrow/pulls/1294","html_url":"https://github.com/apache/arrow/pull/1294","diff_url":"https://github.com/apache/arrow/pull/1294.diff","patch_url":"https://github.com/apache/arrow/pull/1294.patch"},"body":"This PR adds a workaround for reading the metadata layout for C++ dictionary-encoded vectors.\r\n\r\nI added tests that validate against the C++/Java integration suite. In order to make the new tests pass, I had to update the generated flatbuffers format and add a few types the JS version didn't have yet (Bool, Date32, and Timestamp). It also uses the new `isDelta` flag on DictionaryBatches to determine whether the DictionaryBatch vector should replace or append to the existing dictionary.\r\n\r\nI also added a script for generating test arrow files from the C++ and Java implementations, so we don't break the tests updating the format in the future. I saved the generated Arrow files in with the tests because I didn't see a way to pipe the JSON test data through the C++/Java json-to-arrow commands without writing to a file. If I missed something and we can do it all in-memory, I'd be happy to make that change!\r\n\r\nThis PR is marked WIP because I added an [integration test](https://github.com/apache/arrow/commit/6e98874d9f4bfae7758f8f731212ae7ceb3f1321#diff-18c6be12406c482092d4b1f7bd70a8e1R22) that validates the JS reader reads C++ and Java files the same way, but unfortunately it doesn't. Debugging, I noticed a number of other differences between the buffer layout metadata between the C++ and Java versions. If we go ahead with @jacques-n [comment in ARROW-1693](https://issues.apache.org/jira/browse/ARROW-1693?focusedCommentId=16244812&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16244812) and remove/ignore the metadata, this test should pass too.\r\n\r\ncc @TheNeuralBit "},"comment":{"url":"https://api.github.com/repos/apache/arrow/issues/comments/345074775","html_url":"https://github.com/apache/arrow/pull/1294#issuecomment-345074775","issue_url":"https://api.github.com/repos/apache/arrow/issues/1294","id":345074775,"user":{"login":"wesm","id":329591,"avatar_url":"https://avatars2.githubusercontent.com/u/329591?v=4","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"created_at":"2017-11-16T21:53:20Z","updated_at":"2017-11-16T21:53:20Z","author_association":"MEMBER","body":"Does anything still need to be done on this branch? If we are still not close to being able to cut a JS release by early next week I will rearrange my priorities to help out "}}
{"actor":{"display_login":"wesm"}}