{"action":"created","issue":{"url":"https://api.github.com/repos/apache/carbondata/issues/971","repository_url":"https://api.github.com/repos/apache/carbondata","labels_url":"https://api.github.com/repos/apache/carbondata/issues/971/labels{/name}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/971/comments","events_url":"https://api.github.com/repos/apache/carbondata/issues/971/events","html_url":"https://github.com/apache/carbondata/pull/971","id":232117593,"number":971,"title":"[CARBONDATA-1015] Extract interface in data load write step","user":{"login":"jackylk","id":6500698,"avatar_url":"https://avatars0.githubusercontent.com/u/6500698?v=3","gravatar_id":"","url":"https://api.github.com/users/jackylk","html_url":"https://github.com/jackylk","followers_url":"https://api.github.com/users/jackylk/followers","following_url":"https://api.github.com/users/jackylk/following{/other_user}","gists_url":"https://api.github.com/users/jackylk/gists{/gist_id}","starred_url":"https://api.github.com/users/jackylk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jackylk/subscriptions","organizations_url":"https://api.github.com/users/jackylk/orgs","repos_url":"https://api.github.com/users/jackylk/repos","events_url":"https://api.github.com/users/jackylk/events{/privacy}","received_events_url":"https://api.github.com/users/jackylk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-05-30T00:41:43Z","updated_at":"2017-05-30T18:03:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/971","html_url":"https://github.com/apache/carbondata/pull/971","diff_url":"https://github.com/apache/carbondata/pull/971.diff","patch_url":"https://github.com/apache/carbondata/pull/971.patch"},"body":"This PR extracts TablePage/Encoder/TableStatistics/TableSpec interface that used for:\r\n- make write step use unsafe data structure to reduce GC\r\n- Encoding override\r\n- make sort step output ColumnPage in UnsafeInMemoryIntermediateDataMerger\r\n\r\nChanges in this PR:\r\n## New interface added\r\n### TableSpec\r\n- extract dimension spec from schema, dimension are flattened after complex column expansion \r\n- extract measure spec from scheme \r\n- stored in CarbonFactDataHandlerModel and used in write step wherever needed, like in Encoder\r\n- currently it is temporary implementation, in future it may change to hold metadata for key column and non-key columns only\r\n\r\n### TablePage \r\n- contains pages for all columns in a table, pageSize is configurable\r\n- use Unsafe to hold on heap or off heap data, by configuration (future PR)\r\n- current implement holds pages for key columns, no dictionary columns, complex columns and measure columns, in future it is possible that only hold key columns and non-key columns.\r\n\r\n### TablePageStatistics\r\n- holds statistics for TablePage, which is needed in writer\r\n\r\n### Encoder\r\n- take a table page and output encoded byte[][], for both dimension and measure\r\n- a default implementation is added, using RLE/inverted index for dimension, delta/adaptive for measure\r\n\r\n## Following interface and behavior is changed\r\n- CompressionModel is removed, use TablePageStatistics and TableSpec instead\r\n- When encoding dimension, not starting SortBlockThread, use write step thread only. SortBlockThread is removed\r\n- compression of dimension using Snappy is moved from Writer to Producer, so Producer will output encoded and compressed dimension and measure, which is Encoder.EncodedData\r\n- some function name and variable name is changed for better readability\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/carbondata/issues/comments/304960070","html_url":"https://github.com/apache/carbondata/pull/971#issuecomment-304960070","issue_url":"https://api.github.com/repos/apache/carbondata/issues/971","id":304960070,"user":{"login":"asfbot","id":5243949,"avatar_url":"https://avatars0.githubusercontent.com/u/5243949?v=3","gravatar_id":"","url":"https://api.github.com/users/asfbot","html_url":"https://github.com/asfbot","followers_url":"https://api.github.com/users/asfbot/followers","following_url":"https://api.github.com/users/asfbot/following{/other_user}","gists_url":"https://api.github.com/users/asfbot/gists{/gist_id}","starred_url":"https://api.github.com/users/asfbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asfbot/subscriptions","organizations_url":"https://api.github.com/users/asfbot/orgs","repos_url":"https://api.github.com/users/asfbot/repos","events_url":"https://api.github.com/users/asfbot/events{/privacy}","received_events_url":"https://api.github.com/users/asfbot/received_events","type":"User","site_admin":false},"created_at":"2017-05-30T18:03:45Z","updated_at":"2017-05-30T18:03:45Z","body":"Tests Failed for Spark 1.6\n \n"}}
{"actor":{"display_login":"asfbot"}}