{"action":"created","issue":{"url":"https://api.github.com/repos/apache/carbondata/issues/1632","repository_url":"https://api.github.com/repos/apache/carbondata","labels_url":"https://api.github.com/repos/apache/carbondata/issues/1632/labels{/name}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/1632/comments","events_url":"https://api.github.com/repos/apache/carbondata/issues/1632/events","html_url":"https://github.com/apache/carbondata/pull/1632","id":280108669,"number":1632,"title":"[CARBONDATA-1839] [DataLoad]Fix bugs in compressing sort temp files","user":{"login":"xuchuanyin","id":10445758,"avatar_url":"https://avatars1.githubusercontent.com/u/10445758?v=4","gravatar_id":"","url":"https://api.github.com/users/xuchuanyin","html_url":"https://github.com/xuchuanyin","followers_url":"https://api.github.com/users/xuchuanyin/followers","following_url":"https://api.github.com/users/xuchuanyin/following{/other_user}","gists_url":"https://api.github.com/users/xuchuanyin/gists{/gist_id}","starred_url":"https://api.github.com/users/xuchuanyin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuchuanyin/subscriptions","organizations_url":"https://api.github.com/users/xuchuanyin/orgs","repos_url":"https://api.github.com/users/xuchuanyin/repos","events_url":"https://api.github.com/users/xuchuanyin/events{/privacy}","received_events_url":"https://api.github.com/users/xuchuanyin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-12-07T12:16:04Z","updated_at":"2017-12-11T08:34:25Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/1632","html_url":"https://github.com/apache/carbondata/pull/1632","diff_url":"https://github.com/apache/carbondata/pull/1632.diff","patch_url":"https://github.com/apache/carbondata/pull/1632.patch"},"body":"Be sure to do all of the following checklist to help us incorporate \r\nyour contribution quickly and easily:\r\n\r\n - [X] Any interfaces changed?\r\n  `YES, ONLY CHANGE INTERNAL INTERFACES`\r\n - [X] Any backward compatibility impacted?\r\n  `NO`\r\n - [X] Document update required?\r\n  `YES, RELATED DOCUMENT HAS BEEN UPDATED`\r\n - [X] Testing done\r\n        Please provide details on \r\n        - Whether new unit test cases have been added or why no new tests are required?\r\n        `ADDED TESTS`\r\n        - How it is tested? Please attach test report.\r\n        `TESTED IN LOCAL CLUSTER`\r\n        - Is it a performance related change? Please attach the performance test report.\r\n        `YES`\r\n        - Any additional information to help reviewers in testing this change.\r\n        `The key point lies in` **`SortStepRowHandler`**`. It is used to convert raw row to 3-parted row and read/write row from/to sort temp file/unsafe memory`\r\n - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. \r\n        `NOT RELATED`\r\n\r\nRESOLVE\r\n===\r\n\r\n1. Fix bugs in compressing sort temp file\r\n\r\n2. Reduce duplicate code in reading & writing sort temp file\r\n and make it more readable\r\n\r\n3. Optimize sort procedure:\r\n\r\nBefore:\r\n\r\n```flow\r\nst=>start: raw row that has been converted(call it 'RawRow' for short)\r\ne=>end: write 'PartedRow' to DataFile in write procedure\r\nop1=>operation: read RawRow from temp sort file\r\nop2=>operation: sort on RawRow\r\nop3=>operation: write RawRow to temp sort file\r\ncond=>condition: final sort?\r\nop4=>operation: sort on RawRow\r\nop5=>operation: convert each RawRow to 3 'PartedRow'\r\n\r\nst->op1->op2->op3->cond\r\ncond(no)->op1\r\ncond(yes)->op4->op5->e\r\n```\r\nAfter：\r\n\r\n```flow\r\nst=>start: raw row that has been converted(call it 'RawRow' for short)\r\ne=>end: write 'PartedRow' to DataFile in write procedure\r\nop1=>operation: convert RawRow to 3 'PartedRow'\r\nop2=>operation: read PartedRow from temp sort file\r\nop3=>operation: sort on PartedRow\r\nop4=>operation: write PartedRow to temp sort file\r\ncond=>condition: final sort?\r\nop5=>operation: sort on PartedRow\r\n\r\nst->op1->op2->op3->op4->cond\r\ncond(no)->op2\r\ncond(yes)->op5->e\r\n```\r\n\r\n4. Add tests to enable sort_temp_file_compressed while doing data loading\r\n\r\n5. Update related document"},"comment":{"url":"https://api.github.com/repos/apache/carbondata/issues/comments/350655498","html_url":"https://github.com/apache/carbondata/pull/1632#issuecomment-350655498","issue_url":"https://api.github.com/repos/apache/carbondata/issues/1632","id":350655498,"user":{"login":"CarbonDataQA","id":24402133,"avatar_url":"https://avatars0.githubusercontent.com/u/24402133?v=4","gravatar_id":"","url":"https://api.github.com/users/CarbonDataQA","html_url":"https://github.com/CarbonDataQA","followers_url":"https://api.github.com/users/CarbonDataQA/followers","following_url":"https://api.github.com/users/CarbonDataQA/following{/other_user}","gists_url":"https://api.github.com/users/CarbonDataQA/gists{/gist_id}","starred_url":"https://api.github.com/users/CarbonDataQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CarbonDataQA/subscriptions","organizations_url":"https://api.github.com/users/CarbonDataQA/orgs","repos_url":"https://api.github.com/users/CarbonDataQA/repos","events_url":"https://api.github.com/users/CarbonDataQA/events{/privacy}","received_events_url":"https://api.github.com/users/CarbonDataQA/received_events","type":"User","site_admin":false},"created_at":"2017-12-11T08:34:25Z","updated_at":"2017-12-11T08:34:25Z","author_association":"NONE","body":"Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1863/\n"}}
{"actor":{"display_login":"CarbonDataQA"}}