{"push_id":2006855653,"size":17,"distinct_size":2,"ref":"refs/heads/DRILL-4139_rebase","head":"fad26fb3524364dff6dd0caeab147b359191829b","before":"55ae63a159b84af4a53a719fbde5c27ab0e691a1","commits":[{"sha":"fde0a1df1734e0742b49aabdd28b02202ee2b044","author":{"name":"Jinfeng Ni","email":"4c3241b5852156f12cc9d8881689651329b66200@apache.org"},"message":"DRILL-5546: Handle schema change exception failure caused by empty input or empty batch.\n\n1. Modify ScanBatch's logic when it iterates list of RecordReader.\n   1) Skip RecordReader if it returns 0 row && present same schema. A new schema (by calling Mutator.isNewSchema() ) means either a new top level field is added, or a field in a nested field is added, or an existing field type is changed.\n   2) Implicit columns are presumed to have constant schema, and are added to outgoing container before any regular column is added in.\n   3) ScanBatch will return NONE directly (called as \"fast NONE\"), if all its RecordReaders haver empty input and thus are skipped, in stead of returing OK_NEW_SCHEMA first.\n\n2. Modify IteratorValidatorBatchIterator to allow\n   1) fast NONE ( before seeing a OK_NEW_SCHEMA)\n   2) batch with empty list of columns.\n\n2. Modify JsonRecordReader when it get 0 row. Do not insert a nullable-int column for 0 row input. Together with ScanBatch, Drill will skip empty json files.\n\n3. Modify binary operators such as join, union to handle fast none for either one side or both sides. Abstract the logic in AbstractBinaryRecordBatch, except for MergeJoin as its implementation is quite different from others.\n\n4. Fix and refactor union all operator.\n  1) Correct union operator hanndling 0 input rows. Previously, it will ignore inputs with 0 row and put nullable-int into output schema, which causes various of schema change issue in down-stream operator. The new behavior is to take schema with 0 into account\n  in determining the output schema, in the same way with > 0 input rows. By doing that, we ensure Union operator will not behave like a schema-lossy operator.\n  2) Add a UnionInputIterator to simplify the logic to iterate the left/right inputs, removing significant chunk of duplicate codes in previous implementation.\n  The new union all operator reduces the code size into half, comparing the old one.\n\n5. Introduce UntypedNullVector to handle convertFromJson() function, when the input batch contains 0 row.\n  Problem: The function convertFromJSon() is different from other regular functions in that it only knows the output schema after evaluation is performed. When input has 0 row, Drill essentially does not have\n  a way to know the output type, and previously will assume Map type. That works under the assumption other operators like Union would ignore batch with 0 row, which is no longer\n  the case in the current implementation.\n  Solution: Use MinorType.NULL at the output type for convertFromJSON() when input contains 0 row. The new UntypedNullVector is used to represent a column with MinorType.NULL.\n\n6. HBaseGroupScan convert star column into list of row_key and column family. HBaseRecordReader should reject column star since it expectes star has been converted somewhere else.\n  In HBase a column family always has map type, and a non-rowkey column always has nullable varbinary type, this ensures that HBaseRecordReader across different HBase regions will have the same top level schema, even if the region is\n  empty or prune all the rows due to filter pushdown optimization. In other words, we will not see different top level schema from different HBaseRecordReader for the same table.\n  However, such change will not be able to handle hard schema change : c1 exists in cf1 in one region, but not in another region. Further work is required to handle hard schema change.\n\n7. Modify scan cost estimation when the query involves * column. This is to remove the planning randomness since previously two different operators could have same cost.\n\n8. Add a new flag 'outputProj' to Project operator, to indicate if Project is for the query's final output. Such Project is added by TopProjectVisitor, to handle fast NONE when all the inputs to the query are empty\nand are skipped.\n  1) column star is replaced with empty list\n  2) regular column reference is replaced with nullable-int column\n  3) An expression will go through ExpressionTreeMaterializer, and use the type of materialized expression as the output type\n  4) Return an OK_NEW_SCHEMA with the schema using the above logic, then return a NONE to down-stream operator.\n\n9. Add unit test to test operators handling empty input.\n\n10. Add unit test to test query when inputs are all empty.\n\nDRILL-5546: Revise code based on review comments.\n\nHandle implicit column in scan batch. Change interface in ScanBatch's constructor.\n 1) Ensure either the implicit column list is empty, or all the reader has the same set of implicit columns.\n 2) We could skip the implicit columns when check if there is a schema change coming from record reader.\n 3) ScanBatch accept a list in stead of iterator, since we may need go through the implicit column list multiple times, and verify the size of two lists are same.\n\nScanBatch code review comments. Add more unit tests.\n\nShare code path in ProjectBatch to handle normal setupNewSchema() and handleNullInput().\n - Move SimpleRecordBatch out of TopNBatch to make it sharable across different places.\n - Add Unit test verify schema for star column query against multilevel tables.\n\nUnit test framework change\n - Fix memory leak in unit test framework.\n - Allow SchemaTestBuilder to pass in BatchSchema.\n\nclose #906","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/fde0a1df1734e0742b49aabdd28b02202ee2b044"},{"sha":"7a900b71fd269aceee7301afb18fd8d303df5bcd","author":{"name":"Sindhuri Rayavaram","email":"a41c5edb691370e4d711608c71e9d9a10597e983@mapr.local"},"message":"DRILL-5765: Json query profile is not shown on Web UI\n\ncloses #933","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/7a900b71fd269aceee7301afb18fd8d303df5bcd"},{"sha":"c7c8ffd6f5e1b8f4677f9e95c317482d929a1bb5","author":{"name":"Arina Ielchiieva","email":"626ad2a0693a6f3389f12f17b7842fddc2e6ff5f@gmail.com"},"message":"DRILL-5766: Fix XSS vulnerabilities in Drill\n\n1. Bumped up freemarker version to 2.3.26-incubating.\n2. Indicated default output format in Freemarker configuration (HTML).\n3. Fixed Web UI bugs listed in DRILL-5346, DRILL-5341, DRILL-5339, DRILL-5338.\n\ncloses #935","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/c7c8ffd6f5e1b8f4677f9e95c317482d929a1bb5"},{"sha":"aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5","author":{"name":"Padma Penumarthy","email":"1967062571e738a93d00018c7f24e71477563941@yahoo.com"},"message":"DRILL-5697: Improve performance of filter operator for pattern matching\n\ncloses #907","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5"},{"sha":"0b1cb98ad65b2057e5540f84986eb718850dad20","author":{"name":"Vlad Rozov","email":"4194320718389f0492ab61ed3c1011f9bcd6ac44@apache.org"},"message":"DRILL-5269 Make DirectSubScan Jackson JSON deserializable\n\ncloses #926","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/0b1cb98ad65b2057e5540f84986eb718850dad20"},{"sha":"d1552db3850a80b67e4bc33877f70675ec7566a6","author":{"name":"Volodymyr Vysotskyi","email":"3e615dfe4d3298d764da68f17ffdbea0ec730f04@gmail.com"},"message":"DRILL-5761: Disable Lilith ClassicMultiplexSocketAppender by default.\nUnify logback files.","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/d1552db3850a80b67e4bc33877f70675ec7566a6"},{"sha":"df54cc2d00ed5d5056f2af55e7fef8a5696aa933","author":{"name":"Sindhuri Rayavaram","email":"a41c5edb691370e4d711608c71e9d9a10597e983@mapr.local"},"message":"DRILL-5712: Update the pom files with dependency exclusions for commons-codec\n\ncloses #903","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/df54cc2d00ed5d5056f2af55e7fef8a5696aa933"},{"sha":"6adeb986016a769755fd5e8fc66244ee1e8d18e1","author":{"name":"Timothy Farkas","email":"d7fa1e8f226ea2d0b52e38a6cbae81fa1a9dee8a@apache.org"},"message":"DRILL-5723: Added System Internal Options That can be Modified at Runtime\nChanges include:\n\n   1. Addition of internal options.\n   2. Refactoring of OptionManagers and OptionValidators.\n   3. Fixed ambiguity in the meaning of an option type, and changed its name to accessibleScopes.\n   4. Updated javadocs in the Option System classes.\n   5. Added RestClientFixture for testing the Rest API.\n   6. Fixed flakey test in TestExceptionInjection caused by race condition.\n   7. Fixed various tests which started zookeeper but failed to shut it down at the end of tests.\n   8. Added port hunting to the Drill Webserver for testing\n   9. Fixed various flaky tests\n  10. Fix compile issue\n\ncloses #923","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/6adeb986016a769755fd5e8fc66244ee1e8d18e1"},{"sha":"75bd1d04b01d23fc14730d6aba20964582990fa3","author":{"name":"Vlad Rozov","email":"4194320718389f0492ab61ed3c1011f9bcd6ac44@apache.org"},"message":"DRILL-3449: When Foreman node dies, the FragmentExecutor still tries to send status updates to Foreman\n\ncloses #934","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/75bd1d04b01d23fc14730d6aba20964582990fa3"},{"sha":"1c20a323f3941e1b4790332247ca1fe26e960477","author":{"name":"weijie.tong","email":"a98c5d08fa86a51d36d6b47d735be7b550eab5b8@alipay.com"},"message":"DRILL-5749: solve deadlock occured between foreman and netty threads\n\ncloses #943","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/1c20a323f3941e1b4790332247ca1fe26e960477"},{"sha":"b2744a2abf4fcda4ba9b30616bc379018a39af12","author":{"name":"Timothy Farkas","email":"d7fa1e8f226ea2d0b52e38a6cbae81fa1a9dee8a@apache.org"},"message":"DRILL-5798 changes include:\n\n - Fixed unstable StatusResourcesTest\n - Fixed buggy port hunting for the WebService\n - Fixed bug in ClientFixture which does not assign the correct user port when port hunting is done for the user port\n - Fixed unstable hash agg tests\n\ncloses #945","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/b2744a2abf4fcda4ba9b30616bc379018a39af12"},{"sha":"4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa","author":{"name":"Vitalii Diravka","email":"8fb3334f84f598c1aaeaa177deba2700ba28291d@gmail.com"},"message":"DRILL-5002: Using hive's date functions on top of date column gives wrong results for local time-zone\n\ncloses #937","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa"},{"sha":"d77ab31836e5e5b88c124e0c7540bccd0544dedb","author":{"name":"Ben-Zvi","email":"4ae3065b271210fb218e360482f2aea50385dde8@mapr.com"},"message":"DRILL-5694: Handle OOM in HashAggr by spill and retry, reserve memory, spinner","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/d77ab31836e5e5b88c124e0c7540bccd0544dedb"},{"sha":"b06a7bdecef8bdd52eebf4a2821a09c64b296886","author":{"name":"Sorabh Hamirwasia","email":"bebcc65c86baf854ad8a1d03481da2dff0e8ab49@maprtech.com"},"message":"DRILL-5721: Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues\n            Note: 1) To resolve the issue all the fragments including root fragment which are assigned to be executed on Foreman node\n                     are scheduled locally and not sent over Control Tunnel. Also the FragmentStatusReporter is updated to sent the\n                     status update locally by fragments running on Foreman node.\n                  2) Refactor for FragmentManager, setupRootFragment and startNewFragment\n                  3) Update the test added for DRILL-5701 as there is change in behavior","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/b06a7bdecef8bdd52eebf4a2821a09c64b296886"},{"sha":"6cb626d78de75a1061e1381f680478ed8844123a","author":{"name":"Sorabh Hamirwasia","email":"bebcc65c86baf854ad8a1d03481da2dff0e8ab49@maprtech.com"},"message":"DRILL-5721: Update based on merge conflict with DRILL-3449\n            Note: Resolved Merge Conflict and added certain new tests\n\ncloses #919","distinct":false,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/6cb626d78de75a1061e1381f680478ed8844123a"},{"sha":"ef322d986b3fb8a15a0f0d0d9f2b219d0de8788b","author":{"name":"Przemyslaw Maciolek","email":"96e83ef6e9811176490c6b7f69ca91790af1ce6c@collective-sense.com"},"message":"DRILL-4139: Add missing BIT support for Paruet partition pruning","distinct":true,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/ef322d986b3fb8a15a0f0d0d9f2b219d0de8788b"},{"sha":"fad26fb3524364dff6dd0caeab147b359191829b","author":{"name":"Volodymyr Vysotskyi","email":"3e615dfe4d3298d764da68f17ffdbea0ec730f04@gmail.com"},"message":"DRILL-4139: Add missing Interval, VarBinary and Varchar with nulls partition pruning support.\nFix metadata serialization for fixed_len_byte_array types.\nFix partition pruning for decimal type.\nFix loss of scale value for DECIMAL in parquet partition pruning.\nFix partition pruning for primitive types with null values.\nUpdate parquet table metadata version to v3_3.","distinct":true,"url":"https://api.github.com/repos/vvysotskyi/drill/commits/fad26fb3524364dff6dd0caeab147b359191829b"}]}
{"actor":{"display_login":"vvysotskyi"}}