{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18641","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18641/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18641/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18641/events","html_url":"https://github.com/apache/spark/pull/18641","id":243162528,"number":18641,"title":"[SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem in multiple projections with CASE WHEN","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars2.githubusercontent.com/u/1315079?v=4","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":28,"created_at":"2017-07-15T08:40:19Z","updated_at":"2017-11-15T11:00:29Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18641","html_url":"https://github.com/apache/spark/pull/18641","diff_url":"https://github.com/apache/spark/pull/18641.diff","patch_url":"https://github.com/apache/spark/pull/18641.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR changes Case When code generation to place condition and then expressions' generated code into separated methods if these size could be large. When the method is newly generated, variables for `isNull` and `value` are declared as an instance variable to pass these values (e.g. `isNull1409` and ` value1409`) to the callers of the generated method.\r\n\r\nThis PR resolved three cases:\r\n\r\n1. large code size of condition or then expression\r\n2. large code size of else expression\r\n3. large code size of total conditional branches\r\n\r\n\r\nCase 1 before this PR \r\n```java\r\n/* 005 */ class SpecificMutableProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseMutableProjection {\r\n...\r\n/* 034 */   public java.lang.Object apply(java.lang.Object _i) {\r\n/* 035 */     InternalRow i = (InternalRow) _i;\r\n/* 036 */\r\n/* 037 */\r\n/* 038 */\r\n/* 039 */     boolean isNull = true;\r\n/* 040 */     int value = -1;\r\n/* 041 */\r\n/* 042 */\r\n/* 043 */     boolean isNull1 = true;\r\n/* 044 */     boolean value1 = false;\r\n/* 045 */\r\n/* 046 */     boolean isNull2 = true;\r\n/* 047 */     int value2 = -1;\r\n/* 048 */\r\n/* 049 */\r\n/* 050 */     boolean isNull3 = true;\r\n/* 051 */     boolean value3 = false;\r\n/* 052 */\r\n/* 053 */     boolean isNull4 = true;\r\n/* 054 */     int value4 = -1;\r\n/* 055 */\r\n/* 056 */\r\n/* 057 */     boolean isNull5 = true;\r\n/* 058 */     boolean value5 = false;\r\n/* 059 */\r\n/* 060 */     boolean isNull6 = true;\r\n/* 061 */     int value6 = -1;\r\n/* 062 */\r\n/* 063 */\r\n/* 064 */     boolean isNull7 = true;\r\n/* 065 */     boolean value7 = false;\r\n/* 066 */\r\n/* 067 */     boolean isNull8 = true;\r\n/* 068 */     int value8 = -1;\r\n/* 069 */\r\n/* 070 */\r\n/* 071 */     boolean isNull9 = true;\r\n/* 072 */     boolean value9 = false;\r\n/* 073 */\r\n/* 074 */     boolean isNull10 = true;\r\n/* 075 */     int value10 = -1;\r\n/* 076 */\r\n/* 077 */\r\n/* 078 */     boolean isNull11 = true;\r\n/* 079 */     boolean value11 = false;\r\n/* 080 */\r\n/* 081 */     boolean isNull12 = true;\r\n/* 082 */     int value12 = -1;\r\n/* 083 */\r\n/* 084 */\r\n/* 085 */     boolean isNull13 = true;\r\n/* 086 */     boolean value13 = false;\r\n/* 087 */\r\n/* 088 */     boolean isNull14 = true;\r\n/* 089 */     int value14 = -1;\r\n/* 090 */\r\n/* 091 */\r\n/* 092 */     boolean isNull15 = true;\r\n/* 093 */     boolean value15 = false;\r\n/* 094 */\r\n/* 095 */     boolean isNull16 = true;\r\n/* 096 */     int value16 = -1;\r\n/* 097 */\r\n/* 098 */\r\n/* 099 */     boolean isNull17 = true;\r\n/* 100 */     boolean value17 = false;\r\n/* 101 */\r\n/* 102 */     boolean isNull18 = true;\r\n/* 103 */     int value18 = -1;\r\n/* 104 */\r\n/* 105 */\r\n/* 106 */     boolean isNull19 = true;\r\n/* 107 */     boolean value19 = false;\r\n/* 108 */\r\n/* 109 */     boolean isNull20 = i.isNullAt(0);\r\n/* 110 */     int value20 = isNull20 ? -1 : (i.getInt(0));\r\n/* 111 */     if (!isNull20) {\r\n/* 112 */\r\n/* 113 */\r\n/* 114 */       isNull19 = false; // resultCode could change nullability.\r\n/* 115 */       value19 = value20 == 0;\r\n/* 116 */\r\n/* 117 */     }\r\n/* 118 */     if (!isNull19 && value19) {\r\n/* 119 */\r\n/* 120 */       isNull18 = false;\r\n/* 121 */       value18 = -1;\r\n/* 122 */     }\r\n/* 123 */\r\n/* 124 */     else {\r\n/* 125 */\r\n/* 126 */\r\n/* 127 */       boolean isNull23 = i.isNullAt(0);\r\n/* 128 */       int value23 = isNull23 ? -1 : (i.getInt(0));\r\n/* 129 */       isNull18 = isNull23;\r\n/* 130 */       value18 = value23;\r\n/* 131 */     }\r\n...\r\n```\r\n\r\nCase 2 after this PR\r\n```java\r\n/* 005 */ class SpecificMutableProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseMutableProjection {\r\n...\r\n/* 263 */   private boolean isNull1409;\r\n/* 264 */   private boolean value1409;\r\n...\r\n/* 519 */   private boolean isNull2815;\r\n/* 520 */   private boolean value2815;\r\n...\r\n/* 1073 */   public java.lang.Object apply(java.lang.Object _i) {\r\n/* 1074 */     InternalRow i = (InternalRow) _i;\r\n/* 1075 */\r\n/* 1076 */\r\n/* 1077 */\r\n/* 1078 */     boolean isNull = true;\r\n/* 1079 */     int value = -1;\r\n/* 1080 */\r\n/* 1081 */     caseWhenCondExpr255(i);\r\n/* 1082 */     if (!isNull2815 && value2815) {\r\n/* 1083 */\r\n/* 1084 */       isNull = false;\r\n/* 1085 */       value = -1;\r\n/* 1086 */     }\r\n/* 1087 */\r\n/* 1088 */     else {\r\n/* 1089 */\r\n/* 1090 */\r\n/* 1091 */       boolean isNull2816 = true;\r\n/* 1092 */       int value2816 = -1;\r\n/* 1093 */\r\n/* 1094 */       caseWhenCondExpr383(i);\r\n/* 1095 */       if (!isNull4223 && value4223) {\r\n/* 1096 */\r\n/* 1097 */         isNull2816 = false;\r\n/* 1098 */         value2816 = -1;\r\n/* 1099 */       }\r\n...\r\n/* 30171 */   private void caseWhenCondExpr255(InternalRow i) {\r\n/* 30172 */     boolean isNull1 = true;\r\n/* 30173 */     boolean value1 = false;\r\n/* 30174 */\r\n/* 30175 */     boolean isNull2 = true;\r\n/* 30176 */     int value2 = -1;\r\n/* 30177 */\r\n/* 30178 */     caseWhenCondExpr127(i);\r\n/* 30179 */     if (!isNull1409 && value1409) {\r\n/* 30180 */\r\n/* 30181 */       isNull2 = false;\r\n/* 30182 */       value2 = -1;\r\n/* 30183 */     }\r\n/* 30184 */\r\n/* 30185 */     else {\r\n/* 30186 */\r\n/* 30187 */\r\n/* 30188 */       boolean isNull1410 = true;\r\n/* 30189 */       int value1410 = -1;\r\n/* 30190 */\r\n/* 30191 */       caseWhenCondExpr191(i);\r\n/* 30192 */       if (!isNull2113 && value2113) {\r\n/* 30193 */\r\n/* 30194 */         isNull1410 = false;\r\n/* 30195 */         value1410 = -1;\r\n/* 30196 */       }\r\n...\r\n/* 30342 */     if (!isNull2) {\r\n/* 30343 */\r\n/* 30344 */\r\n/* 30345 */       isNull1 = false; // resultCode could change nullability.\r\n/* 30346 */       value1 = value2 == 0;\r\n/* 30347 */\r\n/* 30348 */     }\r\n/* 30349 */     isNull2815 = isNull1;\r\n/* 30350 */     value2815 = value1;\r\n/* 30351 */   }\r\n...\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nAdded new test suites into `CodeGenerationSuite` and `DataFrameSuite`\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/344558810","html_url":"https://github.com/apache/spark/pull/18641#issuecomment-344558810","issue_url":"https://api.github.com/repos/apache/spark/issues/18641","id":344558810,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-11-15T11:00:29Z","updated_at":"2017-11-15T11:00:29Z","author_association":"NONE","body":"**[Test build #83885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83885/testReport)** for PR 18641 at commit [`5466ef0`](https://github.com/apache/spark/commit/5466ef0914e8e702d43019995067ef49ecb90696).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}