{"push_id":2115958409,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"f2da738c76810131045e6c32533a2d13526cdaf6","before":"1c923d7d65dd94996f0fe2cf9851a1ae738c5c0c","commits":[{"sha":"5b41cbf13b6d6f47b8f8f1ffcc7a3348018627ca","author":{"name":"Marco Gaido","email":"737a3c370a64b00d8905aa495392d09750dfa37c@hortonworks.com"},"message":"[SPARK-22473][TEST] Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date\n\n## What changes were proposed in this pull request?\n\nIn `spark-sql` module tests there are deprecations warnings caused by the usage of deprecated methods of `java.sql.Date` and the usage of the deprecated `AsyncAssertions.Waiter` class.\nThis PR replace the deprecated methods of `java.sql.Date` with non-deprecated ones (using `Calendar` where needed). It replaces also the deprecated `org.scalatest.concurrent.AsyncAssertions.Waiter` with `org.scalatest.concurrent.Waiters._`.\n\n## How was this patch tested?\n\nexisting UTs\n\nAuthor: Marco Gaido <mgaido@hortonworks.com>\n\nCloses #19696 from mgaido91/SPARK-22473.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/5b41cbf13b6d6f47b8f8f1ffcc7a3348018627ca"},{"sha":"b70aa9e08b4476746e912c2c2a8b7bdd102305e8","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-22344][SPARKR] clean up install dir if running test as source package\n\n## What changes were proposed in this pull request?\n\nremove spark if spark downloaded & installed\n\n## How was this patch tested?\n\nmanually by building package\nJenkins, AppVeyor\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #19657 from felixcheung/rinstalldir.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/b70aa9e08b4476746e912c2c2a8b7bdd102305e8"},{"sha":"5ebdcd185f2108a90e37a1aa4214c3b6c69a97a4","author":{"name":"Santiago Saavedra","email":"984b3f1be5be99e557545b24b73992d6a5bd031d@openshine.com"},"message":"[SPARK-22294][DEPLOY] Reset spark.driver.bindAddress when starting a Checkpoint\n\n## What changes were proposed in this pull request?\n\nIt seems that recovering from a checkpoint can replace the old\ndriver and executor IP addresses, as the workload can now be taking\nplace in a different cluster configuration. It follows that the\nbindAddress for the master may also have changed. Thus we should not be\nkeeping the old one, and instead be added to the list of properties to\nreset and recreate from the new environment.\n\n## How was this patch tested?\n\nThis patch was tested via manual testing on AWS, using the experimental (not yet merged) Kubernetes scheduler, which uses bindAddress to bind to a Kubernetes service (and thus was how I first encountered the bug too), but it is not a code-path related to the scheduler and this may have slipped through when merging SPARK-4563.\n\nAuthor: Santiago Saavedra <ssaavedra@openshine.com>\n\nCloses #19427 from ssaavedra/fix-checkpointing-master.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/5ebdcd185f2108a90e37a1aa4214c3b6c69a97a4"},{"sha":"24ea781cd30fbc611c714b5c6931f7d993f3a08d","author":{"name":"Shixiong Zhu","email":"c8f54caa5df4e9592a01136ca859e0af6b5a2c2a@gmail.com"},"message":"[SPARK-19644][SQL] Clean up Scala reflection garbage after creating Encoder\n\n## What changes were proposed in this pull request?\n\nBecause of the memory leak issue in `scala.reflect.api.Types.TypeApi.<:<` (https://github.com/scala/bug/issues/8302), creating an encoder may leak memory.\n\nThis PR adds `cleanUpReflectionObjects` to clean up these leaking objects for methods calling `scala.reflect.api.Types.TypeApi.<:<`.\n\n## How was this patch tested?\n\nThe updated unit tests.\n\nAuthor: Shixiong Zhu <zsxwing@gmail.com>\n\nCloses #19687 from zsxwing/SPARK-19644.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/24ea781cd30fbc611c714b5c6931f7d993f3a08d"},{"sha":"f2da738c76810131045e6c32533a2d13526cdaf6","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem in calculating hash for nested structs\n\n## What changes were proposed in this pull request?\n\nThis PR avoids to generate a huge method for calculating a murmur3 hash for nested structs. This PR splits a huge method (e.g. `apply_4`) into multiple smaller methods.\n\nSample program\n```\n  val structOfString = new StructType().add(\"str\", StringType)\n  var inner = new StructType()\n  for (_ <- 0 until 800) {\n    inner = inner1.add(\"structOfString\", structOfString)\n  }\n  var schema = new StructType()\n  for (_ <- 0 until 50) {\n    schema = schema.add(\"structOfStructOfStrings\", inner)\n  }\n  GenerateMutableProjection.generate(Seq(Murmur3Hash(exprs, 42)))\n```\n\nWithout this PR\n```\n/* 005 */ class SpecificMutableProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseMutableProjection {\n/* 006 */\n/* 007 */   private Object[] references;\n/* 008 */   private InternalRow mutableRow;\n/* 009 */   private int value;\n/* 010 */   private int value_0;\n...\n/* 034 */   public java.lang.Object apply(java.lang.Object _i) {\n/* 035 */     InternalRow i = (InternalRow) _i;\n/* 036 */\n/* 037 */\n/* 038 */\n/* 039 */     value = 42;\n/* 040 */     apply_0(i);\n/* 041 */     apply_1(i);\n/* 042 */     apply_2(i);\n/* 043 */     apply_3(i);\n/* 044 */     apply_4(i);\n/* 045 */     nestedClassInstance.apply_5(i);\n...\n/* 089 */     nestedClassInstance8.apply_49(i);\n/* 090 */     value_0 = value;\n/* 091 */\n/* 092 */     // copy all the results into MutableRow\n/* 093 */     mutableRow.setInt(0, value_0);\n/* 094 */     return mutableRow;\n/* 095 */   }\n/* 096 */\n/* 097 */\n/* 098 */   private void apply_4(InternalRow i) {\n/* 099 */\n/* 100 */     boolean isNull5 = i.isNullAt(4);\n/* 101 */     InternalRow value5 = isNull5 ? null : (i.getStruct(4, 800));\n/* 102 */     if (!isNull5) {\n/* 103 */\n/* 104 */       if (!value5.isNullAt(0)) {\n/* 105 */\n/* 106 */         final InternalRow element6400 = value5.getStruct(0, 1);\n/* 107 */\n/* 108 */         if (!element6400.isNullAt(0)) {\n/* 109 */\n/* 110 */           final UTF8String element6401 = element6400.getUTF8String(0);\n/* 111 */           value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element6401.getBaseObject(), element6401.getBaseOffset(), element6401.numBytes(), value);\n/* 112 */\n/* 113 */         }\n/* 114 */\n/* 115 */\n/* 116 */       }\n/* 117 */\n/* 118 */\n/* 119 */       if (!value5.isNullAt(1)) {\n/* 120 */\n/* 121 */         final InternalRow element6402 = value5.getStruct(1, 1);\n/* 122 */\n/* 123 */         if (!element6402.isNullAt(0)) {\n/* 124 */\n/* 125 */           final UTF8String element6403 = element6402.getUTF8String(0);\n/* 126 */           value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element6403.getBaseObject(), element6403.getBaseOffset(), element6403.numBytes(), value);\n/* 127 */\n/* 128 */         }\n/* 128 */         }\n/* 129 */\n/* 130 */\n/* 131 */       }\n/* 132 */\n/* 133 */\n/* 134 */       if (!value5.isNullAt(2)) {\n/* 135 */\n/* 136 */         final InternalRow element6404 = value5.getStruct(2, 1);\n/* 137 */\n/* 138 */         if (!element6404.isNullAt(0)) {\n/* 139 */\n/* 140 */           final UTF8String element6405 = element6404.getUTF8String(0);\n/* 141 */           value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element6405.getBaseObject(), element6405.getBaseOffset(), element6405.numBytes(), value);\n/* 142 */\n/* 143 */         }\n/* 144 */\n/* 145 */\n/* 146 */       }\n/* 147 */\n...\n/* 12074 */       if (!value5.isNullAt(798)) {\n/* 12075 */\n/* 12076 */         final InternalRow element7996 = value5.getStruct(798, 1);\n/* 12077 */\n/* 12078 */         if (!element7996.isNullAt(0)) {\n/* 12079 */\n/* 12080 */           final UTF8String element7997 = element7996.getUTF8String(0);\n/* 12083 */         }\n/* 12084 */\n/* 12085 */\n/* 12086 */       }\n/* 12087 */\n/* 12088 */\n/* 12089 */       if (!value5.isNullAt(799)) {\n/* 12090 */\n/* 12091 */         final InternalRow element7998 = value5.getStruct(799, 1);\n/* 12092 */\n/* 12093 */         if (!element7998.isNullAt(0)) {\n/* 12094 */\n/* 12095 */           final UTF8String element7999 = element7998.getUTF8String(0);\n/* 12096 */           value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element7999.getBaseObject(), element7999.getBaseOffset(), element7999.numBytes(), value);\n/* 12097 */\n/* 12098 */         }\n/* 12099 */\n/* 12100 */\n/* 12101 */       }\n/* 12102 */\n/* 12103 */     }\n/* 12104 */\n/* 12105 */   }\n/* 12106 */\n/* 12106 */\n/* 12107 */\n/* 12108 */   private void apply_1(InternalRow i) {\n...\n```\n\nWith this PR\n```\n/* 005 */ class SpecificMutableProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseMutableProjection {\n/* 006 */\n/* 007 */   private Object[] references;\n/* 008 */   private InternalRow mutableRow;\n/* 009 */   private int value;\n/* 010 */   private int value_0;\n/* 011 */\n...\n/* 034 */   public java.lang.Object apply(java.lang.Object _i) {\n/* 035 */     InternalRow i = (InternalRow) _i;\n/* 036 */\n/* 037 */\n/* 038 */\n/* 039 */     value = 42;\n/* 040 */     nestedClassInstance11.apply50_0(i);\n/* 041 */     nestedClassInstance11.apply50_1(i);\n...\n/* 088 */     nestedClassInstance11.apply50_48(i);\n/* 089 */     nestedClassInstance11.apply50_49(i);\n/* 090 */     value_0 = value;\n/* 091 */\n/* 092 */     // copy all the results into MutableRow\n/* 093 */     mutableRow.setInt(0, value_0);\n/* 094 */     return mutableRow;\n/* 095 */   }\n/* 096 */\n...\n/* 37717 */   private void apply4_0(InternalRow value5, InternalRow i) {\n/* 37718 */\n/* 37719 */     if (!value5.isNullAt(0)) {\n/* 37720 */\n/* 37721 */       final InternalRow element6400 = value5.getStruct(0, 1);\n/* 37722 */\n/* 37723 */       if (!element6400.isNullAt(0)) {\n/* 37724 */\n/* 37725 */         final UTF8String element6401 = element6400.getUTF8String(0);\n/* 37726 */         value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element6401.getBaseObject(), element6401.getBaseOffset(), element6401.numBytes(), value);\n/* 37727 */\n/* 37728 */       }\n/* 37729 */\n/* 37730 */\n/* 37731 */     }\n/* 37732 */\n/* 37733 */     if (!value5.isNullAt(1)) {\n/* 37734 */\n/* 37735 */       final InternalRow element6402 = value5.getStruct(1, 1);\n/* 37736 */\n/* 37737 */       if (!element6402.isNullAt(0)) {\n/* 37738 */\n/* 37739 */         final UTF8String element6403 = element6402.getUTF8String(0);\n/* 37740 */         value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element6403.getBaseObject(), element6403.getBaseOffset(), element6403.numBytes(), value);\n/* 37741 */\n/* 37742 */       }\n/* 37743 */\n/* 37744 */\n/* 37745 */     }\n/* 37746 */\n/* 37747 */     if (!value5.isNullAt(2)) {\n/* 37748 */\n/* 37749 */       final InternalRow element6404 = value5.getStruct(2, 1);\n/* 37750 */\n/* 37751 */       if (!element6404.isNullAt(0)) {\n/* 37752 */\n/* 37753 */         final UTF8String element6405 = element6404.getUTF8String(0);\n/* 37754 */         value = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(element6405.getBaseObject(), element6405.getBaseOffset(), element6405.numBytes(), value);\n/* 37755 */\n/* 37756 */       }\n/* 37757 */\n/* 37758 */\n/* 37759 */     }\n/* 37760 */\n/* 37761 */   }\n...\n/* 218470 */\n/* 218471 */     private void apply50_4(InternalRow i) {\n/* 218472 */\n/* 218473 */       boolean isNull5 = i.isNullAt(4);\n/* 218474 */       InternalRow value5 = isNull5 ? null : (i.getStruct(4, 800));\n/* 218475 */       if (!isNull5) {\n/* 218476 */         apply4_0(value5, i);\n/* 218477 */         apply4_1(value5, i);\n/* 218478 */         apply4_2(value5, i);\n...\n/* 218742 */         nestedClassInstance.apply4_266(value5, i);\n/* 218743 */       }\n/* 218744 */\n/* 218745 */     }\n```\n\n## How was this patch tested?\n\nAdded new test to `HashExpressionsSuite`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #19563 from kiszk/SPARK-22284.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/f2da738c76810131045e6c32533a2d13526cdaf6"}]}
{"actor":{"display_login":"ueshin"}}