{"push_id":1584524037,"size":3,"distinct_size":3,"ref":"refs/heads/master","head":"b405466513bcc02cadf1477b6b682ace95d81658","before":"73530383538ad72fdc3dd4c670485192f12ebc4e","commits":[{"sha":"a350bc16d36c58b48ac01f0258678ffcdb77e793","author":{"name":"windpiger","email":"83bba5ef3ac29b005cb86c055a312c08a5e441d9@outlook.com"},"message":"[SPARK-19748][SQL] refresh function has a wrong order to do cache invalidate and regenerate the inmemory var for InMemoryFileIndex with FileStatusCache\n\n## What changes were proposed in this pull request?\n\nIf we refresh a InMemoryFileIndex with a FileStatusCache, it will first use the FileStatusCache to re-generate the cachedLeafFiles etc, then call FileStatusCache.invalidateAll.\n\nWhile the order to do these two actions is wrong, this lead to the refresh action does not take effect.\n\n```\n  override def refresh(): Unit = {\n    refresh0()\n    fileStatusCache.invalidateAll()\n  }\n\n  private def refresh0(): Unit = {\n    val files = listLeafFiles(rootPaths)\n    cachedLeafFiles =\n      new mutable.LinkedHashMap[Path, FileStatus]() ++= files.map(f => f.getPath -> f)\n    cachedLeafDirToChildrenFiles = files.toArray.groupBy(_.getPath.getParent)\n    cachedPartitionSpec = null\n  }\n```\n## How was this patch tested?\nunit test added\n\nAuthor: windpiger <songjun@outlook.com>\n\nCloses #17079 from windpiger/fixInMemoryFileIndexRefresh.","distinct":true,"url":"https://api.github.com/repos/wangyum/spark/commits/a350bc16d36c58b48ac01f0258678ffcdb77e793"},{"sha":"9b8eca65dcf68129470ead39362ce870ffb0bb1d","author":{"name":"Yuming Wang","email":"b4045a50547081d83941485c6596bd2032665522@gmail.com"},"message":"[SPARK-19660][CORE][SQL] Replace the configuration property names that are deprecated in the version of Hadoop 2.6\n\n## What changes were proposed in this pull request?\n\nReplace all the Hadoop deprecated configuration property names according to [DeprecatedProperties](https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).\n\nexcept:\nhttps://github.com/apache/spark/blob/v2.1.0/python/pyspark/sql/tests.py#L1533\nhttps://github.com/apache/spark/blob/v2.1.0/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala#L987\nhttps://github.com/apache/spark/blob/v2.1.0/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala#L45\nhttps://github.com/apache/spark/blob/v2.1.0/sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L614\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Yuming Wang <wgyumg@gmail.com>\n\nCloses #16990 from wangyum/HadoopDeprecatedProperties.","distinct":true,"url":"https://api.github.com/repos/wangyum/spark/commits/9b8eca65dcf68129470ead39362ce870ffb0bb1d"},{"sha":"b405466513bcc02cadf1477b6b682ace95d81658","author":{"name":"Nick Pentreath","email":"01c7185e211fff3925ce587674cd3ea63724775a@za.ibm.com"},"message":"[SPARK-14489][ML][PYSPARK] ALS unknown user/item prediction strategy\n\nThis PR adds a param to `ALS`/`ALSModel` to set the strategy used when encountering unknown users or items at prediction time in `transform`. This can occur in 2 scenarios: (a) production scoring, and (b) cross-validation & evaluation.\n\nThe current behavior returns `NaN` if a user/item is unknown. In scenario (b), this can easily occur when using `CrossValidator` or `TrainValidationSplit` since some users/items may only occur in the test set and not in the training set. In this case, the evaluator returns `NaN` for all metrics, making model selection impossible.\n\nThe new param, `coldStartStrategy`, defaults to `nan` (the current behavior). The other option supported initially is `drop`, which drops all rows with `NaN` predictions. This flag allows users to use `ALS` in cross-validation settings. It is made an `expertParam`. The param is made a string so that the set of strategies can be extended in future (some options are discussed in [SPARK-14489](https://issues.apache.org/jira/browse/SPARK-14489)).\n## How was this patch tested?\n\nNew unit tests, and manual \"before and after\" tests for Scala & Python using MovieLens `ml-latest-small` as example data. Here, using `CrossValidator` or `TrainValidationSplit` with the default param setting results in metrics that are all `NaN`, while setting `coldStartStrategy` to `drop` results in valid metrics.\n\nAuthor: Nick Pentreath <nickp@za.ibm.com>\n\nCloses #12896 from MLnick/SPARK-14489-als-nan.","distinct":true,"url":"https://api.github.com/repos/wangyum/spark/commits/b405466513bcc02cadf1477b6b682ace95d81658"}]}
{"actor":{"display_login":"wangyum"}}