{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19662","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19662/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19662/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19662/events","html_url":"https://github.com/apache/spark/pull/19662","id":271359014,"number":19662,"title":"[SPARK-22446][SQL][ML] Declare StringIndexerModel indexer udf as nondeterministic","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars1.githubusercontent.com/u/68855?v=4","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-11-06T04:48:43Z","updated_at":"2017-11-06T07:45:50Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19662","html_url":"https://github.com/apache/spark/pull/19662","diff_url":"https://github.com/apache/spark/pull/19662.diff","patch_url":"https://github.com/apache/spark/pull/19662.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nUDFs that can cause runtime exception on invalid data are not safe to pushdown, because its behavior depends on its position in the query plan. Pushdown of it will risk to change its original behavior.\r\n\r\nThe example reported in the JIRA and taken as test case shows this issue. We should declare UDFs that can cause runtime exception on invalid data as non-determinstic.\r\n\r\nThis updates the document of `deterministic` property in `Expression` and states clearly an UDF that can cause runtime exception on some specific input, should be declared as non-determinstic.\r\n\r\n## How was this patch tested?\r\n\r\nAdded test. Manually test."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/342069872","html_url":"https://github.com/apache/spark/pull/19662#issuecomment-342069872","issue_url":"https://api.github.com/repos/apache/spark/issues/19662","id":342069872,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-11-06T07:45:50Z","updated_at":"2017-11-06T07:45:50Z","author_association":"NONE","body":"**[Test build #83471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83471/testReport)** for PR 19662 at commit [`d9bdf25`](https://github.com/apache/spark/commit/d9bdf257164dfe32381d47d75b9e3dad72cd76b5).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}