{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16692","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16692/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16692/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16692/events","html_url":"https://github.com/apache/spark/pull/16692","id":202798766,"number":16692,"title":"[SPARK-19335] Introduce UPSERT feature to SPARK ","user":{"login":"kevinyu98","id":7550280,"avatar_url":"https://avatars0.githubusercontent.com/u/7550280?v=3","gravatar_id":"","url":"https://api.github.com/users/kevinyu98","html_url":"https://github.com/kevinyu98","followers_url":"https://api.github.com/users/kevinyu98/followers","following_url":"https://api.github.com/users/kevinyu98/following{/other_user}","gists_url":"https://api.github.com/users/kevinyu98/gists{/gist_id}","starred_url":"https://api.github.com/users/kevinyu98/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kevinyu98/subscriptions","organizations_url":"https://api.github.com/users/kevinyu98/orgs","repos_url":"https://api.github.com/users/kevinyu98/repos","events_url":"https://api.github.com/users/kevinyu98/events{/privacy}","received_events_url":"https://api.github.com/users/kevinyu98/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-01-24T11:58:35Z","updated_at":"2017-06-27T06:42:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16692","html_url":"https://github.com/apache/spark/pull/16692","diff_url":"https://github.com/apache/spark/pull/16692.diff","patch_url":"https://github.com/apache/spark/pull/16692.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR proposes to add the UPSERT feature support into SPARK through DataFrameWriter's JDBC data source options. \r\n\r\nFor example:\r\nIf the mytable2 in mysql database have unique constraints on column c1, and the user wants to save the dataframe into the mysql database, it will fail with violation of unique constraints.\r\n\r\n`val df = Seq((1,4)).toDF(\"c1\",\"c2\")`\r\n`val url = \"jdbc:mysql://9.30.167.220:3306/mydb\"`\r\n`df.write.mode(org.apache.spark.sql.SaveMode.Append)\r\n.option(\"user\",\"kevin\").option(\"password\",\"kevin\").jdbc(url,\"mytable2\",new java.util.Properties())`\r\n\r\nWith this feature, the user can use this UPSERT options to write the dataframe into the mysql database table.\r\n\r\n`df.write.mode(org.apache.spark.sql.SaveMode.Append)\r\n.option(“upsert”,true).option(“upsertUpdateColumn”,”c1”).option(\"user\",\"kevin\").option(\"password\",\"kevin\").jdbc(url,\"mytable2\",new java.util.Properties())`\r\n\r\nHere is the design doc.\r\n[UPSERT DESIGN DOC](https://drive.google.com/open?id=1IoafDm78v7ATP-npKbTaw2_dTFiXplCd9NzEe8CBH6E)\r\n\r\n\r\n## How was this patch tested?\r\n\r\nLocal test: run the test case from spark-shell and connect to MySQL and Postgresql database\r\nTest case: add test cases in the existing test cases including docker integration suite\r\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/311267693","html_url":"https://github.com/apache/spark/pull/16692#issuecomment-311267693","issue_url":"https://api.github.com/repos/apache/spark/issues/16692","id":311267693,"user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars2.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"created_at":"2017-06-27T06:42:55Z","updated_at":"2017-06-27T06:42:55Z","body":"We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!"}}
{"actor":{"display_login":"gatorsmile"}}