{"action":"created","comment":{"url":"https://api.github.com/repos/debezium/debezium.github.io/pulls/comments/140774386","pull_request_review_id":64881560,"id":140774386,"diff_hunk":"@@ -0,0 +1,222 @@\n+= Streaming data to a downstream database\n+Jiri Pechanec\n+:awestruct-tags: [ mysql, postgres, smt, example ]\n+:awestruct-layout: blog-post\n+\n+Recently there was a short https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/[series] of blogposts on https://www.confluent.io/[Confluent's] blog.\n+\n+In this blogpost we will try to create a simple scenario: we are going to have a MySQL database instance running as a master database that feeds a Kafka broker with events representing a source of truth.\n+Next we will add PostgreSQL database that will listen one of the topics and will replicate the state of the MySQL instance to its own data.\n+\n+We will also demonstrate a new functionality coming with version link:2017/09/21/debezium-0-6-0-released[0.6.0]: a single message transform for link:/docs/configuration/event-flattening[CDC Event Flattening].\n+\n+== Topology\n+The general topology for this scenario is displayed on the following picture:\n+\n+.A General topology\n+[#img-general]\n+[caption=\"Figure 1: \"]\n+image::dbz-to-jdbc.svg[Scenario topology]\n+\n+To simplify the setup a little bit we will use only one Kafka Connect instance that will contain all connectors so the single instance will serve as an event producer and an event consumer.\n+\n+.A Simplified topology\n+[#img-general]\n+[caption=\"Figure 2: \"]\n+image::dbz-to-jdbc-simplified.svg[Scenario topology]\n+\n+== Configuration\n+We will use this https://github.com/debezium/debezium-examples/tree/master/unwrap-smt[compose] for a fast deployment of the demo.\n+The deployment consists of following images\n+\n+* Apache ZooKeeper\n+* Apache Kafka\n+* An enriched Kafka Connect image with changes\n+** PostgreSQL JDBC driver placed into `/kafka/libs` directory\n+** https://docs.confluent.io/current/connect/connect-jdbc/docs/index.html[Kafka Connect JDBC Connector] placed into `/kafka/connect/kafka-connect-jdbc` directory\n+* Pre-populated MySQL used in our link:docs/tutorial[tutorial]\n+* Empty PostgreSQL\n+\n+There are certain impedances between Debezium MySQL Connector and Kafka Connect JDBC connector\n+\n+* event format\n+* delete records/tombstone records handling\n+* topic naming\n+\n+We are going to go through all those issues in detail.\n+\n+=== Event format\n+Debezium emits events in a complex format that contains two record values: a value of the row record before the change and value of the row record after the change is applied.\n+Further metadata are also included and the result is so called `Envelope` format.\n+\n+[source,json,indent=0]\n+----\n+{\n+\t\"op\": \"u\",\n+\t\"source\": {\n+\t\t...\n+\t},\n+\t\"ts_ms\" : \"...\",\n+\t\"before\" : {\n+\t\t\"field1\" : \"oldvalue1\",\n+\t\t\"field2\" : \"oldvalue2\"\n+\t},\n+\t\"after\" : {\n+\t\t\"field1\" : \"newvalue1\",\n+\t\t\"field2\" : \"newvalue2\"\n+\t}\n+}\n+----\n+\n+Kafka Connect JDBC connector (as most of the others) emits outgoing and expects incoming events in a simple format that is generally a new row record.\n+\n+[source,json,indent=0]\n+----\n+{\n+\t\"field1\" : \"newvalue1\",\n+\t\"field2\" : \"newvalue2\"\n+}\n+----\n+\n+Debezium provides a bridge between those two formats in a form of a https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect[single message transform].\n+The `UnwrapFromEnvelope` transformation automatically extracts a new row record and thus effectively _flattens_ the complex record into a simple one consumable by other connectors.\n+The SMT is applicable at both source and sink connectors but we generally recommend to apply it at sink connectors as then the records stored in Kafka will contain an associated context.\n+In our example we apply the SMT at sink connector using configuration properties\n+\n+[source,indent=0]\n+----\n+        \"transforms\": \"unwrap\", \n+        \"transforms.unwrap.type\": \"io.debezium.transforms.UnwrapFromEnvelope\", \n+----\n+\n+=== Delete records\n+When a record is deleted from source database Debezium creates two records: delete record and tombstone record (for compaction).\n+Kafka Connect JDBC Sink is not able to work with neither of those to and any attempt to use them leads to `NullPointerException`.\n+`UnwrapFromEnvelope` by default filters out both delete and tombstone records and no action is requested on users side.\n+If a downstream connector is able to handle those events it is possible to configure `UnwrapFromEnvelope` to pass them downstream.\n+\n+=== Topic naming\n+Last but not least there is a difference in naming of topics.\n+Debezium uses fully qualified naming for target topics representing each table it manages.\n+The naming follows the pattern `<instance-name>,<database-name>.<table-name>`.\n+Kafka Connect JDBC Connector works with simple names `<table-name>`.\n+\n+In complex scenarios the user would deploy https://kafka.apache.org/documentation/streams/[Kafka Streams] framework to establish elaborated routing between source and target routes.","path":"blog/2017-09-25-streaming-to-another-database.adoc","position":104,"original_position":104,"commit_id":"5beee8fa90da4f739fc934d030b1ad7d8ad219f5","original_commit_id":"5beee8fa90da4f739fc934d030b1ad7d8ad219f5","user":{"login":"rhauch","id":84732,"avatar_url":"https://avatars0.githubusercontent.com/u/84732?v=4","gravatar_id":"","url":"https://api.github.com/users/rhauch","html_url":"https://github.com/rhauch","followers_url":"https://api.github.com/users/rhauch/followers","following_url":"https://api.github.com/users/rhauch/following{/other_user}","gists_url":"https://api.github.com/users/rhauch/gists{/gist_id}","starred_url":"https://api.github.com/users/rhauch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhauch/subscriptions","organizations_url":"https://api.github.com/users/rhauch/orgs","repos_url":"https://api.github.com/users/rhauch/repos","events_url":"https://api.github.com/users/rhauch/events{/privacy}","received_events_url":"https://api.github.com/users/rhauch/received_events","type":"User","site_admin":false},"body":"> In a more complex pipeline use case, we might use https://kafka.apache.org/documentation/streams/[Kafka Streams] to perform a more complex transformation, aggregation, and/or join of multiple topics, and this same Streams application could also route the messages to the correct topic.\r\n>\r\n> But in our simpler example we can instead use Kafka Connect's built-in `RegexRouter` SMT to send the messages to a different topic.\r\n\r\n@jpechane, might be useful to say whether this SMT will be used in the source or sink connector. IIUC, you are using it in the source connector, and if that's right you might add something like this to the previous paragraph: \"Again, we could use this SMT in either the source or sink connectors, but for this example we're going to use it in the source connector so we can choose the names of the Kafka topics where the records will be written.\"","created_at":"2017-09-25T13:26:35Z","updated_at":"2017-09-25T13:36:05Z","html_url":"https://github.com/debezium/debezium.github.io/pull/74#discussion_r140774386","pull_request_url":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/debezium/debezium.github.io/pulls/comments/140774386"},"html":{"href":"https://github.com/debezium/debezium.github.io/pull/74#discussion_r140774386"},"pull_request":{"href":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74"}}},"pull_request":{"url":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74","id":142869465,"html_url":"https://github.com/debezium/debezium.github.io/pull/74","diff_url":"https://github.com/debezium/debezium.github.io/pull/74.diff","patch_url":"https://github.com/debezium/debezium.github.io/pull/74.patch","issue_url":"https://api.github.com/repos/debezium/debezium.github.io/issues/74","number":74,"state":"open","locked":false,"title":"Blogspot for DB->DB replication","user":{"login":"jpechane","id":650600,"avatar_url":"https://avatars1.githubusercontent.com/u/650600?v=4","gravatar_id":"","url":"https://api.github.com/users/jpechane","html_url":"https://github.com/jpechane","followers_url":"https://api.github.com/users/jpechane/followers","following_url":"https://api.github.com/users/jpechane/following{/other_user}","gists_url":"https://api.github.com/users/jpechane/gists{/gist_id}","starred_url":"https://api.github.com/users/jpechane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpechane/subscriptions","organizations_url":"https://api.github.com/users/jpechane/orgs","repos_url":"https://api.github.com/users/jpechane/repos","events_url":"https://api.github.com/users/jpechane/events{/privacy}","received_events_url":"https://api.github.com/users/jpechane/received_events","type":"User","site_admin":false},"body":"","created_at":"2017-09-25T12:30:31Z","updated_at":"2017-09-25T13:36:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":"dbbf06a11a9319c727ae4f3674b0e29ef66386d1","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74/commits","review_comments_url":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74/comments","review_comment_url":"https://api.github.com/repos/debezium/debezium.github.io/pulls/comments{/number}","comments_url":"https://api.github.com/repos/debezium/debezium.github.io/issues/74/comments","statuses_url":"https://api.github.com/repos/debezium/debezium.github.io/statuses/5beee8fa90da4f739fc934d030b1ad7d8ad219f5","head":{"label":"jpechane:blog-db2db","ref":"blog-db2db","sha":"5beee8fa90da4f739fc934d030b1ad7d8ad219f5","user":{"login":"jpechane","id":650600,"avatar_url":"https://avatars1.githubusercontent.com/u/650600?v=4","gravatar_id":"","url":"https://api.github.com/users/jpechane","html_url":"https://github.com/jpechane","followers_url":"https://api.github.com/users/jpechane/followers","following_url":"https://api.github.com/users/jpechane/following{/other_user}","gists_url":"https://api.github.com/users/jpechane/gists{/gist_id}","starred_url":"https://api.github.com/users/jpechane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpechane/subscriptions","organizations_url":"https://api.github.com/users/jpechane/orgs","repos_url":"https://api.github.com/users/jpechane/repos","events_url":"https://api.github.com/users/jpechane/events{/privacy}","received_events_url":"https://api.github.com/users/jpechane/received_events","type":"User","site_admin":false},"repo":{"id":96902301,"name":"debezium.github.io","full_name":"jpechane/debezium.github.io","owner":{"login":"jpechane","id":650600,"avatar_url":"https://avatars1.githubusercontent.com/u/650600?v=4","gravatar_id":"","url":"https://api.github.com/users/jpechane","html_url":"https://github.com/jpechane","followers_url":"https://api.github.com/users/jpechane/followers","following_url":"https://api.github.com/users/jpechane/following{/other_user}","gists_url":"https://api.github.com/users/jpechane/gists{/gist_id}","starred_url":"https://api.github.com/users/jpechane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpechane/subscriptions","organizations_url":"https://api.github.com/users/jpechane/orgs","repos_url":"https://api.github.com/users/jpechane/repos","events_url":"https://api.github.com/users/jpechane/events{/privacy}","received_events_url":"https://api.github.com/users/jpechane/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jpechane/debezium.github.io","description":"Source for the Debezium website at http://debezium.io Please log issues in our JIRA at https://issues.jboss.org/projects/DBZ/issues","fork":true,"url":"https://api.github.com/repos/jpechane/debezium.github.io","forks_url":"https://api.github.com/repos/jpechane/debezium.github.io/forks","keys_url":"https://api.github.com/repos/jpechane/debezium.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jpechane/debezium.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jpechane/debezium.github.io/teams","hooks_url":"https://api.github.com/repos/jpechane/debezium.github.io/hooks","issue_events_url":"https://api.github.com/repos/jpechane/debezium.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/jpechane/debezium.github.io/events","assignees_url":"https://api.github.com/repos/jpechane/debezium.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/jpechane/debezium.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/jpechane/debezium.github.io/tags","blobs_url":"https://api.github.com/repos/jpechane/debezium.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jpechane/debezium.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jpechane/debezium.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/jpechane/debezium.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jpechane/debezium.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/jpechane/debezium.github.io/languages","stargazers_url":"https://api.github.com/repos/jpechane/debezium.github.io/stargazers","contributors_url":"https://api.github.com/repos/jpechane/debezium.github.io/contributors","subscribers_url":"https://api.github.com/repos/jpechane/debezium.github.io/subscribers","subscription_url":"https://api.github.com/repos/jpechane/debezium.github.io/subscription","commits_url":"https://api.github.com/repos/jpechane/debezium.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/jpechane/debezium.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/jpechane/debezium.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/jpechane/debezium.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/jpechane/debezium.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/jpechane/debezium.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jpechane/debezium.github.io/merges","archive_url":"https://api.github.com/repos/jpechane/debezium.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jpechane/debezium.github.io/downloads","issues_url":"https://api.github.com/repos/jpechane/debezium.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/jpechane/debezium.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/jpechane/debezium.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/jpechane/debezium.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jpechane/debezium.github.io/labels{/name}","releases_url":"https://api.github.com/repos/jpechane/debezium.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/jpechane/debezium.github.io/deployments","created_at":"2017-07-11T14:27:00Z","updated_at":"2017-07-11T14:27:02Z","pushed_at":"2017-09-25T12:28:47Z","git_url":"git://github.com/jpechane/debezium.github.io.git","ssh_url":"git@github.com:jpechane/debezium.github.io.git","clone_url":"https://github.com/jpechane/debezium.github.io.git","svn_url":"https://github.com/jpechane/debezium.github.io","homepage":"","size":2744,"stargazers_count":0,"watchers_count":0,"language":"CSS","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"debezium:develop","ref":"develop","sha":"89cbc45bac8055f24c96e9dbbcd633a87c822407","user":{"login":"debezium","id":11964329,"avatar_url":"https://avatars2.githubusercontent.com/u/11964329?v=4","gravatar_id":"","url":"https://api.github.com/users/debezium","html_url":"https://github.com/debezium","followers_url":"https://api.github.com/users/debezium/followers","following_url":"https://api.github.com/users/debezium/following{/other_user}","gists_url":"https://api.github.com/users/debezium/gists{/gist_id}","starred_url":"https://api.github.com/users/debezium/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debezium/subscriptions","organizations_url":"https://api.github.com/users/debezium/orgs","repos_url":"https://api.github.com/users/debezium/repos","events_url":"https://api.github.com/users/debezium/events{/privacy}","received_events_url":"https://api.github.com/users/debezium/received_events","type":"Organization","site_admin":false},"repo":{"id":55010150,"name":"debezium.github.io","full_name":"debezium/debezium.github.io","owner":{"login":"debezium","id":11964329,"avatar_url":"https://avatars2.githubusercontent.com/u/11964329?v=4","gravatar_id":"","url":"https://api.github.com/users/debezium","html_url":"https://github.com/debezium","followers_url":"https://api.github.com/users/debezium/followers","following_url":"https://api.github.com/users/debezium/following{/other_user}","gists_url":"https://api.github.com/users/debezium/gists{/gist_id}","starred_url":"https://api.github.com/users/debezium/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debezium/subscriptions","organizations_url":"https://api.github.com/users/debezium/orgs","repos_url":"https://api.github.com/users/debezium/repos","events_url":"https://api.github.com/users/debezium/events{/privacy}","received_events_url":"https://api.github.com/users/debezium/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/debezium/debezium.github.io","description":"Source for the Debezium website at http://debezium.io Please log issues in our JIRA at https://issues.jboss.org/projects/DBZ/issues","fork":false,"url":"https://api.github.com/repos/debezium/debezium.github.io","forks_url":"https://api.github.com/repos/debezium/debezium.github.io/forks","keys_url":"https://api.github.com/repos/debezium/debezium.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/debezium/debezium.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/debezium/debezium.github.io/teams","hooks_url":"https://api.github.com/repos/debezium/debezium.github.io/hooks","issue_events_url":"https://api.github.com/repos/debezium/debezium.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/debezium/debezium.github.io/events","assignees_url":"https://api.github.com/repos/debezium/debezium.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/debezium/debezium.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/debezium/debezium.github.io/tags","blobs_url":"https://api.github.com/repos/debezium/debezium.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/debezium/debezium.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/debezium/debezium.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/debezium/debezium.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/debezium/debezium.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/debezium/debezium.github.io/languages","stargazers_url":"https://api.github.com/repos/debezium/debezium.github.io/stargazers","contributors_url":"https://api.github.com/repos/debezium/debezium.github.io/contributors","subscribers_url":"https://api.github.com/repos/debezium/debezium.github.io/subscribers","subscription_url":"https://api.github.com/repos/debezium/debezium.github.io/subscription","commits_url":"https://api.github.com/repos/debezium/debezium.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/debezium/debezium.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/debezium/debezium.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/debezium/debezium.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/debezium/debezium.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/debezium/debezium.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/debezium/debezium.github.io/merges","archive_url":"https://api.github.com/repos/debezium/debezium.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/debezium/debezium.github.io/downloads","issues_url":"https://api.github.com/repos/debezium/debezium.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/debezium/debezium.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/debezium/debezium.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/debezium/debezium.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/debezium/debezium.github.io/labels{/name}","releases_url":"https://api.github.com/repos/debezium/debezium.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/debezium/debezium.github.io/deployments","created_at":"2016-03-29T20:59:44Z","updated_at":"2016-10-25T22:35:19Z","pushed_at":"2017-09-25T12:30:31Z","git_url":"git://github.com/debezium/debezium.github.io.git","ssh_url":"git@github.com:debezium/debezium.github.io.git","clone_url":"https://github.com/debezium/debezium.github.io.git","svn_url":"https://github.com/debezium/debezium.github.io","homepage":"","size":3509,"stargazers_count":0,"watchers_count":0,"language":"CSS","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":15,"mirror_url":null,"open_issues_count":1,"forks":15,"open_issues":1,"watchers":0,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74"},"html":{"href":"https://github.com/debezium/debezium.github.io/pull/74"},"issue":{"href":"https://api.github.com/repos/debezium/debezium.github.io/issues/74"},"comments":{"href":"https://api.github.com/repos/debezium/debezium.github.io/issues/74/comments"},"review_comments":{"href":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74/comments"},"review_comment":{"href":"https://api.github.com/repos/debezium/debezium.github.io/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/debezium/debezium.github.io/pulls/74/commits"},"statuses":{"href":"https://api.github.com/repos/debezium/debezium.github.io/statuses/5beee8fa90da4f739fc934d030b1ad7d8ad219f5"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"rhauch"}}