{"action":"created","comment":{"url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/comments/152072963","pull_request_review_id":77878632,"id":152072963,"diff_hunk":"@@ -105,4 +166,102 @@ public void close() {\n       }\n     };\n   }\n+\n+  /**\n+   * Get output schema in avro format\n+   *\n+   * @param keySchema the connect schema of the key\n+   * @param valueSchema the connect schema of the value\n+   * @return output schema in avro format\n+   */\n+  private org.apache.avro.Schema mergeSchema(Schema keySchema, Schema valueSchema) {\n+    // Create the connect schema builder\n+    SchemaBuilder builder = SchemaBuilder\n+            .struct()\n+            .name(valueSchema.name())\n+            .doc(valueSchema.doc())\n+            .version(valueSchema.version());\n+\n+    // Process the key schema\n+    if (keySchema.type() == Schema.Type.STRUCT) {\n+      for (Field field: keySchema.fields()) {\n+        builder.field(field.name(), field.schema());\n+      }\n+    } else if (supportedSchemaTypes.contains(keySchema.type())) {\n+      builder.field(\"key\", keySchema);\n+    } else {\n+      throw new ConnectException(\n+              \"Key type: \" + keySchema.type().toString() + \"not supported!\"\n+      );\n+    }\n+\n+    // Process the value schema\n+    if (valueSchema.type() == Schema.Type.STRUCT) {\n+      for (Field field: valueSchema.fields()) {\n+        builder.field(field.name(), field.schema());\n+      }\n+    } else if (supportedSchemaTypes.contains(valueSchema.type())) {\n+      builder.field(\"value\", valueSchema);\n+    } else {\n+      throw new ConnectException(\n+              \"Value type: \" + valueSchema.type().toString() + \"not supported!\"\n+      );\n+    }\n+","path":"kafka-connect-s3/src/main/java/io/confluent/connect/s3/format/avro/AvroRecordWriterProvider.java","position":184,"original_position":184,"commit_id":"92901d4516621d464e5a7ff02c6ded907e4cb1be","original_commit_id":"92901d4516621d464e5a7ff02c6ded907e4cb1be","user":{"login":"bellemare","id":8051579,"avatar_url":"https://avatars1.githubusercontent.com/u/8051579?v=4","gravatar_id":"","url":"https://api.github.com/users/bellemare","html_url":"https://github.com/bellemare","followers_url":"https://api.github.com/users/bellemare/followers","following_url":"https://api.github.com/users/bellemare/following{/other_user}","gists_url":"https://api.github.com/users/bellemare/gists{/gist_id}","starred_url":"https://api.github.com/users/bellemare/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bellemare/subscriptions","organizations_url":"https://api.github.com/users/bellemare/orgs","repos_url":"https://api.github.com/users/bellemare/repos","events_url":"https://api.github.com/users/bellemare/events{/privacy}","received_events_url":"https://api.github.com/users/bellemare/received_events","type":"User","site_admin":false},"body":"Agreed - need to get some real numbers on this all. We can mitigate this with parallelism... I think. Right @johnnycaol ? We can run multiple connector sinks per partition?","created_at":"2017-11-20T18:30:28Z","updated_at":"2017-11-20T18:30:28Z","html_url":"https://github.com/wishabi/kafka-connect-storage-cloud/pull/1#discussion_r152072963","pull_request_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/comments/152072963"},"html":{"href":"https://github.com/wishabi/kafka-connect-storage-cloud/pull/1#discussion_r152072963"},"pull_request":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1"}},"in_reply_to_id":152064178},"pull_request":{"url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1","id":152871079,"html_url":"https://github.com/wishabi/kafka-connect-storage-cloud/pull/1","diff_url":"https://github.com/wishabi/kafka-connect-storage-cloud/pull/1.diff","patch_url":"https://github.com/wishabi/kafka-connect-storage-cloud/pull/1.patch","issue_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/1","number":1,"state":"open","locked":false,"title":"[Feature] DATA-4844: Sink both Key and Value","user":{"login":"johnnycaol","id":3135448,"avatar_url":"https://avatars1.githubusercontent.com/u/3135448?v=4","gravatar_id":"","url":"https://api.github.com/users/johnnycaol","html_url":"https://github.com/johnnycaol","followers_url":"https://api.github.com/users/johnnycaol/followers","following_url":"https://api.github.com/users/johnnycaol/following{/other_user}","gists_url":"https://api.github.com/users/johnnycaol/gists{/gist_id}","starred_url":"https://api.github.com/users/johnnycaol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnnycaol/subscriptions","organizations_url":"https://api.github.com/users/johnnycaol/orgs","repos_url":"https://api.github.com/users/johnnycaol/repos","events_url":"https://api.github.com/users/johnnycaol/events{/privacy}","received_events_url":"https://api.github.com/users/johnnycaol/received_events","type":"User","site_admin":false},"body":"#### [JIRA] What Jira tickets does this PR Address?\r\nhttps://jira.wishabi.com/browse/DATA-4844\r\n\r\n#### [SUMMARY] What's this PR do?\r\n- [x] - Customized S3 connector to enable sinking both key and value\r\n- [x] - Add a config `s3.sink.key` to control if you want to sink the key or not\r\n- [x] - Add test cases for primitive types and nested types\r\n- [x] - Currently, it supports the following types\r\n```\r\nSchema.Type.INT8,\r\nSchema.Type.INT16,\r\nSchema.Type.INT32,\r\nSchema.Type.INT64,\r\nSchema.Type.FLOAT32,\r\nSchema.Type.FLOAT64,\r\nSchema.Type.BOOLEAN,\r\nSchema.Type.STRING,\r\nSchema.Type.STRUCT\r\n```\r\n\r\n#### [TEST CASES] How should this be tested?\r\n- [x] - Run `mvn test` to make sure all test cases pass\r\n\r\n","created_at":"2017-11-15T20:00:56Z","updated_at":"2017-11-20T18:30:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9f0837a34078187ae0e1c7a7f58077cc6926c08e","assignee":{"login":"johnnycaol","id":3135448,"avatar_url":"https://avatars1.githubusercontent.com/u/3135448?v=4","gravatar_id":"","url":"https://api.github.com/users/johnnycaol","html_url":"https://github.com/johnnycaol","followers_url":"https://api.github.com/users/johnnycaol/followers","following_url":"https://api.github.com/users/johnnycaol/following{/other_user}","gists_url":"https://api.github.com/users/johnnycaol/gists{/gist_id}","starred_url":"https://api.github.com/users/johnnycaol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnnycaol/subscriptions","organizations_url":"https://api.github.com/users/johnnycaol/orgs","repos_url":"https://api.github.com/users/johnnycaol/repos","events_url":"https://api.github.com/users/johnnycaol/events{/privacy}","received_events_url":"https://api.github.com/users/johnnycaol/received_events","type":"User","site_admin":false},"assignees":[{"login":"johnnycaol","id":3135448,"avatar_url":"https://avatars1.githubusercontent.com/u/3135448?v=4","gravatar_id":"","url":"https://api.github.com/users/johnnycaol","html_url":"https://github.com/johnnycaol","followers_url":"https://api.github.com/users/johnnycaol/followers","following_url":"https://api.github.com/users/johnnycaol/following{/other_user}","gists_url":"https://api.github.com/users/johnnycaol/gists{/gist_id}","starred_url":"https://api.github.com/users/johnnycaol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnnycaol/subscriptions","organizations_url":"https://api.github.com/users/johnnycaol/orgs","repos_url":"https://api.github.com/users/johnnycaol/repos","events_url":"https://api.github.com/users/johnnycaol/events{/privacy}","received_events_url":"https://api.github.com/users/johnnycaol/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1/commits","review_comments_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1/comments","review_comment_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/comments{/number}","comments_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/1/comments","statuses_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/statuses/92901d4516621d464e5a7ff02c6ded907e4cb1be","head":{"label":"wishabi:sink-key-value","ref":"sink-key-value","sha":"92901d4516621d464e5a7ff02c6ded907e4cb1be","user":{"login":"wishabi","id":1201233,"avatar_url":"https://avatars3.githubusercontent.com/u/1201233?v=4","gravatar_id":"","url":"https://api.github.com/users/wishabi","html_url":"https://github.com/wishabi","followers_url":"https://api.github.com/users/wishabi/followers","following_url":"https://api.github.com/users/wishabi/following{/other_user}","gists_url":"https://api.github.com/users/wishabi/gists{/gist_id}","starred_url":"https://api.github.com/users/wishabi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wishabi/subscriptions","organizations_url":"https://api.github.com/users/wishabi/orgs","repos_url":"https://api.github.com/users/wishabi/repos","events_url":"https://api.github.com/users/wishabi/events{/privacy}","received_events_url":"https://api.github.com/users/wishabi/received_events","type":"Organization","site_admin":false},"repo":{"id":110867888,"name":"kafka-connect-storage-cloud","full_name":"wishabi/kafka-connect-storage-cloud","owner":{"login":"wishabi","id":1201233,"avatar_url":"https://avatars3.githubusercontent.com/u/1201233?v=4","gravatar_id":"","url":"https://api.github.com/users/wishabi","html_url":"https://github.com/wishabi","followers_url":"https://api.github.com/users/wishabi/followers","following_url":"https://api.github.com/users/wishabi/following{/other_user}","gists_url":"https://api.github.com/users/wishabi/gists{/gist_id}","starred_url":"https://api.github.com/users/wishabi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wishabi/subscriptions","organizations_url":"https://api.github.com/users/wishabi/orgs","repos_url":"https://api.github.com/users/wishabi/repos","events_url":"https://api.github.com/users/wishabi/events{/privacy}","received_events_url":"https://api.github.com/users/wishabi/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/wishabi/kafka-connect-storage-cloud","description":"Kafka Connect suite of connectors for Cloud storage (currently including Amazon S3)","fork":true,"url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud","forks_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/forks","keys_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/teams","hooks_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/hooks","issue_events_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/events{/number}","events_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/events","assignees_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/assignees{/user}","branches_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/branches{/branch}","tags_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/tags","blobs_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/refs{/sha}","trees_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/statuses/{sha}","languages_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/languages","stargazers_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/stargazers","contributors_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/contributors","subscribers_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/subscribers","subscription_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/subscription","commits_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/commits{/sha}","git_commits_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/commits{/sha}","comments_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/comments{/number}","issue_comment_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/comments{/number}","contents_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/contents/{+path}","compare_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/merges","archive_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/downloads","issues_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues{/number}","pulls_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls{/number}","milestones_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/milestones{/number}","notifications_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/labels{/name}","releases_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/releases{/id}","deployments_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/deployments","created_at":"2017-11-15T17:59:17Z","updated_at":"2017-11-15T17:59:20Z","pushed_at":"2017-11-15T20:00:57Z","git_url":"git://github.com/wishabi/kafka-connect-storage-cloud.git","ssh_url":"git@github.com:wishabi/kafka-connect-storage-cloud.git","clone_url":"https://github.com/wishabi/kafka-connect-storage-cloud.git","svn_url":"https://github.com/wishabi/kafka-connect-storage-cloud","homepage":"","size":567,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"wishabi:3.3.0-post","ref":"3.3.0-post","sha":"260ddf9bd0e9302c6d8bffef2481f857b575bc40","user":{"login":"wishabi","id":1201233,"avatar_url":"https://avatars3.githubusercontent.com/u/1201233?v=4","gravatar_id":"","url":"https://api.github.com/users/wishabi","html_url":"https://github.com/wishabi","followers_url":"https://api.github.com/users/wishabi/followers","following_url":"https://api.github.com/users/wishabi/following{/other_user}","gists_url":"https://api.github.com/users/wishabi/gists{/gist_id}","starred_url":"https://api.github.com/users/wishabi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wishabi/subscriptions","organizations_url":"https://api.github.com/users/wishabi/orgs","repos_url":"https://api.github.com/users/wishabi/repos","events_url":"https://api.github.com/users/wishabi/events{/privacy}","received_events_url":"https://api.github.com/users/wishabi/received_events","type":"Organization","site_admin":false},"repo":{"id":110867888,"name":"kafka-connect-storage-cloud","full_name":"wishabi/kafka-connect-storage-cloud","owner":{"login":"wishabi","id":1201233,"avatar_url":"https://avatars3.githubusercontent.com/u/1201233?v=4","gravatar_id":"","url":"https://api.github.com/users/wishabi","html_url":"https://github.com/wishabi","followers_url":"https://api.github.com/users/wishabi/followers","following_url":"https://api.github.com/users/wishabi/following{/other_user}","gists_url":"https://api.github.com/users/wishabi/gists{/gist_id}","starred_url":"https://api.github.com/users/wishabi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wishabi/subscriptions","organizations_url":"https://api.github.com/users/wishabi/orgs","repos_url":"https://api.github.com/users/wishabi/repos","events_url":"https://api.github.com/users/wishabi/events{/privacy}","received_events_url":"https://api.github.com/users/wishabi/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/wishabi/kafka-connect-storage-cloud","description":"Kafka Connect suite of connectors for Cloud storage (currently including Amazon S3)","fork":true,"url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud","forks_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/forks","keys_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/teams","hooks_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/hooks","issue_events_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/events{/number}","events_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/events","assignees_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/assignees{/user}","branches_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/branches{/branch}","tags_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/tags","blobs_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/refs{/sha}","trees_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/statuses/{sha}","languages_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/languages","stargazers_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/stargazers","contributors_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/contributors","subscribers_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/subscribers","subscription_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/subscription","commits_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/commits{/sha}","git_commits_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/git/commits{/sha}","comments_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/comments{/number}","issue_comment_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/comments{/number}","contents_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/contents/{+path}","compare_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/merges","archive_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/downloads","issues_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues{/number}","pulls_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls{/number}","milestones_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/milestones{/number}","notifications_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/labels{/name}","releases_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/releases{/id}","deployments_url":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/deployments","created_at":"2017-11-15T17:59:17Z","updated_at":"2017-11-15T17:59:20Z","pushed_at":"2017-11-15T20:00:57Z","git_url":"git://github.com/wishabi/kafka-connect-storage-cloud.git","ssh_url":"git@github.com:wishabi/kafka-connect-storage-cloud.git","clone_url":"https://github.com/wishabi/kafka-connect-storage-cloud.git","svn_url":"https://github.com/wishabi/kafka-connect-storage-cloud","homepage":"","size":567,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1"},"html":{"href":"https://github.com/wishabi/kafka-connect-storage-cloud/pull/1"},"issue":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/1"},"comments":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/issues/1/comments"},"review_comments":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1/comments"},"review_comment":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/pulls/1/commits"},"statuses":{"href":"https://api.github.com/repos/wishabi/kafka-connect-storage-cloud/statuses/92901d4516621d464e5a7ff02c6ded907e4cb1be"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"bellemare"}}