{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16744","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16744/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16744/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16744/events","html_url":"https://github.com/apache/spark/pull/16744","id":204104662,"number":16744,"title":"[SPARK-19405][STREAMING] Support for cross-account Kinesis reads via STS","user":{"login":"budde","id":798699,"avatar_url":"https://avatars.githubusercontent.com/u/798699?v=3","gravatar_id":"","url":"https://api.github.com/users/budde","html_url":"https://github.com/budde","followers_url":"https://api.github.com/users/budde/followers","following_url":"https://api.github.com/users/budde/following{/other_user}","gists_url":"https://api.github.com/users/budde/gists{/gist_id}","starred_url":"https://api.github.com/users/budde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/budde/subscriptions","organizations_url":"https://api.github.com/users/budde/orgs","repos_url":"https://api.github.com/users/budde/repos","events_url":"https://api.github.com/users/budde/events{/privacy}","received_events_url":"https://api.github.com/users/budde/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2017-01-30T19:23:41Z","updated_at":"2017-02-06T19:18:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16744","html_url":"https://github.com/apache/spark/pull/16744","diff_url":"https://github.com/apache/spark/pull/16744.diff","patch_url":"https://github.com/apache/spark/pull/16744.patch"},"body":"\r\n- Add dependency on aws-java-sdk-sts\r\n- Replace SerializableAWSCredentials with new SerializableKCLAuthProvider class\r\n- Make KinesisReceiver take SerializableKCLAuthProvider as argument and\r\n  pass credential provider to KCL\r\n- Add new implementations of KinesisUtils.createStream() that take STS\r\n  arguments\r\n- Make JavaKinesisStreamSuite test the entire KinesisUtils Java API\r\n\r\n## What changes were proposed in this pull request?\r\n\r\n* Replace SerializableAWSCredentials with new SerializableKCLAuthProvider class that takes 5 optional config params for configuring AWS auth and returns the appropriate credential provider object\r\n* Add new public createStream() APIs for specifying these parameters in KinesisUtils\r\n\r\n## How was this patch tested?\r\n\r\n* Manually tested using explicit keypair and instance profile to read data from Kinesis stream in separate account (difficult to write a test orchestrating creation and assumption of IAM roles across separate accounts)\r\n* Expanded JavaKinesisStreamSuite to test the entire Java API in KinesisUtils\r\n\r\n## License acknowledgement\r\nThis contribution is my original work and that I license the work to the project under the project’s open source license."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/277783348","html_url":"https://github.com/apache/spark/pull/16744#issuecomment-277783348","issue_url":"https://api.github.com/repos/apache/spark/issues/16744","id":277783348,"user":{"login":"budde","id":798699,"avatar_url":"https://avatars.githubusercontent.com/u/798699?v=3","gravatar_id":"","url":"https://api.github.com/users/budde","html_url":"https://github.com/budde","followers_url":"https://api.github.com/users/budde/followers","following_url":"https://api.github.com/users/budde/following{/other_user}","gists_url":"https://api.github.com/users/budde/gists{/gist_id}","starred_url":"https://api.github.com/users/budde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/budde/subscriptions","organizations_url":"https://api.github.com/users/budde/orgs","repos_url":"https://api.github.com/users/budde/repos","events_url":"https://api.github.com/users/budde/events{/privacy}","received_events_url":"https://api.github.com/users/budde/received_events","type":"User","site_admin":false},"created_at":"2017-02-06T19:18:46Z","updated_at":"2017-02-06T19:18:46Z","body":"Amending this PR to upgrade the KCL/AWS SDK dependencies to more-current versions (1.7.3 and 1.11.76, respectively). The ```RegionUtils.getRegionByEndpoint()``` API was removed from the SDK, so I've had to replace it with a simple string split method for the examples and test suites that were utilizing it."}}
{"actor":{"display_login":"budde"}}