{"action":"created","issue":{"url":"https://api.github.com/repos/Teradata/kylo/issues/23","repository_url":"https://api.github.com/repos/Teradata/kylo","labels_url":"https://api.github.com/repos/Teradata/kylo/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/Teradata/kylo/issues/23/comments","events_url":"https://api.github.com/repos/Teradata/kylo/issues/23/events","html_url":"https://github.com/Teradata/kylo/pull/23","id":226734172,"number":23,"title":"[KYLO-803] NiFi - Spark Jobserver Integration","user":{"login":"calvin-pietersen","id":16835507,"avatar_url":"https://avatars2.githubusercontent.com/u/16835507?v=4","gravatar_id":"","url":"https://api.github.com/users/calvin-pietersen","html_url":"https://github.com/calvin-pietersen","followers_url":"https://api.github.com/users/calvin-pietersen/followers","following_url":"https://api.github.com/users/calvin-pietersen/following{/other_user}","gists_url":"https://api.github.com/users/calvin-pietersen/gists{/gist_id}","starred_url":"https://api.github.com/users/calvin-pietersen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvin-pietersen/subscriptions","organizations_url":"https://api.github.com/users/calvin-pietersen/orgs","repos_url":"https://api.github.com/users/calvin-pietersen/repos","events_url":"https://api.github.com/users/calvin-pietersen/events{/privacy}","received_events_url":"https://api.github.com/users/calvin-pietersen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-05-06T03:22:43Z","updated_at":"2017-08-02T00:25:08Z","closed_at":"2017-08-02T00:25:08Z","pull_request":{"url":"https://api.github.com/repos/Teradata/kylo/pulls/23","html_url":"https://github.com/Teradata/kylo/pull/23","diff_url":"https://github.com/Teradata/kylo/pull/23.diff","patch_url":"https://github.com/Teradata/kylo/pull/23.patch"},"body":"Adding integration for long running Spark Contexts in NiFi. \r\nAdded a Controller Service SparkJobserverService and a processor ExecuteSparkContextJob.\r\n\r\nTesting requires spark jobserver and a class implementing SparkJob. Spark version should be 1.6.x as the packaged jobserver is version specific. NOTE: the NiFi integration will be compatible with all versions.\r\n\r\ncurl https://s3.amazonaws.com/kylorepository/spark-jobserver/job-server.tar.gz > /tmp/job-server.tar.gz\r\ntar -xzf /tmp/job-server.tar.gz -C /opt/\r\n/opt/job-server/server_start.sh\r\n\r\ncurl https://s3.amazonaws.com/kylorepository/spark-jobserver/kylo-spark-validate-cleanse-app-0.8.0-SNAPSHOT-jar-with-dependencies.jar > /tmp/kylo-spark-validate-cleanse-app-0.8.0-SNAPSHOT-jar-with-dependencies.jar\r\ncurl --data-binary @/tmp/kylo-spark-validate-cleanse-app-0.8.0-SNAPSHOT-jar-with-dependencies.jar localhost:8090/jars/validate-and-cleanse\r\n\r\nModify the Standard Ingestion template\r\n-Delete Validate and Cleanse Job\r\n-Replace with ExecuteSparkContextJob\r\n---validate-and-cleanse\r\n---com.thinkbiganalytics.spark.datavalidator.ValidatorSparkJob\r\n---Hive Context\r\n---${category}-${feed}\r\n---targetDatabase=${category}, entity=${feed}, partition=${feedts}, pathToPolicyFile=${table_field_policy_json_file}\r\n---Configure new SparkJobService jobserver url http://localhost:8090\r\n---60\r\n---false\r\n---2\r\n---512m\r\n---2\r\n\r\nMake sure the correct nars are updated for NiFi to work. Will install when using RPM. On existing env use\r\n\r\n scp -P 2222 ~/git/kylo/integrations/nifi/nifi-nar-bundles/nifi-spark-bundle/nifi-spark-v1-nar/target/kylo-nifi-spark-v1-nar-0.8.1-SNAPSHOT.nar root@localhost:/opt/nifi/data/lib/\r\n scp -P 2222 ~/git/kylo/integrations/nifi/nifi-nar-bundles/nifi-standard-services/nifi-standard-services-nar/target/kylo-nifi-standard-services-nar-0.8.1-SNAPSHOT.nar root@localhost:/opt/nifi/data/lib/\r\n scp -P 2222 ~/git/kylo/integrations/nifi/nifi-nar-bundles/nifi-standard-services/nifi-spark-service-bundle/nifi-spark-service-v1-nar/target/kylo-nifi-spark-service-v1-nar-0.8.1-SNAPSHOT.nar root@localhost:/opt/nifi/data/lib/"},"comment":{"url":"https://api.github.com/repos/Teradata/kylo/issues/comments/319533289","html_url":"https://github.com/Teradata/kylo/pull/23#issuecomment-319533289","issue_url":"https://api.github.com/repos/Teradata/kylo/issues/23","id":319533289,"user":{"login":"freastro","id":297631,"avatar_url":"https://avatars0.githubusercontent.com/u/297631?v=4","gravatar_id":"","url":"https://api.github.com/users/freastro","html_url":"https://github.com/freastro","followers_url":"https://api.github.com/users/freastro/followers","following_url":"https://api.github.com/users/freastro/following{/other_user}","gists_url":"https://api.github.com/users/freastro/gists{/gist_id}","starred_url":"https://api.github.com/users/freastro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freastro/subscriptions","organizations_url":"https://api.github.com/users/freastro/orgs","repos_url":"https://api.github.com/users/freastro/repos","events_url":"https://api.github.com/users/freastro/events{/privacy}","received_events_url":"https://api.github.com/users/freastro/received_events","type":"User","site_admin":false},"created_at":"2017-08-02T00:25:08Z","updated_at":"2017-08-02T00:25:08Z","body":"I resolved the remaining issues and pulled in the changes. Thanks!"}}
{"actor":{"display_login":"freastro"}}