{"action":"created","comment":{"url":"https://api.github.com/repos/Teradata/kylo/pulls/comments/115112301","pull_request_review_id":36636167,"id":115112301,"diff_hunk":"@@ -0,0 +1,58 @@\n+package com.thinkbiganalytics.nifi.v2.spark;\n+\n+/*-\n+ * #%L\n+ * thinkbig-nifi-hadoop-service-api\n+ * %%\n+ * Copyright (C) 2017 ThinkBig Analytics\n+ * %%\n+ * Licensed under the Apache License, Version 2.0 (the \"License\");","path":"integrations/nifi/nifi-nar-bundles/nifi-standard-services/nifi-spark-service-api/src/main/java/com/thinkbiganalytics/nifi/v2/spark/JobService.java","position":9,"original_position":9,"commit_id":"1977e5bc0ebf84a1f1171a6dcaf938e277d1c085","original_commit_id":"1977e5bc0ebf84a1f1171a6dcaf938e277d1c085","user":{"login":"KyloTech","id":27316027,"avatar_url":"https://avatars3.githubusercontent.com/u/27316027?v=3","gravatar_id":"","url":"https://api.github.com/users/KyloTech","html_url":"https://github.com/KyloTech","followers_url":"https://api.github.com/users/KyloTech/followers","following_url":"https://api.github.com/users/KyloTech/following{/other_user}","gists_url":"https://api.github.com/users/KyloTech/gists{/gist_id}","starred_url":"https://api.github.com/users/KyloTech/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KyloTech/subscriptions","organizations_url":"https://api.github.com/users/KyloTech/orgs","repos_url":"https://api.github.com/users/KyloTech/repos","events_url":"https://api.github.com/users/KyloTech/events{/privacy}","received_events_url":"https://api.github.com/users/KyloTech/received_events","type":"User","site_admin":false},"body":"![MAJOR](https://sonarsource.github.io/sonar-github/severity-major.png 'Severity: MAJOR') This block of commented-out lines of code should be removed. [![rule](https://sonarsource.github.io/sonar-github/rule.png)](https://sonarqube.com/coding_rules#rule_key=squid%3ACommentedOutCodeLine)\n","created_at":"2017-05-06T03:39:13Z","updated_at":"2017-05-06T03:39:13Z","html_url":"https://github.com/Teradata/kylo/pull/23#discussion_r115112301","pull_request_url":"https://api.github.com/repos/Teradata/kylo/pulls/23","_links":{"self":{"href":"https://api.github.com/repos/Teradata/kylo/pulls/comments/115112301"},"html":{"href":"https://github.com/Teradata/kylo/pull/23#discussion_r115112301"},"pull_request":{"href":"https://api.github.com/repos/Teradata/kylo/pulls/23"}}},"pull_request":{"url":"https://api.github.com/repos/Teradata/kylo/pulls/23","id":119285163,"html_url":"https://github.com/Teradata/kylo/pull/23","diff_url":"https://github.com/Teradata/kylo/pull/23.diff","patch_url":"https://github.com/Teradata/kylo/pull/23.patch","issue_url":"https://api.github.com/repos/Teradata/kylo/issues/23","number":23,"state":"open","locked":false,"title":"NiFi - Spark Jobserver Integration","user":{"login":"calvin-pietersen","id":16835507,"avatar_url":"https://avatars1.githubusercontent.com/u/16835507?v=3","gravatar_id":"","url":"https://api.github.com/users/calvin-pietersen","html_url":"https://github.com/calvin-pietersen","followers_url":"https://api.github.com/users/calvin-pietersen/followers","following_url":"https://api.github.com/users/calvin-pietersen/following{/other_user}","gists_url":"https://api.github.com/users/calvin-pietersen/gists{/gist_id}","starred_url":"https://api.github.com/users/calvin-pietersen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvin-pietersen/subscriptions","organizations_url":"https://api.github.com/users/calvin-pietersen/orgs","repos_url":"https://api.github.com/users/calvin-pietersen/repos","events_url":"https://api.github.com/users/calvin-pietersen/events{/privacy}","received_events_url":"https://api.github.com/users/calvin-pietersen/received_events","type":"User","site_admin":false},"body":"Adding integration to be able to have long running Spark Contexts in NiFi. \r\nAdded a Controller Service SparkJobserverService and a processor ExecuteSparkContextJob.\r\n\r\nTesting requires spark jobserver and a class implementing SparkJob. Spark version should be 1.6.x as the packaged jobserver is version specific. NOTE: the NiFi integration will be compatible with all versions.\r\n\r\ncurl https://s3.amazonaws.com/kylorepository/spark-jobserver/job-server.tar.gz > /tmp/job-server.tar.gz\r\ntar -xzf /tmp/job-server.tar.gz -C /opt/\r\n/opt/job-server/server_start.sh\r\n\r\ncurl https://s3.amazonaws.com/kylorepository/spark-jobserver/kylo-spark-validate-cleanse-app-0.8.0-SNAPSHOT-jar-with-dependencies.jar > /tmp/kylo-spark-validate-cleanse-app-0.8.0-SNAPSHOT-jar-with-dependencies.jar\r\ncurl --data-binary @/tmp/kylo-spark-validate-cleanse-app-0.8.0-SNAPSHOT-jar-with-dependencies.jar localhost:8090/jars/validate-and-cleanse\r\n\r\nModify the Standard Ingestion template\r\n-Delete Validate and Cleanse Job\r\n-Replace with ExecuteSparkContextJob\r\n---validate-and-cleanse\r\n---com.thinkbiganalytics.spark.datavalidator.ValidatorSparkJob\r\n---Hive Context\r\n---${category}-${feed}\r\n---Configure new SparkJobService jobserver url http://localhost:8090\r\n---60\r\n---false\r\n---2\r\n---512m\r\n---2\r\n\r\nMake sure the correct nars are updated for NiFi to work. Will install when using RPM. On existing env use\r\n\r\n scp -P 2222 ~/git/kylo/integrations/nifi/nifi-nar-bundles/nifi-spark-bundle/nifi-spark-v1-nar/target/kylo-nifi-spark-v1-nar-0.8.1-SNAPSHOT.nar root@localhost:/opt/nifi/data/lib/\r\n scp -P 2222 ~/git/kylo/integrations/nifi/nifi-nar-bundles/nifi-standard-services/nifi-standard-services-nar/target/kylo-nifi-standard-services-nar-0.8.1-SNAPSHOT.nar root@localhost:/opt/nifi/data/lib/\r\n scp -P 2222 ~/git/kylo/integrations/nifi/nifi-nar-bundles/nifi-standard-services/nifi-spark-service-bundle/nifi-spark-service-v1-nar/target/kylo-nifi-spark-service-v1-nar-0.8.1-SNAPSHOT.nar root@localhost:/opt/nifi/data/lib/","created_at":"2017-05-06T03:22:43Z","updated_at":"2017-05-06T03:39:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c01e9a5520f4ae40aa22eece3ca53bb59411a650","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/Teradata/kylo/pulls/23/commits","review_comments_url":"https://api.github.com/repos/Teradata/kylo/pulls/23/comments","review_comment_url":"https://api.github.com/repos/Teradata/kylo/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Teradata/kylo/issues/23/comments","statuses_url":"https://api.github.com/repos/Teradata/kylo/statuses/1977e5bc0ebf84a1f1171a6dcaf938e277d1c085","head":{"label":"calvin-pietersen:master","ref":"master","sha":"1977e5bc0ebf84a1f1171a6dcaf938e277d1c085","user":{"login":"calvin-pietersen","id":16835507,"avatar_url":"https://avatars1.githubusercontent.com/u/16835507?v=3","gravatar_id":"","url":"https://api.github.com/users/calvin-pietersen","html_url":"https://github.com/calvin-pietersen","followers_url":"https://api.github.com/users/calvin-pietersen/followers","following_url":"https://api.github.com/users/calvin-pietersen/following{/other_user}","gists_url":"https://api.github.com/users/calvin-pietersen/gists{/gist_id}","starred_url":"https://api.github.com/users/calvin-pietersen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvin-pietersen/subscriptions","organizations_url":"https://api.github.com/users/calvin-pietersen/orgs","repos_url":"https://api.github.com/users/calvin-pietersen/repos","events_url":"https://api.github.com/users/calvin-pietersen/events{/privacy}","received_events_url":"https://api.github.com/users/calvin-pietersen/received_events","type":"User","site_admin":false},"repo":{"id":89030784,"name":"kylo","full_name":"calvin-pietersen/kylo","owner":{"login":"calvin-pietersen","id":16835507,"avatar_url":"https://avatars1.githubusercontent.com/u/16835507?v=3","gravatar_id":"","url":"https://api.github.com/users/calvin-pietersen","html_url":"https://github.com/calvin-pietersen","followers_url":"https://api.github.com/users/calvin-pietersen/followers","following_url":"https://api.github.com/users/calvin-pietersen/following{/other_user}","gists_url":"https://api.github.com/users/calvin-pietersen/gists{/gist_id}","starred_url":"https://api.github.com/users/calvin-pietersen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvin-pietersen/subscriptions","organizations_url":"https://api.github.com/users/calvin-pietersen/orgs","repos_url":"https://api.github.com/users/calvin-pietersen/repos","events_url":"https://api.github.com/users/calvin-pietersen/events{/privacy}","received_events_url":"https://api.github.com/users/calvin-pietersen/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/calvin-pietersen/kylo","description":"Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on Apache Hadoop and Spark. Kylo is licensed under Apache 2.0 and contributed by Think Big, A Teradata Company","fork":true,"url":"https://api.github.com/repos/calvin-pietersen/kylo","forks_url":"https://api.github.com/repos/calvin-pietersen/kylo/forks","keys_url":"https://api.github.com/repos/calvin-pietersen/kylo/keys{/key_id}","collaborators_url":"https://api.github.com/repos/calvin-pietersen/kylo/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/calvin-pietersen/kylo/teams","hooks_url":"https://api.github.com/repos/calvin-pietersen/kylo/hooks","issue_events_url":"https://api.github.com/repos/calvin-pietersen/kylo/issues/events{/number}","events_url":"https://api.github.com/repos/calvin-pietersen/kylo/events","assignees_url":"https://api.github.com/repos/calvin-pietersen/kylo/assignees{/user}","branches_url":"https://api.github.com/repos/calvin-pietersen/kylo/branches{/branch}","tags_url":"https://api.github.com/repos/calvin-pietersen/kylo/tags","blobs_url":"https://api.github.com/repos/calvin-pietersen/kylo/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/calvin-pietersen/kylo/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/calvin-pietersen/kylo/git/refs{/sha}","trees_url":"https://api.github.com/repos/calvin-pietersen/kylo/git/trees{/sha}","statuses_url":"https://api.github.com/repos/calvin-pietersen/kylo/statuses/{sha}","languages_url":"https://api.github.com/repos/calvin-pietersen/kylo/languages","stargazers_url":"https://api.github.com/repos/calvin-pietersen/kylo/stargazers","contributors_url":"https://api.github.com/repos/calvin-pietersen/kylo/contributors","subscribers_url":"https://api.github.com/repos/calvin-pietersen/kylo/subscribers","subscription_url":"https://api.github.com/repos/calvin-pietersen/kylo/subscription","commits_url":"https://api.github.com/repos/calvin-pietersen/kylo/commits{/sha}","git_commits_url":"https://api.github.com/repos/calvin-pietersen/kylo/git/commits{/sha}","comments_url":"https://api.github.com/repos/calvin-pietersen/kylo/comments{/number}","issue_comment_url":"https://api.github.com/repos/calvin-pietersen/kylo/issues/comments{/number}","contents_url":"https://api.github.com/repos/calvin-pietersen/kylo/contents/{+path}","compare_url":"https://api.github.com/repos/calvin-pietersen/kylo/compare/{base}...{head}","merges_url":"https://api.github.com/repos/calvin-pietersen/kylo/merges","archive_url":"https://api.github.com/repos/calvin-pietersen/kylo/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/calvin-pietersen/kylo/downloads","issues_url":"https://api.github.com/repos/calvin-pietersen/kylo/issues{/number}","pulls_url":"https://api.github.com/repos/calvin-pietersen/kylo/pulls{/number}","milestones_url":"https://api.github.com/repos/calvin-pietersen/kylo/milestones{/number}","notifications_url":"https://api.github.com/repos/calvin-pietersen/kylo/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/calvin-pietersen/kylo/labels{/name}","releases_url":"https://api.github.com/repos/calvin-pietersen/kylo/releases{/id}","deployments_url":"https://api.github.com/repos/calvin-pietersen/kylo/deployments","created_at":"2017-04-21T23:18:34Z","updated_at":"2017-04-21T23:18:42Z","pushed_at":"2017-05-06T02:52:07Z","git_url":"git://github.com/calvin-pietersen/kylo.git","ssh_url":"git@github.com:calvin-pietersen/kylo.git","clone_url":"https://github.com/calvin-pietersen/kylo.git","svn_url":"https://github.com/calvin-pietersen/kylo","homepage":"http://kylo.io","size":34107,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"Teradata:master","ref":"master","sha":"395b523bb1e0fc60930e5275faa537ccaff0812a","user":{"login":"Teradata","id":1615979,"avatar_url":"https://avatars2.githubusercontent.com/u/1615979?v=3","gravatar_id":"","url":"https://api.github.com/users/Teradata","html_url":"https://github.com/Teradata","followers_url":"https://api.github.com/users/Teradata/followers","following_url":"https://api.github.com/users/Teradata/following{/other_user}","gists_url":"https://api.github.com/users/Teradata/gists{/gist_id}","starred_url":"https://api.github.com/users/Teradata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Teradata/subscriptions","organizations_url":"https://api.github.com/users/Teradata/orgs","repos_url":"https://api.github.com/users/Teradata/repos","events_url":"https://api.github.com/users/Teradata/events{/privacy}","received_events_url":"https://api.github.com/users/Teradata/received_events","type":"Organization","site_admin":false},"repo":{"id":81384608,"name":"kylo","full_name":"Teradata/kylo","owner":{"login":"Teradata","id":1615979,"avatar_url":"https://avatars2.githubusercontent.com/u/1615979?v=3","gravatar_id":"","url":"https://api.github.com/users/Teradata","html_url":"https://github.com/Teradata","followers_url":"https://api.github.com/users/Teradata/followers","following_url":"https://api.github.com/users/Teradata/following{/other_user}","gists_url":"https://api.github.com/users/Teradata/gists{/gist_id}","starred_url":"https://api.github.com/users/Teradata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Teradata/subscriptions","organizations_url":"https://api.github.com/users/Teradata/orgs","repos_url":"https://api.github.com/users/Teradata/repos","events_url":"https://api.github.com/users/Teradata/events{/privacy}","received_events_url":"https://api.github.com/users/Teradata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Teradata/kylo","description":"Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on Apache Hadoop and Spark. Kylo is licensed under Apache 2.0 and contributed by Think Big, A Teradata Company","fork":false,"url":"https://api.github.com/repos/Teradata/kylo","forks_url":"https://api.github.com/repos/Teradata/kylo/forks","keys_url":"https://api.github.com/repos/Teradata/kylo/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Teradata/kylo/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Teradata/kylo/teams","hooks_url":"https://api.github.com/repos/Teradata/kylo/hooks","issue_events_url":"https://api.github.com/repos/Teradata/kylo/issues/events{/number}","events_url":"https://api.github.com/repos/Teradata/kylo/events","assignees_url":"https://api.github.com/repos/Teradata/kylo/assignees{/user}","branches_url":"https://api.github.com/repos/Teradata/kylo/branches{/branch}","tags_url":"https://api.github.com/repos/Teradata/kylo/tags","blobs_url":"https://api.github.com/repos/Teradata/kylo/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Teradata/kylo/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Teradata/kylo/git/refs{/sha}","trees_url":"https://api.github.com/repos/Teradata/kylo/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Teradata/kylo/statuses/{sha}","languages_url":"https://api.github.com/repos/Teradata/kylo/languages","stargazers_url":"https://api.github.com/repos/Teradata/kylo/stargazers","contributors_url":"https://api.github.com/repos/Teradata/kylo/contributors","subscribers_url":"https://api.github.com/repos/Teradata/kylo/subscribers","subscription_url":"https://api.github.com/repos/Teradata/kylo/subscription","commits_url":"https://api.github.com/repos/Teradata/kylo/commits{/sha}","git_commits_url":"https://api.github.com/repos/Teradata/kylo/git/commits{/sha}","comments_url":"https://api.github.com/repos/Teradata/kylo/comments{/number}","issue_comment_url":"https://api.github.com/repos/Teradata/kylo/issues/comments{/number}","contents_url":"https://api.github.com/repos/Teradata/kylo/contents/{+path}","compare_url":"https://api.github.com/repos/Teradata/kylo/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Teradata/kylo/merges","archive_url":"https://api.github.com/repos/Teradata/kylo/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Teradata/kylo/downloads","issues_url":"https://api.github.com/repos/Teradata/kylo/issues{/number}","pulls_url":"https://api.github.com/repos/Teradata/kylo/pulls{/number}","milestones_url":"https://api.github.com/repos/Teradata/kylo/milestones{/number}","notifications_url":"https://api.github.com/repos/Teradata/kylo/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Teradata/kylo/labels{/name}","releases_url":"https://api.github.com/repos/Teradata/kylo/releases{/id}","deployments_url":"https://api.github.com/repos/Teradata/kylo/deployments","created_at":"2017-02-08T22:54:46Z","updated_at":"2017-05-05T08:47:16Z","pushed_at":"2017-05-06T03:22:43Z","git_url":"git://github.com/Teradata/kylo.git","ssh_url":"git@github.com:Teradata/kylo.git","clone_url":"https://github.com/Teradata/kylo.git","svn_url":"https://github.com/Teradata/kylo","homepage":"http://kylo.io","size":34422,"stargazers_count":158,"watchers_count":158,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":64,"mirror_url":null,"open_issues_count":5,"forks":64,"open_issues":5,"watchers":158,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Teradata/kylo/pulls/23"},"html":{"href":"https://github.com/Teradata/kylo/pull/23"},"issue":{"href":"https://api.github.com/repos/Teradata/kylo/issues/23"},"comments":{"href":"https://api.github.com/repos/Teradata/kylo/issues/23/comments"},"review_comments":{"href":"https://api.github.com/repos/Teradata/kylo/pulls/23/comments"},"review_comment":{"href":"https://api.github.com/repos/Teradata/kylo/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Teradata/kylo/pulls/23/commits"},"statuses":{"href":"https://api.github.com/repos/Teradata/kylo/statuses/1977e5bc0ebf84a1f1171a6dcaf938e277d1c085"}}}}
{"actor":{"display_login":"KyloTech"}}