{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-predictionio/issues/421","repository_url":"https://api.github.com/repos/apache/incubator-predictionio","labels_url":"https://api.github.com/repos/apache/incubator-predictionio/issues/421/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-predictionio/issues/421/comments","events_url":"https://api.github.com/repos/apache/incubator-predictionio/issues/421/events","html_url":"https://github.com/apache/incubator-predictionio/pull/421","id":250138575,"number":421,"title":"Elasticsearch 5.x singleton client with authentication","user":{"login":"mars","id":17164,"avatar_url":"https://avatars1.githubusercontent.com/u/17164?v=4","gravatar_id":"","url":"https://api.github.com/users/mars","html_url":"https://github.com/mars","followers_url":"https://api.github.com/users/mars/followers","following_url":"https://api.github.com/users/mars/following{/other_user}","gists_url":"https://api.github.com/users/mars/gists{/gist_id}","starred_url":"https://api.github.com/users/mars/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mars/subscriptions","organizations_url":"https://api.github.com/users/mars/orgs","repos_url":"https://api.github.com/users/mars/repos","events_url":"https://api.github.com/users/mars/events{/privacy}","received_events_url":"https://api.github.com/users/mars/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-08-14T20:05:09Z","updated_at":"2017-08-29T15:24:15Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-predictionio/pulls/421","html_url":"https://github.com/apache/incubator-predictionio/pull/421","diff_url":"https://github.com/apache/incubator-predictionio/pull/421.diff","patch_url":"https://github.com/apache/incubator-predictionio/pull/421.patch"},"body":"Fixes both [PIO-106](https://issues.apache.org/jira/browse/PIO-106) & [PIO-114](https://issues.apache.org/jira/browse/PIO-114), replacing https://github.com/apache/incubator-predictionio/pull/372 & https://github.com/apache/incubator-predictionio/pull/420. These are combined because they each heavily revise the same class.\r\n\r\n## Authentication\r\n\r\nAdd optional username-password configuration for the new Elasticsearch 5 client; in `pio-env.sh` config:\r\n\r\n```bash\r\n# Optional basic HTTP auth\r\nPIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name\r\nPIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret\r\n```\r\n\r\nThese credentials are sent in each Elasticsearch request as an HTTP Basic Authorization header.\r\n\r\nEnables use of public-cloud, hosted Elasticsearch clusters, such as [Bonsai on Heroku](https://elements.heroku.com/addons/bonsai).\r\n\r\n## Singleton client\r\n\r\nThis PR moves to a singleton Elasticsearch RestClient which has built-in HTTP keep-alive and TCP connection pooling. Running on this branch, we've seen a 2x speed-up in predictions from the Universal Recommender with ES5, and the feared \"cannot assign requested address\" 😱 Elasticsearch connection errors have completely disappeared. Running `pio batchpredict` for 160K queries results in only 7 total TCP connections to Elasticsearch. Previously that would escalate to ~25,000 connections before denying further connections.\r\n\r\n**This fundamentally changes the interface for the new [Elasticsearch 5.x REST client](https://github.com/apache/incubator-predictionio/tree/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch)** introduced with PredictionIO 0.11.0-incubating. With this changeset, the `client` is a single instance of [`org.elasticsearch.client.RestClient`](https://github.com/elastic/elasticsearch/blob/master/client/rest/src/main/java/org/elasticsearch/client/RestClient.java).\r\n\r\n🚨 **As a result of this change, any engine templates that directly use the Elasticsearch 5 StorageClient would require an update for compatibility.** The change is this:\r\n\r\n### Original \r\n\r\n```scala\r\nval client: StorageClient = … // code to instantiate client\r\nval restClient: RestClient = client.open()\r\ntry {\r\n  restClient.performRequest(…)\r\n} finally {\r\n  restClient.close()\r\n}\r\n```\r\n\r\n### With this PR\r\n\r\n```scala\r\nval client: RestClient = … // code to instantiate client\r\nclient.performRequest(…)\r\n```\r\n\r\n*No more balancing `open` & `close` as this is handled by using a new `CleanupFunctions` hook added to the framework in this PR.*\r\n\r\n[Universal Recommender](https://github.com/actionml/universal-recommender) is the only template that I know of which directly uses the ES StorageClient outside of PredictionIO core. See example [UR changes for compatibility with this PR](https://github.com/heroku/predictionio-engine-ur/pull/7/files).\r\n\r\n### Elasticsearch StorageClient changes\r\n\r\n* reimplemented as singleton\r\n* installs a cleanup function\r\n\r\nSee [StorageClient](https://github.com/apache/incubator-predictionio/compare/develop...mars:esclient-singleton-with-auth?expand=1#diff-2926f4cfd93ccb02320e2a9503ccd223)\r\n\r\n### Core changes\r\n\r\nA new [`CleanupFunctions`](https://github.com/apache/incubator-predictionio/compare/develop...mars:esclient-singleton-with-auth?expand=1#diff-2a958821ac58f019fbce38540c775f19) hook has been added which enables developers of storage modules to register anonymous functions with `CleanupFunctions.add { … }` to be executed after Spark-related commands/workflows. The hook is called in a `finally { CleanupFunctions.run() }` from within:\r\n\r\n* `pio import`\r\n* `pio export`\r\n* `pio train`\r\n* `pio batchpredict`\r\n\r\nApologies for the huge indentation shifts from the requisite try-finally blocks:\r\n\r\n```scala\r\ntry {\r\n  // Freshly indented code.\r\n} finally {\r\n  CleanupFunctions.run()\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/incubator-predictionio/issues/comments/325699629","html_url":"https://github.com/apache/incubator-predictionio/pull/421#issuecomment-325699629","issue_url":"https://api.github.com/repos/apache/incubator-predictionio/issues/421","id":325699629,"user":{"login":"mars","id":17164,"avatar_url":"https://avatars1.githubusercontent.com/u/17164?v=4","gravatar_id":"","url":"https://api.github.com/users/mars","html_url":"https://github.com/mars","followers_url":"https://api.github.com/users/mars/followers","following_url":"https://api.github.com/users/mars/following{/other_user}","gists_url":"https://api.github.com/users/mars/gists{/gist_id}","starred_url":"https://api.github.com/users/mars/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mars/subscriptions","organizations_url":"https://api.github.com/users/mars/orgs","repos_url":"https://api.github.com/users/mars/repos","events_url":"https://api.github.com/users/mars/events{/privacy}","received_events_url":"https://api.github.com/users/mars/received_events","type":"User","site_admin":false},"created_at":"2017-08-29T15:24:15Z","updated_at":"2017-08-29T15:24:15Z","body":"I will resolve these conflicts today and then merge this PR."}}
{"actor":{"display_login":"mars"}}