{"action":"created","comment":{"url":"https://api.github.com/repos/seznam/euphoria/pulls/comments/118498160","pull_request_review_id":40288172,"id":118498160,"diff_hunk":"@@ -32,6 +33,9 @@ default ExecutorEnvironment newExecutorEnvironment() throws Exception {\n     RocksDBStateBackend backend = new RocksDBStateBackend(\"file://\" + path);\n     FlinkExecutor executor = new TestFlinkExecutor(ModuloInputSplitAssigner::new)\n         .setStateBackend(backend);\n+    // TODO should use some local test implementation of counters\n+    // that is queryable in assertions","path":"euphoria-flink/src/test/java/cz/seznam/euphoria/flink/testkit/FlinkExecutorProvider.java","position":13,"original_position":13,"commit_id":"a2fc80c18bf161290bd1918e0c77541b29f0a2f4","original_commit_id":"a2fc80c18bf161290bd1918e0c77541b29f0a2f4","user":{"login":"je-ik","id":14139263,"avatar_url":"https://avatars2.githubusercontent.com/u/14139263?v=3","gravatar_id":"","url":"https://api.github.com/users/je-ik","html_url":"https://github.com/je-ik","followers_url":"https://api.github.com/users/je-ik/followers","following_url":"https://api.github.com/users/je-ik/following{/other_user}","gists_url":"https://api.github.com/users/je-ik/gists{/gist_id}","starred_url":"https://api.github.com/users/je-ik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/je-ik/subscriptions","organizations_url":"https://api.github.com/users/je-ik/orgs","repos_url":"https://api.github.com/users/je-ik/repos","events_url":"https://api.github.com/users/je-ik/events{/privacy}","received_events_url":"https://api.github.com/users/je-ik/received_events","type":"User","site_admin":false},"body":"Is this comment meant for the sake of unittesting user flows? Isn't this what `InMemExecutor` is for?","created_at":"2017-05-25T14:30:26Z","updated_at":"2017-05-25T14:38:27Z","html_url":"https://github.com/seznam/euphoria/pull/132#discussion_r118498160","pull_request_url":"https://api.github.com/repos/seznam/euphoria/pulls/132","_links":{"self":{"href":"https://api.github.com/repos/seznam/euphoria/pulls/comments/118498160"},"html":{"href":"https://github.com/seznam/euphoria/pull/132#discussion_r118498160"},"pull_request":{"href":"https://api.github.com/repos/seznam/euphoria/pulls/132"}}},"pull_request":{"url":"https://api.github.com/repos/seznam/euphoria/pulls/132","id":122041661,"html_url":"https://github.com/seznam/euphoria/pull/132","diff_url":"https://github.com/seznam/euphoria/pull/132.diff","patch_url":"https://github.com/seznam/euphoria/pull/132.patch","issue_url":"https://api.github.com/repos/seznam/euphoria/issues/132","number":132,"state":"open","locked":false,"title":"#31 [euphoria-core] Implementation of accumulator API","user":{"login":"vanekjar","id":6866668,"avatar_url":"https://avatars0.githubusercontent.com/u/6866668?v=3","gravatar_id":"","url":"https://api.github.com/users/vanekjar","html_url":"https://github.com/vanekjar","followers_url":"https://api.github.com/users/vanekjar/followers","following_url":"https://api.github.com/users/vanekjar/following{/other_user}","gists_url":"https://api.github.com/users/vanekjar/gists{/gist_id}","starred_url":"https://api.github.com/users/vanekjar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanekjar/subscriptions","organizations_url":"https://api.github.com/users/vanekjar/orgs","repos_url":"https://api.github.com/users/vanekjar/repos","events_url":"https://api.github.com/users/vanekjar/events{/privacy}","received_events_url":"https://api.github.com/users/vanekjar/received_events","type":"User","site_admin":false},"body":"I implemented a basic accumulator API under `euphoria-core` as well with a sample implementation in `euphoria-flink` executor. Other implementations will come later.\r\n\r\nThere are three fixed types of accumulators:\r\n * Counter - Basic Hadoop style integral counter\r\n * Histogram - Accumulator collecting distribution of values\r\n * Timer - Collects distribution of time measurements (very similar to Histogram, extended by time units)\r\n\r\nFrom user function perspective accumulators can be used in the following way:\r\n\r\n```java\r\n// MapElements without counters, uses UnaryFunction lambda\r\nDataset<String> words = MapElements.of(dataset)\r\n    .using(s -> s.toLowerCase())\r\n   \r\n// MapElements with access to context, uses UnaryFunctionEnv lambda.\r\n// Obtains read-only context (without access to .collect() method) as an argument.\r\nDataset<String> mapped = MapElements.of(dataset)\r\n    .using((input, context) -> {\r\n      // use simple counter\r\n      context.getCounter(\"my-counter\").increment();\r\n  \r\n      // use histogram\r\n      context.getHistogram(\"my-hist\").add(5L);\r\n\r\n      // use timer\r\n      context.getTimer(\"my-timer\").add(Duration.ofSeconds(5L));\r\n  \r\n      return input.toLowerCase();\r\n    })\r\n```\r\n\r\nWhen creating an `executor` user can define preferred accumulator backend:\r\n\r\n```java\r\nFlinkExecutor executor = new FlinkExecutor();\r\nexecutor.setAccumulatorProvider(FlinkNativeAccumulators.getFactory());\r\n// or \r\nexecutor.setAccumulatorProvider(VoidAccumulatorProvider.getFactory());\r\n```\r\n\r\n-----\r\n\r\nI appreciate a thorough review of the suggested core API as well as any feedback.\r\n\r\nThanks!\r\n\r\ncloses #31 ","created_at":"2017-05-23T16:27:59Z","updated_at":"2017-05-25T14:38:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"972c9414f71e18e0ac78b4461b2ed75ce38c4616","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/seznam/euphoria/pulls/132/commits","review_comments_url":"https://api.github.com/repos/seznam/euphoria/pulls/132/comments","review_comment_url":"https://api.github.com/repos/seznam/euphoria/pulls/comments{/number}","comments_url":"https://api.github.com/repos/seznam/euphoria/issues/132/comments","statuses_url":"https://api.github.com/repos/seznam/euphoria/statuses/a2fc80c18bf161290bd1918e0c77541b29f0a2f4","head":{"label":"seznam:vanekjar/31/Accumulators","ref":"vanekjar/31/Accumulators","sha":"a2fc80c18bf161290bd1918e0c77541b29f0a2f4","user":{"login":"seznam","id":192214,"avatar_url":"https://avatars0.githubusercontent.com/u/192214?v=3","gravatar_id":"","url":"https://api.github.com/users/seznam","html_url":"https://github.com/seznam","followers_url":"https://api.github.com/users/seznam/followers","following_url":"https://api.github.com/users/seznam/following{/other_user}","gists_url":"https://api.github.com/users/seznam/gists{/gist_id}","starred_url":"https://api.github.com/users/seznam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seznam/subscriptions","organizations_url":"https://api.github.com/users/seznam/orgs","repos_url":"https://api.github.com/users/seznam/repos","events_url":"https://api.github.com/users/seznam/events{/privacy}","received_events_url":"https://api.github.com/users/seznam/received_events","type":"Organization","site_admin":false},"repo":{"id":79946335,"name":"euphoria","full_name":"seznam/euphoria","owner":{"login":"seznam","id":192214,"avatar_url":"https://avatars0.githubusercontent.com/u/192214?v=3","gravatar_id":"","url":"https://api.github.com/users/seznam","html_url":"https://github.com/seznam","followers_url":"https://api.github.com/users/seznam/followers","following_url":"https://api.github.com/users/seznam/following{/other_user}","gists_url":"https://api.github.com/users/seznam/gists{/gist_id}","starred_url":"https://api.github.com/users/seznam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seznam/subscriptions","organizations_url":"https://api.github.com/users/seznam/orgs","repos_url":"https://api.github.com/users/seznam/repos","events_url":"https://api.github.com/users/seznam/events{/privacy}","received_events_url":"https://api.github.com/users/seznam/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/seznam/euphoria","description":"Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.","fork":false,"url":"https://api.github.com/repos/seznam/euphoria","forks_url":"https://api.github.com/repos/seznam/euphoria/forks","keys_url":"https://api.github.com/repos/seznam/euphoria/keys{/key_id}","collaborators_url":"https://api.github.com/repos/seznam/euphoria/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/seznam/euphoria/teams","hooks_url":"https://api.github.com/repos/seznam/euphoria/hooks","issue_events_url":"https://api.github.com/repos/seznam/euphoria/issues/events{/number}","events_url":"https://api.github.com/repos/seznam/euphoria/events","assignees_url":"https://api.github.com/repos/seznam/euphoria/assignees{/user}","branches_url":"https://api.github.com/repos/seznam/euphoria/branches{/branch}","tags_url":"https://api.github.com/repos/seznam/euphoria/tags","blobs_url":"https://api.github.com/repos/seznam/euphoria/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/seznam/euphoria/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/seznam/euphoria/git/refs{/sha}","trees_url":"https://api.github.com/repos/seznam/euphoria/git/trees{/sha}","statuses_url":"https://api.github.com/repos/seznam/euphoria/statuses/{sha}","languages_url":"https://api.github.com/repos/seznam/euphoria/languages","stargazers_url":"https://api.github.com/repos/seznam/euphoria/stargazers","contributors_url":"https://api.github.com/repos/seznam/euphoria/contributors","subscribers_url":"https://api.github.com/repos/seznam/euphoria/subscribers","subscription_url":"https://api.github.com/repos/seznam/euphoria/subscription","commits_url":"https://api.github.com/repos/seznam/euphoria/commits{/sha}","git_commits_url":"https://api.github.com/repos/seznam/euphoria/git/commits{/sha}","comments_url":"https://api.github.com/repos/seznam/euphoria/comments{/number}","issue_comment_url":"https://api.github.com/repos/seznam/euphoria/issues/comments{/number}","contents_url":"https://api.github.com/repos/seznam/euphoria/contents/{+path}","compare_url":"https://api.github.com/repos/seznam/euphoria/compare/{base}...{head}","merges_url":"https://api.github.com/repos/seznam/euphoria/merges","archive_url":"https://api.github.com/repos/seznam/euphoria/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/seznam/euphoria/downloads","issues_url":"https://api.github.com/repos/seznam/euphoria/issues{/number}","pulls_url":"https://api.github.com/repos/seznam/euphoria/pulls{/number}","milestones_url":"https://api.github.com/repos/seznam/euphoria/milestones{/number}","notifications_url":"https://api.github.com/repos/seznam/euphoria/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/seznam/euphoria/labels{/name}","releases_url":"https://api.github.com/repos/seznam/euphoria/releases{/id}","deployments_url":"https://api.github.com/repos/seznam/euphoria/deployments","created_at":"2017-01-24T19:32:26Z","updated_at":"2017-05-06T19:59:42Z","pushed_at":"2017-05-23T16:27:59Z","git_url":"git://github.com/seznam/euphoria.git","ssh_url":"git@github.com:seznam/euphoria.git","clone_url":"https://github.com/seznam/euphoria.git","svn_url":"https://github.com/seznam/euphoria","homepage":"","size":2521,"stargazers_count":31,"watchers_count":31,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":40,"forks":4,"open_issues":40,"watchers":31,"default_branch":"master"}},"base":{"label":"seznam:master","ref":"master","sha":"f511f8ee300a93aa3482a064123698d2f43a27a2","user":{"login":"seznam","id":192214,"avatar_url":"https://avatars0.githubusercontent.com/u/192214?v=3","gravatar_id":"","url":"https://api.github.com/users/seznam","html_url":"https://github.com/seznam","followers_url":"https://api.github.com/users/seznam/followers","following_url":"https://api.github.com/users/seznam/following{/other_user}","gists_url":"https://api.github.com/users/seznam/gists{/gist_id}","starred_url":"https://api.github.com/users/seznam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seznam/subscriptions","organizations_url":"https://api.github.com/users/seznam/orgs","repos_url":"https://api.github.com/users/seznam/repos","events_url":"https://api.github.com/users/seznam/events{/privacy}","received_events_url":"https://api.github.com/users/seznam/received_events","type":"Organization","site_admin":false},"repo":{"id":79946335,"name":"euphoria","full_name":"seznam/euphoria","owner":{"login":"seznam","id":192214,"avatar_url":"https://avatars0.githubusercontent.com/u/192214?v=3","gravatar_id":"","url":"https://api.github.com/users/seznam","html_url":"https://github.com/seznam","followers_url":"https://api.github.com/users/seznam/followers","following_url":"https://api.github.com/users/seznam/following{/other_user}","gists_url":"https://api.github.com/users/seznam/gists{/gist_id}","starred_url":"https://api.github.com/users/seznam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seznam/subscriptions","organizations_url":"https://api.github.com/users/seznam/orgs","repos_url":"https://api.github.com/users/seznam/repos","events_url":"https://api.github.com/users/seznam/events{/privacy}","received_events_url":"https://api.github.com/users/seznam/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/seznam/euphoria","description":"Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.","fork":false,"url":"https://api.github.com/repos/seznam/euphoria","forks_url":"https://api.github.com/repos/seznam/euphoria/forks","keys_url":"https://api.github.com/repos/seznam/euphoria/keys{/key_id}","collaborators_url":"https://api.github.com/repos/seznam/euphoria/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/seznam/euphoria/teams","hooks_url":"https://api.github.com/repos/seznam/euphoria/hooks","issue_events_url":"https://api.github.com/repos/seznam/euphoria/issues/events{/number}","events_url":"https://api.github.com/repos/seznam/euphoria/events","assignees_url":"https://api.github.com/repos/seznam/euphoria/assignees{/user}","branches_url":"https://api.github.com/repos/seznam/euphoria/branches{/branch}","tags_url":"https://api.github.com/repos/seznam/euphoria/tags","blobs_url":"https://api.github.com/repos/seznam/euphoria/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/seznam/euphoria/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/seznam/euphoria/git/refs{/sha}","trees_url":"https://api.github.com/repos/seznam/euphoria/git/trees{/sha}","statuses_url":"https://api.github.com/repos/seznam/euphoria/statuses/{sha}","languages_url":"https://api.github.com/repos/seznam/euphoria/languages","stargazers_url":"https://api.github.com/repos/seznam/euphoria/stargazers","contributors_url":"https://api.github.com/repos/seznam/euphoria/contributors","subscribers_url":"https://api.github.com/repos/seznam/euphoria/subscribers","subscription_url":"https://api.github.com/repos/seznam/euphoria/subscription","commits_url":"https://api.github.com/repos/seznam/euphoria/commits{/sha}","git_commits_url":"https://api.github.com/repos/seznam/euphoria/git/commits{/sha}","comments_url":"https://api.github.com/repos/seznam/euphoria/comments{/number}","issue_comment_url":"https://api.github.com/repos/seznam/euphoria/issues/comments{/number}","contents_url":"https://api.github.com/repos/seznam/euphoria/contents/{+path}","compare_url":"https://api.github.com/repos/seznam/euphoria/compare/{base}...{head}","merges_url":"https://api.github.com/repos/seznam/euphoria/merges","archive_url":"https://api.github.com/repos/seznam/euphoria/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/seznam/euphoria/downloads","issues_url":"https://api.github.com/repos/seznam/euphoria/issues{/number}","pulls_url":"https://api.github.com/repos/seznam/euphoria/pulls{/number}","milestones_url":"https://api.github.com/repos/seznam/euphoria/milestones{/number}","notifications_url":"https://api.github.com/repos/seznam/euphoria/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/seznam/euphoria/labels{/name}","releases_url":"https://api.github.com/repos/seznam/euphoria/releases{/id}","deployments_url":"https://api.github.com/repos/seznam/euphoria/deployments","created_at":"2017-01-24T19:32:26Z","updated_at":"2017-05-06T19:59:42Z","pushed_at":"2017-05-23T16:27:59Z","git_url":"git://github.com/seznam/euphoria.git","ssh_url":"git@github.com:seznam/euphoria.git","clone_url":"https://github.com/seznam/euphoria.git","svn_url":"https://github.com/seznam/euphoria","homepage":"","size":2521,"stargazers_count":31,"watchers_count":31,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":40,"forks":4,"open_issues":40,"watchers":31,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/seznam/euphoria/pulls/132"},"html":{"href":"https://github.com/seznam/euphoria/pull/132"},"issue":{"href":"https://api.github.com/repos/seznam/euphoria/issues/132"},"comments":{"href":"https://api.github.com/repos/seznam/euphoria/issues/132/comments"},"review_comments":{"href":"https://api.github.com/repos/seznam/euphoria/pulls/132/comments"},"review_comment":{"href":"https://api.github.com/repos/seznam/euphoria/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/seznam/euphoria/pulls/132/commits"},"statuses":{"href":"https://api.github.com/repos/seznam/euphoria/statuses/a2fc80c18bf161290bd1918e0c77541b29f0a2f4"}}}}
{"actor":{"display_login":"je-ik"}}