{"action":"created","comment":{"url":"https://api.github.com/repos/apache/flink/pulls/comments/119153392","pull_request_review_id":41003410,"id":119153392,"diff_hunk":"@@ -39,17 +39,293 @@ Dynamic table\n Time Attributes\n ---------------\n \n-### Event-time\n+Flink supports different notions of *time* in streaming programs.\n \n-* DataStream: Timestamps & WMs required, `.rowtime` (replace attribute or extend schema)\n-* TableSource: Timestamps & WMs & DefinedRowtimeAttribute\n+- *Processing time* refers to the system time of the machine (also known as \"wall-clock time\") that is executing the respective operation.\n+- *Event time* is the time that each individual event occurred on its producing device.\n+- *Ingestion time* is the time that events enter Flink, internally, it is treated similar to event time.\n \n-{% top %}\n+For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({{ site.baseurl }}/dev/event_time.html).\n+\n+Table programs assume that a corresponding time characteristic has been specified for the streaming environment:\n+\n+<div class=\"codetabs\" markdown=\"1\">\n+<div data-lang=\"java\" markdown=\"1\">\n+{% highlight java %}\n+final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();\n+\n+env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); // default\n+\n+// alternatively:\n+// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);\n+// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);\n+{% endhighlight %}\n+</div>\n+<div data-lang=\"scala\" markdown=\"1\">\n+{% highlight scala %}\n+val env = StreamExecutionEnvironment.getExecutionEnvironment\n+\n+env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) //default\n+\n+// alternatively:\n+// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)\n+// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)\n+{% endhighlight %}\n+</div>\n+</div>\n+\n+Time-based operations such as [windows]({{ site.baseurl }}/dev/table/tableApi.html) in both the Table API and SQL require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.\n+\n+Time attributes can be part of every table schema. They are defined when creating a table from a `DataStream` or pre-defined when using a `TableSource`. Once a time attribute is defined at the beginning, it can be referenced as field and used in time-based operations.\n+\n+As long as a time attribute is not modified and simply forwarded from one part of the query to another, it remains a valid time attribute. Time attributes behave like regular timestamps and can be accessed for calculations. If a time attribute is used in a calculation, it will be materialized and becomes a regular timestamp. Regular timestamps do not cooperate with Flink's time and watermarking system and can thus not be used for time-based operations anymore.\n \n ### Processing time\n \n-* DataStream: `.proctime` (only extend schema)\n-* TableSource: DefinedProctimeAttribute\n+Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time but does not provide determinism. It does neither require timestamp extraction nor watermark generation.\n+\n+There are two ways to define a processing time attribute.\n+\n+#### During DataStream-to-Table Conversion\n+\n+The processing time attribute is defined with the `.proctime` property during schema definition. The time attribute must only extend the physical schema by an additional logical field. Thus, it can only be defined at the end of the schema definition.\n+\n+<div class=\"codetabs\" markdown=\"1\">\n+<div data-lang=\"java\" markdown=\"1\">\n+{% highlight java %}\n+DataStream<Tuple2<String, String>> stream = ...;\n+\n+// declare an additional logical field as processing time attribute\n+Table table = tEnv.fromDataStream(stream, \"Username, Data, UserActionTime.proctime\");\n+\n+WindowedTable windowedTable = table.window(Tumble.over(\"10.minutes\").on(\"UserActionTime\").as(\"userActionWindow\"));\n+{% endhighlight %}\n+</div>\n+<div data-lang=\"scala\" markdown=\"1\">\n+{% highlight scala %}\n+val stream: DataStream[(String, String)] = ...\n+\n+// declare an additional logical field as processing time attribute\n+val table = tEnv.fromDataStream(stream, 'UserActionTimestamp, 'Username, 'Data, 'UserActionTime.proctime)\n+\n+val windowedTable = table.window(Tumble over 10.minutes on 'UserActionTime as 'userActionWindow)\n+{% endhighlight %}\n+</div>\n+</div>\n+\n+#### Using a TableSource\n+\n+The processing time attribute is defined by a `TableSource` that implements the `DefinedProctimeAttribute` interface. The logical time attribute is appended to the physical schema defined by the return type of the `TableSource`.\n+\n+<div class=\"codetabs\" markdown=\"1\">\n+<div data-lang=\"java\" markdown=\"1\">\n+{% highlight java %}\n+// define table source with processing attribute\n+public class UserActionSource implements StreamTableSource<Row>, DefinedProctimeAttribute {\n+\n+\t@Override\n+\tpublic TypeInformation<Row> getReturnType() {\n+\t\tString[] names = new String[] {\"Username\" , \"Data\"};\n+\t\tTypeInformation[] types = new TypeInformation[] {Types.STRING(), Types.STRING()};\n+\t\treturn Types.ROW(names, types);\n+\t}\n+\n+\t@Override\n+\tpublic DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {\n+\t\t// create stream \n+\t\tDataStream<Row> stream = ...;\n+\t\treturn stream;\n+\t}\n+\n+\t@Override\n+\tpublic String getProctimeAttribute() {\n+\t\t// field with this name will be appended as a third field \n+\t\treturn \"UserActionTime\";\n+\t}\n+}\n+\n+// register table source\n+tEnv.registerTableSource(\"UserActions\", new UserActionSource());\n+\n+WindowedTable windowedTable = tEnv\n+\t.scan(\"UserActions\")\n+\t.window(Tumble.over(\"10.minutes\").on(\"UserActionTime\").as(\"userActionWindow\"));\n+{% endhighlight %}\n+</div>\n+<div data-lang=\"scala\" markdown=\"1\">\n+{% highlight scala %}\n+// define table source with processing attribute\n+class UserActionSource extends StreamTableSource[Row] with DefinedProctimeAttribute {\n+\n+\toverride def getReturnType = {\n+\t\tval names = Array[String](\"Username\" , \"Data\")\n+\t\tval types = Array[TypeInformation[_]](Types.STRING, Types.STRING)\n+\t\tTypes.ROW(names, types)\n+\t}\n+\n+\toverride def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {\n+\t\t// create stream\n+\t\tval stream = ...\n+\t\tstream\n+\t}\n+\n+\toverride def getProctimeAttribute = {\n+\t\t// field with this name will be appended as a third field \n+\t\t\"UserActionTime\"\n+\t}\n+}\n+\n+// register table source\n+tEnv.registerTableSource(\"UserActions\", new UserActionSource)\n+\n+val windowedTable = tEnv\n+\t.scan(\"UserActions\")\n+\t.window(Tumble over 10.minutes on 'UserActionTime as 'userActionWindow)\n+{% endhighlight %}\n+</div>\n+</div>\n+\n+### Event time\n+\n+Event time allows a table program to produce results based on the time that is contained in every record. This allows for consistent results even in case of out-of-order events or late events. It also ensures replayable results of table program when reading records from persistent storage. \n+\n+Additionally, event time allows for unified syntax for table programs in both a batch and streaming environment. A time attribute in streaming can be a regular field of a record in a batch environment.\n+\n+In order to handle out-of-order events and distinguish between on-time and late events in streaming, Flink needs to extract timestamps from events and make some kind of progress in time (so-called [watermarks]({{ site.baseurl }}/dev/event_time.html)).\n+\n+The Table API & SQL assumes that timestamps and watermarks have been generated in the [underlying DataStream API]({{ site.baseurl }}/dev/event_timestamps_watermarks.html) before. Ideally, this happens within a TableSource with knowledge about the incoming data's characteristics and hidden from the API end user.","path":"docs/dev/table/streaming.md","position":164,"original_position":164,"commit_id":"4041c56ac00e98745a01b4de05334986545bacdd","original_commit_id":"4041c56ac00e98745a01b4de05334986545bacdd","user":{"login":"alpinegizmo","id":43608,"avatar_url":"https://avatars1.githubusercontent.com/u/43608?v=3","gravatar_id":"","url":"https://api.github.com/users/alpinegizmo","html_url":"https://github.com/alpinegizmo","followers_url":"https://api.github.com/users/alpinegizmo/followers","following_url":"https://api.github.com/users/alpinegizmo/following{/other_user}","gists_url":"https://api.github.com/users/alpinegizmo/gists{/gist_id}","starred_url":"https://api.github.com/users/alpinegizmo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alpinegizmo/subscriptions","organizations_url":"https://api.github.com/users/alpinegizmo/orgs","repos_url":"https://api.github.com/users/alpinegizmo/repos","events_url":"https://api.github.com/users/alpinegizmo/events{/privacy}","received_events_url":"https://api.github.com/users/alpinegizmo/received_events","type":"User","site_admin":false},"body":"The Table API & SQL assume ...\r\n\r\n... and is hidden from the end user of the API.","created_at":"2017-05-30T16:45:22Z","updated_at":"2017-05-30T16:53:44Z","html_url":"https://github.com/apache/flink/pull/4020#discussion_r119153392","pull_request_url":"https://api.github.com/repos/apache/flink/pulls/4020","_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/comments/119153392"},"html":{"href":"https://github.com/apache/flink/pull/4020#discussion_r119153392"},"pull_request":{"href":"https://api.github.com/repos/apache/flink/pulls/4020"}}},"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/4020","id":123078023,"html_url":"https://github.com/apache/flink/pull/4020","diff_url":"https://github.com/apache/flink/pull/4020.diff","patch_url":"https://github.com/apache/flink/pull/4020.patch","issue_url":"https://api.github.com/repos/apache/flink/issues/4020","number":4020,"state":"open","locked":false,"title":"[FLINK-6747] [table] [docs] Time attributes section added","user":{"login":"twalthr","id":5746567,"avatar_url":"https://avatars2.githubusercontent.com/u/5746567?v=3","gravatar_id":"","url":"https://api.github.com/users/twalthr","html_url":"https://github.com/twalthr","followers_url":"https://api.github.com/users/twalthr/followers","following_url":"https://api.github.com/users/twalthr/following{/other_user}","gists_url":"https://api.github.com/users/twalthr/gists{/gist_id}","starred_url":"https://api.github.com/users/twalthr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twalthr/subscriptions","organizations_url":"https://api.github.com/users/twalthr/orgs","repos_url":"https://api.github.com/users/twalthr/repos","events_url":"https://api.github.com/users/twalthr/events{/privacy}","received_events_url":"https://api.github.com/users/twalthr/received_events","type":"User","site_admin":false},"body":"This documents the Table API's time attributes.\r\n\r\nCC @fhueske @alpinegizmo ","created_at":"2017-05-30T15:27:24Z","updated_at":"2017-05-30T16:53:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"99d44e6eb6d579e387b5ac01f6ef8bc330054ac3","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/flink/pulls/4020/commits","review_comments_url":"https://api.github.com/repos/apache/flink/pulls/4020/comments","review_comment_url":"https://api.github.com/repos/apache/flink/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/flink/issues/4020/comments","statuses_url":"https://api.github.com/repos/apache/flink/statuses/4041c56ac00e98745a01b4de05334986545bacdd","head":{"label":"twalthr:FLINK-6747","ref":"FLINK-6747","sha":"4041c56ac00e98745a01b4de05334986545bacdd","user":{"login":"twalthr","id":5746567,"avatar_url":"https://avatars2.githubusercontent.com/u/5746567?v=3","gravatar_id":"","url":"https://api.github.com/users/twalthr","html_url":"https://github.com/twalthr","followers_url":"https://api.github.com/users/twalthr/followers","following_url":"https://api.github.com/users/twalthr/following{/other_user}","gists_url":"https://api.github.com/users/twalthr/gists{/gist_id}","starred_url":"https://api.github.com/users/twalthr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twalthr/subscriptions","organizations_url":"https://api.github.com/users/twalthr/orgs","repos_url":"https://api.github.com/users/twalthr/repos","events_url":"https://api.github.com/users/twalthr/events{/privacy}","received_events_url":"https://api.github.com/users/twalthr/received_events","type":"User","site_admin":false},"repo":{"id":20658760,"name":"flink","full_name":"twalthr/flink","owner":{"login":"twalthr","id":5746567,"avatar_url":"https://avatars2.githubusercontent.com/u/5746567?v=3","gravatar_id":"","url":"https://api.github.com/users/twalthr","html_url":"https://github.com/twalthr","followers_url":"https://api.github.com/users/twalthr/followers","following_url":"https://api.github.com/users/twalthr/following{/other_user}","gists_url":"https://api.github.com/users/twalthr/gists{/gist_id}","starred_url":"https://api.github.com/users/twalthr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twalthr/subscriptions","organizations_url":"https://api.github.com/users/twalthr/orgs","repos_url":"https://api.github.com/users/twalthr/repos","events_url":"https://api.github.com/users/twalthr/events{/privacy}","received_events_url":"https://api.github.com/users/twalthr/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/twalthr/flink","description":"Mirror of Apache Flink","fork":true,"url":"https://api.github.com/repos/twalthr/flink","forks_url":"https://api.github.com/repos/twalthr/flink/forks","keys_url":"https://api.github.com/repos/twalthr/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/twalthr/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/twalthr/flink/teams","hooks_url":"https://api.github.com/repos/twalthr/flink/hooks","issue_events_url":"https://api.github.com/repos/twalthr/flink/issues/events{/number}","events_url":"https://api.github.com/repos/twalthr/flink/events","assignees_url":"https://api.github.com/repos/twalthr/flink/assignees{/user}","branches_url":"https://api.github.com/repos/twalthr/flink/branches{/branch}","tags_url":"https://api.github.com/repos/twalthr/flink/tags","blobs_url":"https://api.github.com/repos/twalthr/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/twalthr/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/twalthr/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/twalthr/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/twalthr/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/twalthr/flink/languages","stargazers_url":"https://api.github.com/repos/twalthr/flink/stargazers","contributors_url":"https://api.github.com/repos/twalthr/flink/contributors","subscribers_url":"https://api.github.com/repos/twalthr/flink/subscribers","subscription_url":"https://api.github.com/repos/twalthr/flink/subscription","commits_url":"https://api.github.com/repos/twalthr/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/twalthr/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/twalthr/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/twalthr/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/twalthr/flink/contents/{+path}","compare_url":"https://api.github.com/repos/twalthr/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/twalthr/flink/merges","archive_url":"https://api.github.com/repos/twalthr/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/twalthr/flink/downloads","issues_url":"https://api.github.com/repos/twalthr/flink/issues{/number}","pulls_url":"https://api.github.com/repos/twalthr/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/twalthr/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/twalthr/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/twalthr/flink/labels{/name}","releases_url":"https://api.github.com/repos/twalthr/flink/releases{/id}","deployments_url":"https://api.github.com/repos/twalthr/flink/deployments","created_at":"2014-06-09T19:08:51Z","updated_at":"2016-11-15T11:08:50Z","pushed_at":"2017-05-30T15:26:16Z","git_url":"git://github.com/twalthr/flink.git","ssh_url":"git@github.com:twalthr/flink.git","clone_url":"https://github.com/twalthr/flink.git","svn_url":"https://github.com/twalthr/flink","homepage":null,"size":121618,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:tableDocs","ref":"tableDocs","sha":"e5f24713ac4c487dd778de448c70629bd04ea4ea","user":{"login":"apache","id":47359,"avatar_url":"https://avatars3.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20587599,"name":"flink","full_name":"apache/flink","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars3.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/flink","description":"Mirror of Apache Flink","fork":false,"url":"https://api.github.com/repos/apache/flink","forks_url":"https://api.github.com/repos/apache/flink/forks","keys_url":"https://api.github.com/repos/apache/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/flink/teams","hooks_url":"https://api.github.com/repos/apache/flink/hooks","issue_events_url":"https://api.github.com/repos/apache/flink/issues/events{/number}","events_url":"https://api.github.com/repos/apache/flink/events","assignees_url":"https://api.github.com/repos/apache/flink/assignees{/user}","branches_url":"https://api.github.com/repos/apache/flink/branches{/branch}","tags_url":"https://api.github.com/repos/apache/flink/tags","blobs_url":"https://api.github.com/repos/apache/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/flink/languages","stargazers_url":"https://api.github.com/repos/apache/flink/stargazers","contributors_url":"https://api.github.com/repos/apache/flink/contributors","subscribers_url":"https://api.github.com/repos/apache/flink/subscribers","subscription_url":"https://api.github.com/repos/apache/flink/subscription","commits_url":"https://api.github.com/repos/apache/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/flink/contents/{+path}","compare_url":"https://api.github.com/repos/apache/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/flink/merges","archive_url":"https://api.github.com/repos/apache/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/flink/downloads","issues_url":"https://api.github.com/repos/apache/flink/issues{/number}","pulls_url":"https://api.github.com/repos/apache/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/flink/labels{/name}","releases_url":"https://api.github.com/repos/apache/flink/releases{/id}","deployments_url":"https://api.github.com/repos/apache/flink/deployments","created_at":"2014-06-07T07:00:10Z","updated_at":"2017-05-30T08:13:01Z","pushed_at":"2017-05-30T16:03:32Z","git_url":"git://github.com/apache/flink.git","ssh_url":"git@github.com:apache/flink.git","clone_url":"https://github.com/apache/flink.git","svn_url":"https://github.com/apache/flink","homepage":null,"size":124737,"stargazers_count":2353,"watchers_count":2353,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1653,"mirror_url":"git://git.apache.org/flink.git","open_issues_count":228,"forks":1653,"open_issues":228,"watchers":2353,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/4020"},"html":{"href":"https://github.com/apache/flink/pull/4020"},"issue":{"href":"https://api.github.com/repos/apache/flink/issues/4020"},"comments":{"href":"https://api.github.com/repos/apache/flink/issues/4020/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/flink/pulls/4020/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/flink/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/flink/pulls/4020/commits"},"statuses":{"href":"https://api.github.com/repos/apache/flink/statuses/4041c56ac00e98745a01b4de05334986545bacdd"}}}}
{"actor":{"display_login":"alpinegizmo"}}