{"action":"created","comment":{"url":"https://api.github.com/repos/moby/moby/pulls/comments/155965630","pull_request_review_id":82368346,"id":155965630,"diff_hunk":"@@ -450,47 +443,42 @@ func (l *logStream) collectBatch() {\n // bytes per event (defined in maximumBytesPerEvent).  There is a fixed per-event\n // byte overhead (defined in perEventBytes) which is accounted for in split- and\n // batch-calculations.\n-func (l *logStream) processEvent(events []wrappedEvent, unprocessedLine []byte, timestamp int64) []wrappedEvent {\n-\tbytes := 0\n+func (l *logStream) processEvent(batch *eventBatch, unprocessedLine []byte, timestamp int64) {\n \tfor len(unprocessedLine) > 0 {\n \t\t// Split line length so it does not exceed the maximum\n \t\tlineBytes := len(unprocessedLine)\n \t\tif lineBytes > maximumBytesPerEvent {\n \t\t\tlineBytes = maximumBytesPerEvent\n \t\t}\n \t\tline := unprocessedLine[:lineBytes]\n-\t\tunprocessedLine = unprocessedLine[lineBytes:]\n-\t\tif (len(events) >= maximumLogEventsPerPut) || (bytes+lineBytes+perEventBytes > maximumBytesPerPut) {\n-\t\t\t// Publish an existing batch if it's already over the maximum number of events or if adding this\n-\t\t\t// event would push it over the maximum number of total bytes.\n-\t\t\tl.publishBatch(events)\n-\t\t\tevents = events[:0]\n-\t\t\tbytes = 0\n-\t\t}\n-\t\tevents = append(events, wrappedEvent{\n+\n+\t\tevent := wrappedEvent{\n \t\t\tinputLogEvent: &cloudwatchlogs.InputLogEvent{\n \t\t\t\tMessage:   aws.String(string(line)),\n \t\t\t\tTimestamp: aws.Int64(timestamp),\n \t\t\t},\n-\t\t\tinsertOrder: len(events),\n-\t\t})\n-\t\tbytes += (lineBytes + perEventBytes)\n+\t\t\tinsertOrder: batch.count(),\n+\t\t}\n+\n+\t\tswitch batch.add(event, lineBytes) {\n+\t\tcase errBatchCountTooLarge, errBatchSizeTooLarge:","path":"daemon/logger/awslogs/cloudwatchlogs.go","position":416,"original_position":416,"commit_id":"0b4df70ac0a79757d0bce4ae84471f426976f14f","original_commit_id":"0b4df70ac0a79757d0bce4ae84471f426976f14f","user":{"login":"samuelkarp","id":737750,"avatar_url":"https://avatars3.githubusercontent.com/u/737750?v=4","gravatar_id":"","url":"https://api.github.com/users/samuelkarp","html_url":"https://github.com/samuelkarp","followers_url":"https://api.github.com/users/samuelkarp/followers","following_url":"https://api.github.com/users/samuelkarp/following{/other_user}","gists_url":"https://api.github.com/users/samuelkarp/gists{/gist_id}","starred_url":"https://api.github.com/users/samuelkarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samuelkarp/subscriptions","organizations_url":"https://api.github.com/users/samuelkarp/orgs","repos_url":"https://api.github.com/users/samuelkarp/repos","events_url":"https://api.github.com/users/samuelkarp/events{/privacy}","received_events_url":"https://api.github.com/users/samuelkarp/received_events","type":"User","site_admin":false},"body":"Both of these errors result in the exact same code being run.  Why not just return `bool` so you can do\r\n\r\n```go\r\nif ok := batch.add(event, lineBytes); ok {\r\n\tunprocessedLine = unprocessedLine[lineBytes:]\r\n} else {\r\n\tl.publishBatch(batch)\r\n}\r\n```","created_at":"2017-12-11T00:15:46Z","updated_at":"2017-12-11T00:17:51Z","html_url":"https://github.com/moby/moby/pull/35726#discussion_r155965630","pull_request_url":"https://api.github.com/repos/moby/moby/pulls/35726","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/moby/moby/pulls/comments/155965630"},"html":{"href":"https://github.com/moby/moby/pull/35726#discussion_r155965630"},"pull_request":{"href":"https://api.github.com/repos/moby/moby/pulls/35726"}}},"pull_request":{"url":"https://api.github.com/repos/moby/moby/pulls/35726","id":156873101,"html_url":"https://github.com/moby/moby/pull/35726","diff_url":"https://github.com/moby/moby/pull/35726.diff","patch_url":"https://github.com/moby/moby/pull/35726.patch","issue_url":"https://api.github.com/repos/moby/moby/issues/35726","number":35726,"state":"open","locked":false,"title":"Fix awslogs batch size calculation","user":{"login":"jahkeup","id":1139752,"avatar_url":"https://avatars2.githubusercontent.com/u/1139752?v=4","gravatar_id":"","url":"https://api.github.com/users/jahkeup","html_url":"https://github.com/jahkeup","followers_url":"https://api.github.com/users/jahkeup/followers","following_url":"https://api.github.com/users/jahkeup/following{/other_user}","gists_url":"https://api.github.com/users/jahkeup/gists{/gist_id}","starred_url":"https://api.github.com/users/jahkeup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jahkeup/subscriptions","organizations_url":"https://api.github.com/users/jahkeup/orgs","repos_url":"https://api.github.com/users/jahkeup/repos","events_url":"https://api.github.com/users/jahkeup/events{/privacy}","received_events_url":"https://api.github.com/users/jahkeup/received_events","type":"User","site_admin":false},"body":"Fixes https://github.com/moby/moby/issues/35725\r\n\r\n**- What I did**\r\n\r\n* Added a type to encapsulate the batch and its associated size counter.\r\n* Cleaned up types and declarations for code cleanliness\r\n\r\n**- How I did it**\r\n\r\nThe added type holds the batching mechanism and the counter used for determining whether or not to submit a batch of log events.\r\n\r\n**- How to verify it**\r\n\r\nThe change can be verified by using the steps from #35725 to reproduce the issue and then comparing it to a build from this changeset. The appropriate test and test mock have been updated to validate this behavior.\r\n\r\n**- Description for the changelog**\r\nawslogs: fix batch size calculation for large logs\r\n\r\n**- A picture of a cute animal (not mandatory but encouraged)**\r\n<img height=\"200px\" src=\"https://i.imgur.com/f6VINwq.jpg\" alt=\"https://www.reddit.com/r/aww/comments/7hzq4m\" />\r\n\r\ncc @samuelkarp @adnxn ","created_at":"2017-12-07T00:08:40Z","updated_at":"2017-12-11T00:17:51Z","closed_at":null,"merged_at":null,"merge_commit_sha":"23a9a8346d5da362e5940e6aa58b8f03ba471139","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/moby/moby/pulls/35726/commits","review_comments_url":"https://api.github.com/repos/moby/moby/pulls/35726/comments","review_comment_url":"https://api.github.com/repos/moby/moby/pulls/comments{/number}","comments_url":"https://api.github.com/repos/moby/moby/issues/35726/comments","statuses_url":"https://api.github.com/repos/moby/moby/statuses/0b4df70ac0a79757d0bce4ae84471f426976f14f","head":{"label":"jahkeup:awslogs-batching","ref":"awslogs-batching","sha":"0b4df70ac0a79757d0bce4ae84471f426976f14f","user":{"login":"jahkeup","id":1139752,"avatar_url":"https://avatars2.githubusercontent.com/u/1139752?v=4","gravatar_id":"","url":"https://api.github.com/users/jahkeup","html_url":"https://github.com/jahkeup","followers_url":"https://api.github.com/users/jahkeup/followers","following_url":"https://api.github.com/users/jahkeup/following{/other_user}","gists_url":"https://api.github.com/users/jahkeup/gists{/gist_id}","starred_url":"https://api.github.com/users/jahkeup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jahkeup/subscriptions","organizations_url":"https://api.github.com/users/jahkeup/orgs","repos_url":"https://api.github.com/users/jahkeup/repos","events_url":"https://api.github.com/users/jahkeup/events{/privacy}","received_events_url":"https://api.github.com/users/jahkeup/received_events","type":"User","site_admin":false},"repo":{"id":103456426,"name":"moby","full_name":"jahkeup/moby","owner":{"login":"jahkeup","id":1139752,"avatar_url":"https://avatars2.githubusercontent.com/u/1139752?v=4","gravatar_id":"","url":"https://api.github.com/users/jahkeup","html_url":"https://github.com/jahkeup","followers_url":"https://api.github.com/users/jahkeup/followers","following_url":"https://api.github.com/users/jahkeup/following{/other_user}","gists_url":"https://api.github.com/users/jahkeup/gists{/gist_id}","starred_url":"https://api.github.com/users/jahkeup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jahkeup/subscriptions","organizations_url":"https://api.github.com/users/jahkeup/orgs","repos_url":"https://api.github.com/users/jahkeup/repos","events_url":"https://api.github.com/users/jahkeup/events{/privacy}","received_events_url":"https://api.github.com/users/jahkeup/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jahkeup/moby","description":"Moby Project - a collaborative project for the container ecosystem to assemble container-based systems","fork":true,"url":"https://api.github.com/repos/jahkeup/moby","forks_url":"https://api.github.com/repos/jahkeup/moby/forks","keys_url":"https://api.github.com/repos/jahkeup/moby/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jahkeup/moby/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jahkeup/moby/teams","hooks_url":"https://api.github.com/repos/jahkeup/moby/hooks","issue_events_url":"https://api.github.com/repos/jahkeup/moby/issues/events{/number}","events_url":"https://api.github.com/repos/jahkeup/moby/events","assignees_url":"https://api.github.com/repos/jahkeup/moby/assignees{/user}","branches_url":"https://api.github.com/repos/jahkeup/moby/branches{/branch}","tags_url":"https://api.github.com/repos/jahkeup/moby/tags","blobs_url":"https://api.github.com/repos/jahkeup/moby/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jahkeup/moby/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jahkeup/moby/git/refs{/sha}","trees_url":"https://api.github.com/repos/jahkeup/moby/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jahkeup/moby/statuses/{sha}","languages_url":"https://api.github.com/repos/jahkeup/moby/languages","stargazers_url":"https://api.github.com/repos/jahkeup/moby/stargazers","contributors_url":"https://api.github.com/repos/jahkeup/moby/contributors","subscribers_url":"https://api.github.com/repos/jahkeup/moby/subscribers","subscription_url":"https://api.github.com/repos/jahkeup/moby/subscription","commits_url":"https://api.github.com/repos/jahkeup/moby/commits{/sha}","git_commits_url":"https://api.github.com/repos/jahkeup/moby/git/commits{/sha}","comments_url":"https://api.github.com/repos/jahkeup/moby/comments{/number}","issue_comment_url":"https://api.github.com/repos/jahkeup/moby/issues/comments{/number}","contents_url":"https://api.github.com/repos/jahkeup/moby/contents/{+path}","compare_url":"https://api.github.com/repos/jahkeup/moby/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jahkeup/moby/merges","archive_url":"https://api.github.com/repos/jahkeup/moby/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jahkeup/moby/downloads","issues_url":"https://api.github.com/repos/jahkeup/moby/issues{/number}","pulls_url":"https://api.github.com/repos/jahkeup/moby/pulls{/number}","milestones_url":"https://api.github.com/repos/jahkeup/moby/milestones{/number}","notifications_url":"https://api.github.com/repos/jahkeup/moby/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jahkeup/moby/labels{/name}","releases_url":"https://api.github.com/repos/jahkeup/moby/releases{/id}","deployments_url":"https://api.github.com/repos/jahkeup/moby/deployments","created_at":"2017-09-13T22:01:40Z","updated_at":"2017-09-13T22:02:02Z","pushed_at":"2017-12-08T23:37:48Z","git_url":"git://github.com/jahkeup/moby.git","ssh_url":"git@github.com:jahkeup/moby.git","clone_url":"https://github.com/jahkeup/moby.git","svn_url":"https://github.com/jahkeup/moby","homepage":"https://mobyproject.org/","size":137708,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"moby:master","ref":"master","sha":"c7256dc38ed8fe5a428c85002a674c19754d8dd3","user":{"login":"moby","id":27259197,"avatar_url":"https://avatars1.githubusercontent.com/u/27259197?v=4","gravatar_id":"","url":"https://api.github.com/users/moby","html_url":"https://github.com/moby","followers_url":"https://api.github.com/users/moby/followers","following_url":"https://api.github.com/users/moby/following{/other_user}","gists_url":"https://api.github.com/users/moby/gists{/gist_id}","starred_url":"https://api.github.com/users/moby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moby/subscriptions","organizations_url":"https://api.github.com/users/moby/orgs","repos_url":"https://api.github.com/users/moby/repos","events_url":"https://api.github.com/users/moby/events{/privacy}","received_events_url":"https://api.github.com/users/moby/received_events","type":"Organization","site_admin":false},"repo":{"id":7691631,"name":"moby","full_name":"moby/moby","owner":{"login":"moby","id":27259197,"avatar_url":"https://avatars1.githubusercontent.com/u/27259197?v=4","gravatar_id":"","url":"https://api.github.com/users/moby","html_url":"https://github.com/moby","followers_url":"https://api.github.com/users/moby/followers","following_url":"https://api.github.com/users/moby/following{/other_user}","gists_url":"https://api.github.com/users/moby/gists{/gist_id}","starred_url":"https://api.github.com/users/moby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moby/subscriptions","organizations_url":"https://api.github.com/users/moby/orgs","repos_url":"https://api.github.com/users/moby/repos","events_url":"https://api.github.com/users/moby/events{/privacy}","received_events_url":"https://api.github.com/users/moby/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/moby/moby","description":"Moby Project - a collaborative project for the container ecosystem to assemble container-based systems","fork":false,"url":"https://api.github.com/repos/moby/moby","forks_url":"https://api.github.com/repos/moby/moby/forks","keys_url":"https://api.github.com/repos/moby/moby/keys{/key_id}","collaborators_url":"https://api.github.com/repos/moby/moby/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/moby/moby/teams","hooks_url":"https://api.github.com/repos/moby/moby/hooks","issue_events_url":"https://api.github.com/repos/moby/moby/issues/events{/number}","events_url":"https://api.github.com/repos/moby/moby/events","assignees_url":"https://api.github.com/repos/moby/moby/assignees{/user}","branches_url":"https://api.github.com/repos/moby/moby/branches{/branch}","tags_url":"https://api.github.com/repos/moby/moby/tags","blobs_url":"https://api.github.com/repos/moby/moby/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/moby/moby/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/moby/moby/git/refs{/sha}","trees_url":"https://api.github.com/repos/moby/moby/git/trees{/sha}","statuses_url":"https://api.github.com/repos/moby/moby/statuses/{sha}","languages_url":"https://api.github.com/repos/moby/moby/languages","stargazers_url":"https://api.github.com/repos/moby/moby/stargazers","contributors_url":"https://api.github.com/repos/moby/moby/contributors","subscribers_url":"https://api.github.com/repos/moby/moby/subscribers","subscription_url":"https://api.github.com/repos/moby/moby/subscription","commits_url":"https://api.github.com/repos/moby/moby/commits{/sha}","git_commits_url":"https://api.github.com/repos/moby/moby/git/commits{/sha}","comments_url":"https://api.github.com/repos/moby/moby/comments{/number}","issue_comment_url":"https://api.github.com/repos/moby/moby/issues/comments{/number}","contents_url":"https://api.github.com/repos/moby/moby/contents/{+path}","compare_url":"https://api.github.com/repos/moby/moby/compare/{base}...{head}","merges_url":"https://api.github.com/repos/moby/moby/merges","archive_url":"https://api.github.com/repos/moby/moby/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/moby/moby/downloads","issues_url":"https://api.github.com/repos/moby/moby/issues{/number}","pulls_url":"https://api.github.com/repos/moby/moby/pulls{/number}","milestones_url":"https://api.github.com/repos/moby/moby/milestones{/number}","notifications_url":"https://api.github.com/repos/moby/moby/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/moby/moby/labels{/name}","releases_url":"https://api.github.com/repos/moby/moby/releases{/id}","deployments_url":"https://api.github.com/repos/moby/moby/deployments","created_at":"2013-01-18T18:10:57Z","updated_at":"2017-12-10T22:53:45Z","pushed_at":"2017-12-10T18:51:14Z","git_url":"git://github.com/moby/moby.git","ssh_url":"git@github.com:moby/moby.git","clone_url":"https://github.com/moby/moby.git","svn_url":"https://github.com/moby/moby","homepage":"https://mobyproject.org/","size":137829,"stargazers_count":46751,"watchers_count":46751,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":13877,"mirror_url":null,"archived":false,"open_issues_count":3024,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":13877,"open_issues":3024,"watchers":46751,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/moby/moby/pulls/35726"},"html":{"href":"https://github.com/moby/moby/pull/35726"},"issue":{"href":"https://api.github.com/repos/moby/moby/issues/35726"},"comments":{"href":"https://api.github.com/repos/moby/moby/issues/35726/comments"},"review_comments":{"href":"https://api.github.com/repos/moby/moby/pulls/35726/comments"},"review_comment":{"href":"https://api.github.com/repos/moby/moby/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/moby/moby/pulls/35726/commits"},"statuses":{"href":"https://api.github.com/repos/moby/moby/statuses/0b4df70ac0a79757d0bce4ae84471f426976f14f"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"samuelkarp"}}