{"action":"created","comment":{"url":"https://api.github.com/repos/atlassian/gostatsd/pulls/comments/143373028","pull_request_review_id":67875556,"id":143373028,"diff_hunk":"@@ -96,99 +83,24 @@ func (mr *MetricReceiver) Receive(ctx context.Context, c net.PacketConn) {\n \t\t\tcontinue\n \t\t}\n \t\t// TODO consider updating counter for every N-th iteration to reduce contention\n-\t\tatomic.AddUint64(&mr.packetsReceived, uint64(packetCount))\n-\t\tatomic.StoreInt64(&mr.lastPacket, time.Now().UnixNano())\n-\t\tatomic.AddUint64(&mr.batchesRead, 1)\n+\t\tatomic.AddUint64(&dr.packetsReceived, uint64(packetCount))\n+\t\tatomic.StoreInt64(&dr.lastPacket, time.Now().UnixNano())\n+\t\tatomic.AddUint64(&dr.batchesRead, 1)\n \t\tfor i := 0; i < packetCount; i++ {\n \t\t\taddr := messages[i].Addr\n \t\t\tnbytes := messages[i].N\n \t\t\tbuf := messages[i].Buffers[0]\n-\t\t\tif err := mr.handlePacket(ctx, addr, buf[:nbytes]); err != nil {\n-\t\t\t\tif err == context.Canceled || err == context.DeadlineExceeded {\n-\t\t\t\t\treturn\n-\t\t\t\t}\n-\t\t\t\tlog.Warnf(\"Failed to handle packet: %v\", err)\n+\t\t\tmsg := make([]byte, nbytes)\n+\t\t\tcopy(msg, buf)","path":"pkg/statsd/receiver.go","position":118,"original_position":118,"commit_id":"ba50eaca093ec4307d4e94b294ea60a072c0a993","original_commit_id":"ba50eaca093ec4307d4e94b294ea60a072c0a993","user":{"login":"tiedotguy","id":29549188,"avatar_url":"https://avatars1.githubusercontent.com/u/29549188?v=4","gravatar_id":"","url":"https://api.github.com/users/tiedotguy","html_url":"https://github.com/tiedotguy","followers_url":"https://api.github.com/users/tiedotguy/followers","following_url":"https://api.github.com/users/tiedotguy/following{/other_user}","gists_url":"https://api.github.com/users/tiedotguy/gists{/gist_id}","starred_url":"https://api.github.com/users/tiedotguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiedotguy/subscriptions","organizations_url":"https://api.github.com/users/tiedotguy/orgs","repos_url":"https://api.github.com/users/tiedotguy/repos","events_url":"https://api.github.com/users/tiedotguy/events{/privacy}","received_events_url":"https://api.github.com/users/tiedotguy/received_events","type":"User","site_admin":false},"body":"I think it's more the copying overhead?  I'm vaguely curious about the GC overhead, I think it's up there, but I suspect the channel overhead is the big one - once this is in, I'm (or if you want to, go for it) going to try adding a small local buffer on the decoders, so multiple metrics are sent to the channel at a time.","created_at":"2017-10-09T01:08:28Z","updated_at":"2017-10-09T01:08:28Z","html_url":"https://github.com/atlassian/gostatsd/pull/141#discussion_r143373028","pull_request_url":"https://api.github.com/repos/atlassian/gostatsd/pulls/141","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/atlassian/gostatsd/pulls/comments/143373028"},"html":{"href":"https://github.com/atlassian/gostatsd/pull/141#discussion_r143373028"},"pull_request":{"href":"https://api.github.com/repos/atlassian/gostatsd/pulls/141"}},"in_reply_to_id":143103063},"pull_request":{"url":"https://api.github.com/repos/atlassian/gostatsd/pulls/141","id":145063675,"html_url":"https://github.com/atlassian/gostatsd/pull/141","diff_url":"https://github.com/atlassian/gostatsd/pull/141.diff","patch_url":"https://github.com/atlassian/gostatsd/pull/141.patch","issue_url":"https://api.github.com/repos/atlassian/gostatsd/issues/141","number":141,"state":"open","locked":false,"title":"Separate MetricReceiver into DatagramReceiver and DatagramParser","user":{"login":"brentthew","id":1680042,"avatar_url":"https://avatars2.githubusercontent.com/u/1680042?v=4","gravatar_id":"","url":"https://api.github.com/users/brentthew","html_url":"https://github.com/brentthew","followers_url":"https://api.github.com/users/brentthew/followers","following_url":"https://api.github.com/users/brentthew/following{/other_user}","gists_url":"https://api.github.com/users/brentthew/gists{/gist_id}","starred_url":"https://api.github.com/users/brentthew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brentthew/subscriptions","organizations_url":"https://api.github.com/users/brentthew/orgs","repos_url":"https://api.github.com/users/brentthew/repos","events_url":"https://api.github.com/users/brentthew/events{/privacy}","received_events_url":"https://api.github.com/users/brentthew/received_events","type":"User","site_admin":false},"body":"Current MetricReceiver:\r\n1. Reads off datagrams \r\n2. Splits them into lines \r\n3. Lexes lines into metrics/events\r\n4. Passes off to handler \r\n\r\nSteps 2-4 are cpu bound and will benefit from higher concurrency. Step 1 is not, and we've seen higher concurrency lead to lower socket read performance. \r\n\r\nI've split these into \r\n\r\nDatagramReceiver:\r\n1. Reads datagram from socket\r\n2. Pushes them to a chan\r\n\r\nand \r\n\r\nDatagramParser: \r\n1. Read datagrams from chan\r\n2. Splits them into lines \r\n3. Lexes lines into metrics/events\r\n4. Passes off to handler \r\n\r\nWhich can be scaled individually. I've set defaults of DatagramParser (MaxParsers) as num cpu, and DatagramReceiver (MaxReaders) to 8. Optimal value of MaxReaders may need some investigation. We found 8 to be performant in our testing. ","created_at":"2017-10-06T03:04:31Z","updated_at":"2017-10-09T01:08:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"997f67752304be41f3242d4109e2fed42f28c6f8","assignee":null,"assignees":[],"requested_reviewers":[{"login":"adampetrovic","id":88030,"avatar_url":"https://avatars1.githubusercontent.com/u/88030?v=4","gravatar_id":"","url":"https://api.github.com/users/adampetrovic","html_url":"https://github.com/adampetrovic","followers_url":"https://api.github.com/users/adampetrovic/followers","following_url":"https://api.github.com/users/adampetrovic/following{/other_user}","gists_url":"https://api.github.com/users/adampetrovic/gists{/gist_id}","starred_url":"https://api.github.com/users/adampetrovic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adampetrovic/subscriptions","organizations_url":"https://api.github.com/users/adampetrovic/orgs","repos_url":"https://api.github.com/users/adampetrovic/repos","events_url":"https://api.github.com/users/adampetrovic/events{/privacy}","received_events_url":"https://api.github.com/users/adampetrovic/received_events","type":"User","site_admin":false},{"login":"aelse","id":165789,"avatar_url":"https://avatars1.githubusercontent.com/u/165789?v=4","gravatar_id":"","url":"https://api.github.com/users/aelse","html_url":"https://github.com/aelse","followers_url":"https://api.github.com/users/aelse/followers","following_url":"https://api.github.com/users/aelse/following{/other_user}","gists_url":"https://api.github.com/users/aelse/gists{/gist_id}","starred_url":"https://api.github.com/users/aelse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aelse/subscriptions","organizations_url":"https://api.github.com/users/aelse/orgs","repos_url":"https://api.github.com/users/aelse/repos","events_url":"https://api.github.com/users/aelse/events{/privacy}","received_events_url":"https://api.github.com/users/aelse/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/atlassian/gostatsd/pulls/141/commits","review_comments_url":"https://api.github.com/repos/atlassian/gostatsd/pulls/141/comments","review_comment_url":"https://api.github.com/repos/atlassian/gostatsd/pulls/comments{/number}","comments_url":"https://api.github.com/repos/atlassian/gostatsd/issues/141/comments","statuses_url":"https://api.github.com/repos/atlassian/gostatsd/statuses/ba50eaca093ec4307d4e94b294ea60a072c0a993","head":{"label":"atlassian:separate-receiver-parser","ref":"separate-receiver-parser","sha":"ba50eaca093ec4307d4e94b294ea60a072c0a993","user":{"login":"atlassian","id":168166,"avatar_url":"https://avatars3.githubusercontent.com/u/168166?v=4","gravatar_id":"","url":"https://api.github.com/users/atlassian","html_url":"https://github.com/atlassian","followers_url":"https://api.github.com/users/atlassian/followers","following_url":"https://api.github.com/users/atlassian/following{/other_user}","gists_url":"https://api.github.com/users/atlassian/gists{/gist_id}","starred_url":"https://api.github.com/users/atlassian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atlassian/subscriptions","organizations_url":"https://api.github.com/users/atlassian/orgs","repos_url":"https://api.github.com/users/atlassian/repos","events_url":"https://api.github.com/users/atlassian/events{/privacy}","received_events_url":"https://api.github.com/users/atlassian/received_events","type":"Organization","site_admin":false},"repo":{"id":55813517,"name":"gostatsd","full_name":"atlassian/gostatsd","owner":{"login":"atlassian","id":168166,"avatar_url":"https://avatars3.githubusercontent.com/u/168166?v=4","gravatar_id":"","url":"https://api.github.com/users/atlassian","html_url":"https://github.com/atlassian","followers_url":"https://api.github.com/users/atlassian/followers","following_url":"https://api.github.com/users/atlassian/following{/other_user}","gists_url":"https://api.github.com/users/atlassian/gists{/gist_id}","starred_url":"https://api.github.com/users/atlassian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atlassian/subscriptions","organizations_url":"https://api.github.com/users/atlassian/orgs","repos_url":"https://api.github.com/users/atlassian/repos","events_url":"https://api.github.com/users/atlassian/events{/privacy}","received_events_url":"https://api.github.com/users/atlassian/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/atlassian/gostatsd","description":"An implementation of Etsy's statsd in Go with tags support","fork":false,"url":"https://api.github.com/repos/atlassian/gostatsd","forks_url":"https://api.github.com/repos/atlassian/gostatsd/forks","keys_url":"https://api.github.com/repos/atlassian/gostatsd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/atlassian/gostatsd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/atlassian/gostatsd/teams","hooks_url":"https://api.github.com/repos/atlassian/gostatsd/hooks","issue_events_url":"https://api.github.com/repos/atlassian/gostatsd/issues/events{/number}","events_url":"https://api.github.com/repos/atlassian/gostatsd/events","assignees_url":"https://api.github.com/repos/atlassian/gostatsd/assignees{/user}","branches_url":"https://api.github.com/repos/atlassian/gostatsd/branches{/branch}","tags_url":"https://api.github.com/repos/atlassian/gostatsd/tags","blobs_url":"https://api.github.com/repos/atlassian/gostatsd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/atlassian/gostatsd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/atlassian/gostatsd/git/refs{/sha}","trees_url":"https://api.github.com/repos/atlassian/gostatsd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/atlassian/gostatsd/statuses/{sha}","languages_url":"https://api.github.com/repos/atlassian/gostatsd/languages","stargazers_url":"https://api.github.com/repos/atlassian/gostatsd/stargazers","contributors_url":"https://api.github.com/repos/atlassian/gostatsd/contributors","subscribers_url":"https://api.github.com/repos/atlassian/gostatsd/subscribers","subscription_url":"https://api.github.com/repos/atlassian/gostatsd/subscription","commits_url":"https://api.github.com/repos/atlassian/gostatsd/commits{/sha}","git_commits_url":"https://api.github.com/repos/atlassian/gostatsd/git/commits{/sha}","comments_url":"https://api.github.com/repos/atlassian/gostatsd/comments{/number}","issue_comment_url":"https://api.github.com/repos/atlassian/gostatsd/issues/comments{/number}","contents_url":"https://api.github.com/repos/atlassian/gostatsd/contents/{+path}","compare_url":"https://api.github.com/repos/atlassian/gostatsd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/atlassian/gostatsd/merges","archive_url":"https://api.github.com/repos/atlassian/gostatsd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/atlassian/gostatsd/downloads","issues_url":"https://api.github.com/repos/atlassian/gostatsd/issues{/number}","pulls_url":"https://api.github.com/repos/atlassian/gostatsd/pulls{/number}","milestones_url":"https://api.github.com/repos/atlassian/gostatsd/milestones{/number}","notifications_url":"https://api.github.com/repos/atlassian/gostatsd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/atlassian/gostatsd/labels{/name}","releases_url":"https://api.github.com/repos/atlassian/gostatsd/releases{/id}","deployments_url":"https://api.github.com/repos/atlassian/gostatsd/deployments","created_at":"2016-04-08T22:29:32Z","updated_at":"2017-09-29T15:33:13Z","pushed_at":"2017-10-06T15:19:18Z","git_url":"git://github.com/atlassian/gostatsd.git","ssh_url":"git@github.com:atlassian/gostatsd.git","clone_url":"https://github.com/atlassian/gostatsd.git","svn_url":"https://github.com/atlassian/gostatsd","homepage":"","size":674,"stargazers_count":114,"watchers_count":114,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":29,"mirror_url":null,"open_issues_count":19,"forks":29,"open_issues":19,"watchers":114,"default_branch":"master"}},"base":{"label":"atlassian:master","ref":"master","sha":"cc247eae677725584ef14ce5e170217a044aed5d","user":{"login":"atlassian","id":168166,"avatar_url":"https://avatars3.githubusercontent.com/u/168166?v=4","gravatar_id":"","url":"https://api.github.com/users/atlassian","html_url":"https://github.com/atlassian","followers_url":"https://api.github.com/users/atlassian/followers","following_url":"https://api.github.com/users/atlassian/following{/other_user}","gists_url":"https://api.github.com/users/atlassian/gists{/gist_id}","starred_url":"https://api.github.com/users/atlassian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atlassian/subscriptions","organizations_url":"https://api.github.com/users/atlassian/orgs","repos_url":"https://api.github.com/users/atlassian/repos","events_url":"https://api.github.com/users/atlassian/events{/privacy}","received_events_url":"https://api.github.com/users/atlassian/received_events","type":"Organization","site_admin":false},"repo":{"id":55813517,"name":"gostatsd","full_name":"atlassian/gostatsd","owner":{"login":"atlassian","id":168166,"avatar_url":"https://avatars3.githubusercontent.com/u/168166?v=4","gravatar_id":"","url":"https://api.github.com/users/atlassian","html_url":"https://github.com/atlassian","followers_url":"https://api.github.com/users/atlassian/followers","following_url":"https://api.github.com/users/atlassian/following{/other_user}","gists_url":"https://api.github.com/users/atlassian/gists{/gist_id}","starred_url":"https://api.github.com/users/atlassian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atlassian/subscriptions","organizations_url":"https://api.github.com/users/atlassian/orgs","repos_url":"https://api.github.com/users/atlassian/repos","events_url":"https://api.github.com/users/atlassian/events{/privacy}","received_events_url":"https://api.github.com/users/atlassian/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/atlassian/gostatsd","description":"An implementation of Etsy's statsd in Go with tags support","fork":false,"url":"https://api.github.com/repos/atlassian/gostatsd","forks_url":"https://api.github.com/repos/atlassian/gostatsd/forks","keys_url":"https://api.github.com/repos/atlassian/gostatsd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/atlassian/gostatsd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/atlassian/gostatsd/teams","hooks_url":"https://api.github.com/repos/atlassian/gostatsd/hooks","issue_events_url":"https://api.github.com/repos/atlassian/gostatsd/issues/events{/number}","events_url":"https://api.github.com/repos/atlassian/gostatsd/events","assignees_url":"https://api.github.com/repos/atlassian/gostatsd/assignees{/user}","branches_url":"https://api.github.com/repos/atlassian/gostatsd/branches{/branch}","tags_url":"https://api.github.com/repos/atlassian/gostatsd/tags","blobs_url":"https://api.github.com/repos/atlassian/gostatsd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/atlassian/gostatsd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/atlassian/gostatsd/git/refs{/sha}","trees_url":"https://api.github.com/repos/atlassian/gostatsd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/atlassian/gostatsd/statuses/{sha}","languages_url":"https://api.github.com/repos/atlassian/gostatsd/languages","stargazers_url":"https://api.github.com/repos/atlassian/gostatsd/stargazers","contributors_url":"https://api.github.com/repos/atlassian/gostatsd/contributors","subscribers_url":"https://api.github.com/repos/atlassian/gostatsd/subscribers","subscription_url":"https://api.github.com/repos/atlassian/gostatsd/subscription","commits_url":"https://api.github.com/repos/atlassian/gostatsd/commits{/sha}","git_commits_url":"https://api.github.com/repos/atlassian/gostatsd/git/commits{/sha}","comments_url":"https://api.github.com/repos/atlassian/gostatsd/comments{/number}","issue_comment_url":"https://api.github.com/repos/atlassian/gostatsd/issues/comments{/number}","contents_url":"https://api.github.com/repos/atlassian/gostatsd/contents/{+path}","compare_url":"https://api.github.com/repos/atlassian/gostatsd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/atlassian/gostatsd/merges","archive_url":"https://api.github.com/repos/atlassian/gostatsd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/atlassian/gostatsd/downloads","issues_url":"https://api.github.com/repos/atlassian/gostatsd/issues{/number}","pulls_url":"https://api.github.com/repos/atlassian/gostatsd/pulls{/number}","milestones_url":"https://api.github.com/repos/atlassian/gostatsd/milestones{/number}","notifications_url":"https://api.github.com/repos/atlassian/gostatsd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/atlassian/gostatsd/labels{/name}","releases_url":"https://api.github.com/repos/atlassian/gostatsd/releases{/id}","deployments_url":"https://api.github.com/repos/atlassian/gostatsd/deployments","created_at":"2016-04-08T22:29:32Z","updated_at":"2017-09-29T15:33:13Z","pushed_at":"2017-10-06T15:19:18Z","git_url":"git://github.com/atlassian/gostatsd.git","ssh_url":"git@github.com:atlassian/gostatsd.git","clone_url":"https://github.com/atlassian/gostatsd.git","svn_url":"https://github.com/atlassian/gostatsd","homepage":"","size":674,"stargazers_count":114,"watchers_count":114,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":29,"mirror_url":null,"open_issues_count":19,"forks":29,"open_issues":19,"watchers":114,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/atlassian/gostatsd/pulls/141"},"html":{"href":"https://github.com/atlassian/gostatsd/pull/141"},"issue":{"href":"https://api.github.com/repos/atlassian/gostatsd/issues/141"},"comments":{"href":"https://api.github.com/repos/atlassian/gostatsd/issues/141/comments"},"review_comments":{"href":"https://api.github.com/repos/atlassian/gostatsd/pulls/141/comments"},"review_comment":{"href":"https://api.github.com/repos/atlassian/gostatsd/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/atlassian/gostatsd/pulls/141/commits"},"statuses":{"href":"https://api.github.com/repos/atlassian/gostatsd/statuses/ba50eaca093ec4307d4e94b294ea60a072c0a993"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"tiedotguy"}}