{"action":"created","issue":{"url":"https://api.github.com/repos/vmware/vic/issues/5047","repository_url":"https://api.github.com/repos/vmware/vic","labels_url":"https://api.github.com/repos/vmware/vic/issues/5047/labels{/name}","comments_url":"https://api.github.com/repos/vmware/vic/issues/5047/comments","events_url":"https://api.github.com/repos/vmware/vic/issues/5047/events","html_url":"https://github.com/vmware/vic/pull/5047","id":227102864,"number":5047,"title":"WIP - state change missed [specific ci=1-10-Docker-PS]","user":{"login":"cgtexmex","id":726112,"avatar_url":"https://avatars1.githubusercontent.com/u/726112?v=3","gravatar_id":"","url":"https://api.github.com/users/cgtexmex","html_url":"https://github.com/cgtexmex","followers_url":"https://api.github.com/users/cgtexmex/followers","following_url":"https://api.github.com/users/cgtexmex/following{/other_user}","gists_url":"https://api.github.com/users/cgtexmex/gists{/gist_id}","starred_url":"https://api.github.com/users/cgtexmex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgtexmex/subscriptions","organizations_url":"https://api.github.com/users/cgtexmex/orgs","repos_url":"https://api.github.com/users/cgtexmex/repos","events_url":"https://api.github.com/users/cgtexmex/events{/privacy}","received_events_url":"https://api.github.com/users/cgtexmex/received_events","type":"User","site_admin":false},"labels":[{"id":353174341,"url":"https://api.github.com/repos/vmware/vic/labels/cla-not-required","name":"cla-not-required","color":"159818","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-05-08T16:33:57Z","updated_at":"2017-05-11T17:10:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/vmware/vic/pulls/5047","html_url":"https://github.com/vmware/vic/pull/5047","diff_url":"https://github.com/vmware/vic/pull/5047.diff","patch_url":"https://github.com/vmware/vic/pull/5047.patch"},"body":"For now added logging to attempt to recreate on CI\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/vmware/vic/issues/comments/300855172","html_url":"https://github.com/vmware/vic/pull/5047#issuecomment-300855172","issue_url":"https://api.github.com/repos/vmware/vic/issues/5047","id":300855172,"user":{"login":"cgtexmex","id":726112,"avatar_url":"https://avatars1.githubusercontent.com/u/726112?v=3","gravatar_id":"","url":"https://api.github.com/users/cgtexmex","html_url":"https://github.com/cgtexmex","followers_url":"https://api.github.com/users/cgtexmex/followers","following_url":"https://api.github.com/users/cgtexmex/following{/other_user}","gists_url":"https://api.github.com/users/cgtexmex/gists{/gist_id}","starred_url":"https://api.github.com/users/cgtexmex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgtexmex/subscriptions","organizations_url":"https://api.github.com/users/cgtexmex/orgs","repos_url":"https://api.github.com/users/cgtexmex/repos","events_url":"https://api.github.com/users/cgtexmex/events{/privacy}","received_events_url":"https://api.github.com/users/cgtexmex/received_events","type":"User","site_admin":false},"created_at":"2017-05-11T17:10:30Z","updated_at":"2017-05-11T17:10:30Z","body":"I've looked at about 5 failures of this test.  In all of the cases the first containerVM that was created / started never was transitioned to a stopped state based on the poweredOff event.  The reason it never transitioned was the portLayer never receives the event.  \r\n\r\nI think the reason we are missing the event is because the vSphere `EventHistoryCollector` is created with a [page size of 1](https://github.com/vmware/vic/blob/master/lib/portlayer/event/collector/vsphere/collector.go#L121).  This means that the collector will keep the most recent event on the last page of the event collector, but also means that if a new event comes it will evict the oldest event -- which in this case is the only event.  This causes an issue when you have a spike of events and their is latency on the system -- specifically we miss events.    \r\n\r\nTo remedy this I've increased the page size to 25.  I've run roughly 10 test jobs with this setting and have not seen the issue occur again.  One concern I had with a page size greater than 1 would be the possibility of repeated events -- I have not seen that in the testing with increased page size.  I'll run this thru the CI some more (specifically when there is latency) for additional testing.\r\n\r\n@dougm please take a look at this approach and see if it looks reasonable."}}
{"actor":{"display_login":"cgtexmex"}}