{"action":"created","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/3413","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/3413/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/3413/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/3413/events","html_url":"https://github.com/flynn/flynn/issues/3413","id":178494816,"number":3413,"title":"Postgres recovery issue, possibly caused by lack of garbage collection on jobs listing ","user":{"login":"temujin9","id":627414,"avatar_url":"https://avatars2.githubusercontent.com/u/627414?v=3","gravatar_id":"","url":"https://api.github.com/users/temujin9","html_url":"https://github.com/temujin9","followers_url":"https://api.github.com/users/temujin9/followers","following_url":"https://api.github.com/users/temujin9/following{/other_user}","gists_url":"https://api.github.com/users/temujin9/gists{/gist_id}","starred_url":"https://api.github.com/users/temujin9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/temujin9/subscriptions","organizations_url":"https://api.github.com/users/temujin9/orgs","repos_url":"https://api.github.com/users/temujin9/repos","events_url":"https://api.github.com/users/temujin9/events{/privacy}","received_events_url":"https://api.github.com/users/temujin9/received_events","type":"User","site_admin":false},"labels":[{"id":120399065,"url":"https://api.github.com/repos/flynn/flynn/labels/component/host","name":"component/host","color":"fef2c0","default":false},{"id":267414489,"url":"https://api.github.com/repos/flynn/flynn/labels/component/scheduler","name":"component/scheduler","color":"fef2c0","default":false},{"id":47091203,"url":"https://api.github.com/repos/flynn/flynn/labels/kind/bug","name":"kind/bug","color":"d93f0b","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-09-22T00:34:20Z","updated_at":"2017-03-02T00:46:07Z","closed_at":"2017-03-02T00:46:07Z","body":"A test cluster produced a 12 hour long Postgres degradation alert, via monitoring of the status app. This appeared to coincide with an earlier ramp up to 100% CPU usage (but low load) on one node, which was traced to the flynn-host daemon and the flynn-scheduler container. That scheduler job was listed as down for a long time (via `flynn ps -a`) with no attempt to replace it visible as it normally would be on that list. \n\nRunning `flynn-host collect-debug-info` on that node resulted in a huge load spike on the wedged node, an OOM partway through the collection, a subsequent drop in CPU usage, and the a clearing of the pg status alert. This all coincided relatively closely, making me think the symptoms are related despite broadly different start times.\n\nAlso probably salient is that there were roughly 80K of terminated jobs (via `flynn-host ps -a`), from an old flapping job which had (a while before this) been turned off.\n\nhttps://botbot.me/freenode/flynn/2016-09-21/?msg=73387396&page=1 has more of the conversation around debugging it, including recovered logs and the process list.\n"},"comment":{"url":"https://api.github.com/repos/flynn/flynn/issues/comments/283520072","html_url":"https://github.com/flynn/flynn/issues/3413#issuecomment-283520072","issue_url":"https://api.github.com/repos/flynn/flynn/issues/3413","id":283520072,"user":{"login":"lmars","id":488515,"avatar_url":"https://avatars1.githubusercontent.com/u/488515?v=3","gravatar_id":"","url":"https://api.github.com/users/lmars","html_url":"https://github.com/lmars","followers_url":"https://api.github.com/users/lmars/followers","following_url":"https://api.github.com/users/lmars/following{/other_user}","gists_url":"https://api.github.com/users/lmars/gists{/gist_id}","starred_url":"https://api.github.com/users/lmars/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmars/subscriptions","organizations_url":"https://api.github.com/users/lmars/orgs","repos_url":"https://api.github.com/users/lmars/repos","events_url":"https://api.github.com/users/lmars/events{/privacy}","received_events_url":"https://api.github.com/users/lmars/received_events","type":"User","site_admin":false},"created_at":"2017-03-02T00:46:07Z","updated_at":"2017-03-02T00:46:07Z","body":"Fixed in #3767."}}
{"actor":{"display_login":"lmars"}}