{"action":"created","issue":{"url":"https://api.github.com/repos/lxc/lxd/issues/3412","repository_url":"https://api.github.com/repos/lxc/lxd","labels_url":"https://api.github.com/repos/lxc/lxd/issues/3412/labels{/name}","comments_url":"https://api.github.com/repos/lxc/lxd/issues/3412/comments","events_url":"https://api.github.com/repos/lxc/lxd/issues/3412/events","html_url":"https://github.com/lxc/lxd/issues/3412","id":235975999,"number":3412,"title":"Containers sporadically not starting after host reboot","user":{"login":"wwwtyro","id":929443,"avatar_url":"https://avatars2.githubusercontent.com/u/929443?v=3","gravatar_id":"","url":"https://api.github.com/users/wwwtyro","html_url":"https://github.com/wwwtyro","followers_url":"https://api.github.com/users/wwwtyro/followers","following_url":"https://api.github.com/users/wwwtyro/following{/other_user}","gists_url":"https://api.github.com/users/wwwtyro/gists{/gist_id}","starred_url":"https://api.github.com/users/wwwtyro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wwwtyro/subscriptions","organizations_url":"https://api.github.com/users/wwwtyro/orgs","repos_url":"https://api.github.com/users/wwwtyro/repos","events_url":"https://api.github.com/users/wwwtyro/events{/privacy}","received_events_url":"https://api.github.com/users/wwwtyro/received_events","type":"User","site_admin":false},"labels":[{"id":144672267,"url":"https://api.github.com/repos/lxc/lxd/labels/Bug","name":"Bug","color":"eb6420","default":false}],"state":"open","locked":false,"assignee":{"login":"brauner","id":7353573,"avatar_url":"https://avatars3.githubusercontent.com/u/7353573?v=3","gravatar_id":"","url":"https://api.github.com/users/brauner","html_url":"https://github.com/brauner","followers_url":"https://api.github.com/users/brauner/followers","following_url":"https://api.github.com/users/brauner/following{/other_user}","gists_url":"https://api.github.com/users/brauner/gists{/gist_id}","starred_url":"https://api.github.com/users/brauner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brauner/subscriptions","organizations_url":"https://api.github.com/users/brauner/orgs","repos_url":"https://api.github.com/users/brauner/repos","events_url":"https://api.github.com/users/brauner/events{/privacy}","received_events_url":"https://api.github.com/users/brauner/received_events","type":"User","site_admin":false},"assignees":[{"login":"brauner","id":7353573,"avatar_url":"https://avatars3.githubusercontent.com/u/7353573?v=3","gravatar_id":"","url":"https://api.github.com/users/brauner","html_url":"https://github.com/brauner","followers_url":"https://api.github.com/users/brauner/followers","following_url":"https://api.github.com/users/brauner/following{/other_user}","gists_url":"https://api.github.com/users/brauner/gists{/gist_id}","starred_url":"https://api.github.com/users/brauner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brauner/subscriptions","organizations_url":"https://api.github.com/users/brauner/orgs","repos_url":"https://api.github.com/users/brauner/repos","events_url":"https://api.github.com/users/brauner/events{/privacy}","received_events_url":"https://api.github.com/users/brauner/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/lxc/lxd/milestones/60","html_url":"https://github.com/lxc/lxd/milestone/60","labels_url":"https://api.github.com/repos/lxc/lxd/milestones/60/labels","id":2475262,"number":60,"title":"lxd-2.15","description":"","creator":{"login":"stgraber","id":2230599,"avatar_url":"https://avatars3.githubusercontent.com/u/2230599?v=3","gravatar_id":"","url":"https://api.github.com/users/stgraber","html_url":"https://github.com/stgraber","followers_url":"https://api.github.com/users/stgraber/followers","following_url":"https://api.github.com/users/stgraber/following{/other_user}","gists_url":"https://api.github.com/users/stgraber/gists{/gist_id}","starred_url":"https://api.github.com/users/stgraber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stgraber/subscriptions","organizations_url":"https://api.github.com/users/stgraber/orgs","repos_url":"https://api.github.com/users/stgraber/repos","events_url":"https://api.github.com/users/stgraber/events{/privacy}","received_events_url":"https://api.github.com/users/stgraber/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":4,"state":"open","created_at":"2017-04-24T01:09:40Z","updated_at":"2017-06-14T19:43:26Z","due_on":"2017-06-27T07:00:00Z","closed_at":null},"comments":1,"created_at":"2017-06-14T18:52:34Z","updated_at":"2017-06-14T20:43:40Z","closed_at":null,"body":" * Distribution: ubuntu\r\n * Distribution version: xenial\r\n * [lxc info](https://gist.github.com/wwwtyro/15c52c8da2f831671487f1a131de0bba)\r\n\r\nWe have 42 containers running on a single VM with a ZFS backend. After reboot, some of the\r\ncontainers fail to start. We see errors in `/var/log/lxd/<container_id>/lxc.log`:\r\n\r\n```\r\nroot@ip-172-31-15-253:/var/log/lxd/juju-690256-1# cat lxc.log\r\n            lxc 20170614160510.317 ERROR    lxc_conf - conf.c:mount_rootfs:798 - No such file or directory - Failed to get real path for \"/var/lib/lxd/containers/juju-690256-1/rootfs\".\r\n            lxc 20170614160510.317 ERROR    lxc_conf - conf.c:setup_rootfs:1220 - Failed to mount rootfs \"/var/lib/lxd/containers/juju-690256-1/rootfs\" onto \"/usr/lib/x86_64-linux-gnu/lxc\" with options \"(null)\".\r\n            lxc 20170614160510.317 ERROR    lxc_conf - conf.c:do_rootfs_setup:3899 - failed to setup rootfs for 'juju-690256-1'\r\n            lxc 20170614160510.317 ERROR    lxc_conf - conf.c:lxc_setup:3981 - Error setting up rootfs mount after spawn\r\n            lxc 20170614160510.317 ERROR    lxc_start - start.c:do_start:811 - Failed to setup container \"juju-690256-1\".\r\n            lxc 20170614160510.317 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)\r\n            lxc 20170614160510.464 ERROR    lxc_start - start.c:__lxc_start:1358 - Failed to spawn container \"juju-690256-1\".\r\n            lxc 20170614160511.187 ERROR    lxc_conf - conf.c:run_buffer:416 - Script exited with status 1.\r\n            lxc 20170614160511.187 ERROR    lxc_start - start.c:lxc_fini:546 - Failed to run lxc.hook.post-stop for container \"juju-690256-1\".\r\n            lxc 20170614160511.187 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:177 - Command get_cgroup failed to receive response: Connection reset by peer.\r\n            lxc 20170614160511.187 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:177 - Command get_cgroup failed to receive response: Connection reset by peer.\r\n```\r\n\r\nAnd related logs in `/var/log/lxd/lxd.log`:\r\n\r\n```\r\nroot@ip-172-31-15-253:/var/log/lxd# cat lxd.log | grep 690256-1\r\nephemeral=false lvl=info msg=\"Creating container\" name=juju-690256-1 t=2017-06-14T13:55:43+0000\r\nephemeral=false lvl=info msg=\"Created container\" name=juju-690256-1 t=2017-06-14T13:55:43+0000\r\naction=start created=2017-06-14T13:55:43+0000 ephemeral=false lvl=info msg=\"Starting container\" name=juju-690256-1 stateful=false t=2017-06-14T13:55:43+0000 used=1970-01-01T00:00:00+0000\r\naction=start created=2017-06-14T13:55:43+0000 ephemeral=false lvl=info msg=\"Started container\" name=juju-690256-1 stateful=false t=2017-06-14T13:55:43+0000 used=1970-01-01T00:00:00+0000\r\naction=shutdown created=2017-06-14T13:55:43+0000 ephemeral=false lvl=info msg=\"Shutting down container\" name=juju-690256-1 t=2017-06-14T14:56:12+0000 timeout=30s used=2017-06-14T13:55:43+0000\r\naction=shutdown created=2017-06-14T13:55:43+0000 ephemeral=false lvl=info msg=\"Shut down container\" name=juju-690256-1 t=2017-06-14T14:56:34+0000 timeout=30s used=2017-06-14T13:55:43+0000\r\nlvl=warn msg=\"Unable to update backup.yaml at this time.\" name=juju-690256-1 t=2017-06-14T14:56:34+0000\r\nlvl=warn msg=\"ZFS returned EBUSY while \\\"/var/lib/lxd/storage-pools/default/containers/juju-690256-1\\\" is actually not a mountpoint.\" t=2017-06-14T16:05:10+0000\r\naction=start created=2017-06-14T13:55:43+0000 ephemeral=false lvl=info msg=\"Starting container\" name=juju-690256-1 stateful=false t=2017-06-14T16:05:10+0000 used=2017-06-14T13:55:43+0000\r\nlvl=warn msg=\"ZFS returned EBUSY while \\\"/var/lib/lxd/storage-pools/default/containers/juju-690256-1\\\" is actually not a mountpoint.\" t=2017-06-14T16:05:10+0000\r\ncontainer=juju-690256-1 err=\"Container is already running a start operation\" lvl=eror msg=\"stop hook failed\" t=2017-06-14T16:05:11+0000\r\naction=start created=2017-06-14T13:55:43+0000 ephemeral=false lvl=eror msg=\"Failed starting container\" name=juju-690256-1 stateful=false t=2017-06-14T16:05:16+0000 used=2017-06-14T13:55:43+0000\r\n```\r\n\r\nThis is a recent development, so it might be related to #3228 and #3229, which came out in 2.14.\r\n\r\nPinging @brauner in case you have any insight here - thanks!"},"comment":{"url":"https://api.github.com/repos/lxc/lxd/issues/comments/308552384","html_url":"https://github.com/lxc/lxd/issues/3412#issuecomment-308552384","issue_url":"https://api.github.com/repos/lxc/lxd/issues/3412","id":308552384,"user":{"login":"brauner","id":7353573,"avatar_url":"https://avatars3.githubusercontent.com/u/7353573?v=3","gravatar_id":"","url":"https://api.github.com/users/brauner","html_url":"https://github.com/brauner","followers_url":"https://api.github.com/users/brauner/followers","following_url":"https://api.github.com/users/brauner/following{/other_user}","gists_url":"https://api.github.com/users/brauner/gists{/gist_id}","starred_url":"https://api.github.com/users/brauner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brauner/subscriptions","organizations_url":"https://api.github.com/users/brauner/orgs","repos_url":"https://api.github.com/users/brauner/repos","events_url":"https://api.github.com/users/brauner/events{/privacy}","received_events_url":"https://api.github.com/users/brauner/received_events","type":"User","site_admin":false},"created_at":"2017-06-14T20:43:40Z","updated_at":"2017-06-14T20:43:40Z","body":"I'm not sure. I guess the fact that we mount and umount as needed to not pollute the host namespace just raises the probability that we run into the zfs bug."}}
{"actor":{"display_login":"brauner"}}