{"action":"created","issue":{"url":"https://api.github.com/repos/ganeti/ganeti/issues/52","repository_url":"https://api.github.com/repos/ganeti/ganeti","labels_url":"https://api.github.com/repos/ganeti/ganeti/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/ganeti/ganeti/issues/52/comments","events_url":"https://api.github.com/repos/ganeti/ganeti/issues/52/events","html_url":"https://github.com/ganeti/ganeti/issues/52","id":235871093,"number":52,"title":"Stale instance files after user poweroff and gnt-instance shutdown ","user":{"login":"yiannist","id":547024,"avatar_url":"https://avatars2.githubusercontent.com/u/547024?v=3","gravatar_id":"","url":"https://api.github.com/users/yiannist","html_url":"https://github.com/yiannist","followers_url":"https://api.github.com/users/yiannist/followers","following_url":"https://api.github.com/users/yiannist/following{/other_user}","gists_url":"https://api.github.com/users/yiannist/gists{/gist_id}","starred_url":"https://api.github.com/users/yiannist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yiannist/subscriptions","organizations_url":"https://api.github.com/users/yiannist/orgs","repos_url":"https://api.github.com/users/yiannist/repos","events_url":"https://api.github.com/users/yiannist/events{/privacy}","received_events_url":"https://api.github.com/users/yiannist/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-06-14T13:00:48Z","updated_at":"2017-06-14T14:11:28Z","closed_at":null,"body":"We noticed a strange corner case that leads to stale files in `/var/run/ganeti/`. The scenario is:\r\n- User issues a `poweroff` from inside the VM\r\n- The admin or a RAPI client does a `gnt-instance shutdown` (or even worse a `gnt-instance remove` which, in turn, issues an instance shutdown) **before** the watcher notices the `poweroff` \r\n- The instance turns to `ADMIN_down` state (or gets removed) but the cleanup of the runtime files is never performed:\r\n```\r\n➜ ok12-05.okeanos /var/run/ganeti  # find | grep snf-40846\r\n./kvm-hypervisor/nic/snf-40846\r\n./kvm-hypervisor/nic/snf-40846/ae009d01-99b1-4486-9cdd-1654a2aca05e\r\n./kvm-hypervisor/nic/snf-40846/0\r\n./kvm-hypervisor/conf/snf-40846.runtime\r\n./kvm-hypervisor/ctrl/snf-40846.qmp\r\n./kvm-hypervisor/ctrl/snf-40846.monitor\r\n./kvm-hypervisor/uid/snf-40846\r\n./kvm-hypervisor/pid/snf-40846\r\n./instance-reason/snf-40846\r\n./instance-disks/snf-40846:0\r\n```\r\n\r\nWe noticed the following in our `node-daemon.log`:\r\n```\r\n2017-05-31 18:33:12,043: ganeti-noded pid=26133 INFO Instance snf-136122 not running, doing nothing\r\n```\r\n\r\nDigging into this, I 'd say that the [`InstanceShutdown`](https://github.com/ganeti/ganeti/blob/master/lib/backend.py#L2973) job aborts execution too early (because the process is dead) and never executes the [cleanup](https://github.com/ganeti/ganeti/blob/master/lib/backend.py#L3020). However, the instance's status is already changed to [`ADMIN_down`](https://github.com/ganeti/ganeti/blob/master/lib/cmdlib/instance_operation.py#L296). Note here that the instance has `admin_state = \"up\"` after the `poweroff` and thus is considered `INSTANCE_ONLINE`:\r\n```\r\ndio.test.gnt.grnet.gr:~# gnt-instance list -oname,status,admin_state | grep migrate\r\nmigrate.me             running    up\r\n# Connect to instance's console and issue a poweroff. \r\ndio.test.gnt.grnet.gr:~# gnt-instance list -oname,status,admin_state | grep migrate\r\nmigrate.me             ERROR_down up\r\n```\r\n\r\nI would say that the fix should be straightforward: always try to cleanup. That is, remove the [`return`](https://github.com/ganeti/ganeti/blob/master/lib/backend.py#L2975) and put the shutdown code in an else statement. Thus, the cleanup snippet will be executed anyway. The rationale is that the person/administrator that triggered the `gnt-instance shutdown` command shouldn't be left with stale runtime files even if the KVM process is already died. We confirmed this in 2.10, 2.15 and 2.16 (KVM) clusters with `user_shutdown` option disabled. If you agree, I could submit a PR for this against stable-2.16.\r\n\r\nThe fun fact is that enabling `user_shutdown` works because `_GetInstanceInfo` takes into consideration the [`USER_down` state ](https://github.com/ganeti/ganeti/blob/master/lib/hypervisor/hv_kvm/__init__.py#L1025) and doesn't exit the `InstanceShutdown` function early."},"comment":{"url":"https://api.github.com/repos/ganeti/ganeti/issues/comments/308443567","html_url":"https://github.com/ganeti/ganeti/issues/52#issuecomment-308443567","issue_url":"https://api.github.com/repos/ganeti/ganeti/issues/52","id":308443567,"user":{"login":"yiannist","id":547024,"avatar_url":"https://avatars2.githubusercontent.com/u/547024?v=3","gravatar_id":"","url":"https://api.github.com/users/yiannist","html_url":"https://github.com/yiannist","followers_url":"https://api.github.com/users/yiannist/followers","following_url":"https://api.github.com/users/yiannist/following{/other_user}","gists_url":"https://api.github.com/users/yiannist/gists{/gist_id}","starred_url":"https://api.github.com/users/yiannist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yiannist/subscriptions","organizations_url":"https://api.github.com/users/yiannist/orgs","repos_url":"https://api.github.com/users/yiannist/repos","events_url":"https://api.github.com/users/yiannist/events{/privacy}","received_events_url":"https://api.github.com/users/yiannist/received_events","type":"User","site_admin":false},"created_at":"2017-06-14T14:11:28Z","updated_at":"2017-06-14T14:11:28Z","body":"The fix that I suggest is in #53."}}
{"actor":{"display_login":"yiannist"}}