{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48716","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48716/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48716/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48716/events","html_url":"https://github.com/kubernetes/kubernetes/issues/48716","id":241783022,"number":48716,"title":"Kubelet is not cleaning up pod volumes, leaving statefulset in pending state","user":{"login":"xrl","id":693059,"avatar_url":"https://avatars1.githubusercontent.com/u/693059?v=3","gravatar_id":"","url":"https://api.github.com/users/xrl","html_url":"https://github.com/xrl","followers_url":"https://api.github.com/users/xrl/followers","following_url":"https://api.github.com/users/xrl/following{/other_user}","gists_url":"https://api.github.com/users/xrl/gists{/gist_id}","starred_url":"https://api.github.com/users/xrl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xrl/subscriptions","organizations_url":"https://api.github.com/users/xrl/orgs","repos_url":"https://api.github.com/users/xrl/repos","events_url":"https://api.github.com/users/xrl/events{/privacy}","received_events_url":"https://api.github.com/users/xrl/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":169428334,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2017-07-10T17:10:22Z","updated_at":"2017-07-12T09:48:41Z","closed_at":null,"body":"**Bug report**:\r\n\r\n/kind bug\r\n/kind sig-openstack\r\n/kind sig-storage\r\n\r\n**What happened**:\r\nI created a statefulset for job runners, each with some mounted volumes for scratch space, and after a few recreates the pods were stuck in _pending_. Reading over kubelet logs it looks like the pods are not marked as free because volumes are not disappearing.\r\n\r\n**What you expected to happen**:\r\nStatefulset up and down should have worked clean, removing whatever was on the kube node, and making the pod eligible to move.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nCreate a statefulset, cycle it a few times. Maybe use Openstack Volumes too.\r\n\r\n**Anything else we need to know?**:\r\n\r\nSample kubelet errors:\r\n\r\n```\r\nE0710 17:05:18.472909    5658 kubelet_volumes.go:110] Orphaned pod \"352193f6-5871-11e7-a01f-fa163ee1f782\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:18.473150    5658 kubelet_volumes.go:110] Orphaned pod \"6ff3b92c-587b-11e7-8b7e-fa163e7835c1\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:18.473421    5658 kubelet_volumes.go:110] Orphaned pod \"82a19d7f-3505-11e7-bb41-fa163e11ae1a\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:20.476006    5658 kubelet_volumes.go:110] Orphaned pod \"352193f6-5871-11e7-a01f-fa163ee1f782\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:20.476240    5658 kubelet_volumes.go:110] Orphaned pod \"6ff3b92c-587b-11e7-8b7e-fa163e7835c1\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:20.476448    5658 kubelet_volumes.go:110] Orphaned pod \"82a19d7f-3505-11e7-bb41-fa163e11ae1a\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:22.483983    5658 kubelet_volumes.go:110] Orphaned pod \"352193f6-5871-11e7-a01f-fa163ee1f782\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:22.484162    5658 kubelet_volumes.go:110] Orphaned pod \"6ff3b92c-587b-11e7-8b7e-fa163e7835c1\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:22.484264    5658 kubelet_volumes.go:110] Orphaned pod \"82a19d7f-3505-11e7-bb41-fa163e11ae1a\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:24.472593    5658 kubelet_volumes.go:110] Orphaned pod \"352193f6-5871-11e7-a01f-fa163ee1f782\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:24.472978    5658 kubelet_volumes.go:110] Orphaned pod \"6ff3b92c-587b-11e7-8b7e-fa163e7835c1\" found, but error <nil> occured during reading volume dir from disk\r\nE0710 17:05:24.473200    5658 kubelet_volumes.go:110] Orphaned pod \"82a19d7f-3505-11e7-bb41-fa163e11ae1a\" found, but error <nil> occured during reading volume dir from disk\r\n```\r\n\r\nHere's what's inside one of the kubelet directories from that error log:\r\n```\r\nkube-node-03 core # cd /var/lib/kubelet/pods/352193f6-5871-11e7-a01f-fa163ee1f782\r\nkube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782 # ls\r\ncontainers  etc-hosts  plugins  volumes\r\nkube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782 # ls *\r\netc-hosts\r\n\r\ncontainers:\r\njob-runner\r\n\r\nplugins:\r\nkubernetes.io~empty-dir\r\n\r\nvolumes:\r\nkubernetes.io~cinder  kubernetes.io~secret\r\n```\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n\r\n```\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"6\", GitVersion:\"v1.6.2\", GitCommit:\"477efc3cbe6a7effca06bd1452fa356e2201e1ee\", GitTreeState:\"clean\", BuildDate:\"2017-04-19T22:51:55Z\", GoVersion:\"go1.8.1\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"5+\", GitVersion:\"1.5-cinder_v2_api-os-verbose-b7-2-g41406bdbbc3c2e-dirty\", GitCommit:\"41406bdbbc3c2e9fd83c7b80825f39ec4008b699\", GitTreeState:\"dirty\", BuildDate:\"2017-07-08T00:04:34Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\nI run a custom build with some openstack API fixes.\r\n\r\n- Cloud provider or hardware configuration**:\r\n\r\n * OpenStack\r\n * Kubespray distribution\r\n\r\n- OS (e.g. from /etc/os-release):\r\n\r\n```\r\nkube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782 # cat /etc/os-release\r\nNAME=\"Container Linux by CoreOS\"\r\nID=coreos\r\nVERSION=1298.7.0\r\nVERSION_ID=1298.7.0\r\nBUILD_ID=2017-03-31-0215\r\nPRETTY_NAME=\"Container Linux by CoreOS 1298.7.0 (Ladybug)\"\r\nANSI_COLOR=\"38;5;75\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://github.com/coreos/bugs/issues\"\r\n```\r\n\r\n- Kernel (e.g. `uname -a`):\r\n\r\n```\r\nkube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782 # uname -a\r\nLinux kube-node-03.openstacklocal 4.9.16-coreos-r1 #1 SMP Fri Mar 31 02:07:42 UTC 2017 x86_64 Intel Xeon E312xx (Sandy Bridge) GenuineIntel GNU/Linux\r\n```\r\n\r\n- Install tools:\r\n- Others:\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/314712691","html_url":"https://github.com/kubernetes/kubernetes/issues/48716#issuecomment-314712691","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48716","id":314712691,"user":{"login":"yastij","id":7813699,"avatar_url":"https://avatars0.githubusercontent.com/u/7813699?v=3","gravatar_id":"","url":"https://api.github.com/users/yastij","html_url":"https://github.com/yastij","followers_url":"https://api.github.com/users/yastij/followers","following_url":"https://api.github.com/users/yastij/following{/other_user}","gists_url":"https://api.github.com/users/yastij/gists{/gist_id}","starred_url":"https://api.github.com/users/yastij/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yastij/subscriptions","organizations_url":"https://api.github.com/users/yastij/orgs","repos_url":"https://api.github.com/users/yastij/repos","events_url":"https://api.github.com/users/yastij/events{/privacy}","received_events_url":"https://api.github.com/users/yastij/received_events","type":"User","site_admin":false},"created_at":"2017-07-12T09:48:41Z","updated_at":"2017-07-12T09:48:41Z","body":"@xrl -my guess is that this is the correct behaviour:\r\n\r\nwhen trying to clean we use: \r\n```go\r\n// cleanupOrphanedPodDirs removes the volumes of pods that should not be\r\n// running and that have no containers running.  Note that we roll up logs here since it runs in the main loop.\r\nfunc (kl *Kubelet) cleanupOrphanedPodDirs(pods []*v1.Pod, runningPods []*kubecontainer.Pod) error {\r\n```\r\nbut since you are running a StatefulSet volumes are not removed, hence this condition is defaulting to true : https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/kubelet_volumes.go#L109 with no error, that's why you are seeing the nil. \r\n\r\nthis was fixed in 1.6 i think in order to distinguish errors by introducing this: \r\n\r\n```go\r\nif len(volumePaths) > 0 {\r\n          orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf(\"Orphaned pod %q found, but volume paths are still present on disk.\", uid))\r\n\t   continue\r\n}\r\n```\r\n\r\nSince i'm not sure, ping @janetkuo or @mikedanese  for validation."}}
{"actor":{"display_login":"yastij"}}