{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48703","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48703/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48703/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48703/events","html_url":"https://github.com/kubernetes/kubernetes/issues/48703","id":241704458,"number":48703,"title":"kubelet keeps attempting to reclaim nodefs after upgrade to 1.7.0","user":{"login":"clenimar","id":410648,"avatar_url":"https://avatars3.githubusercontent.com/u/410648?v=4","gravatar_id":"","url":"https://api.github.com/users/clenimar","html_url":"https://github.com/clenimar","followers_url":"https://api.github.com/users/clenimar/followers","following_url":"https://api.github.com/users/clenimar/following{/other_user}","gists_url":"https://api.github.com/users/clenimar/gists{/gist_id}","starred_url":"https://api.github.com/users/clenimar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clenimar/subscriptions","organizations_url":"https://api.github.com/users/clenimar/orgs","repos_url":"https://api.github.com/users/clenimar/repos","events_url":"https://api.github.com/users/clenimar/events{/privacy}","received_events_url":"https://api.github.com/users/clenimar/received_events","type":"User","site_admin":false},"labels":[{"id":611767149,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/approved-for-milestone","name":"approved-for-milestone","color":"fef2c0","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false},{"id":239914558,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/openstack","name":"sig/openstack","color":"d2b48c","default":false},{"id":125550211,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"dashpole","id":3262098,"avatar_url":"https://avatars1.githubusercontent.com/u/3262098?v=4","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},"assignees":[{"login":"dashpole","id":3262098,"avatar_url":"https://avatars1.githubusercontent.com/u/3262098?v=4","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},{"login":"jingxu97","id":19172805,"avatar_url":"https://avatars3.githubusercontent.com/u/19172805?v=4","gravatar_id":"","url":"https://api.github.com/users/jingxu97","html_url":"https://github.com/jingxu97","followers_url":"https://api.github.com/users/jingxu97/followers","following_url":"https://api.github.com/users/jingxu97/following{/other_user}","gists_url":"https://api.github.com/users/jingxu97/gists{/gist_id}","starred_url":"https://api.github.com/users/jingxu97/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jingxu97/subscriptions","organizations_url":"https://api.github.com/users/jingxu97/orgs","repos_url":"https://api.github.com/users/jingxu97/repos","events_url":"https://api.github.com/users/jingxu97/events{/privacy}","received_events_url":"https://api.github.com/users/jingxu97/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/32","html_url":"https://github.com/kubernetes/kubernetes/milestone/32","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/32/labels","id":2264750,"number":32,"title":"v1.7","description":"","creator":{"login":"pwittrock","id":7882754,"avatar_url":"https://avatars0.githubusercontent.com/u/7882754?v=4","gravatar_id":"","url":"https://api.github.com/users/pwittrock","html_url":"https://github.com/pwittrock","followers_url":"https://api.github.com/users/pwittrock/followers","following_url":"https://api.github.com/users/pwittrock/following{/other_user}","gists_url":"https://api.github.com/users/pwittrock/gists{/gist_id}","starred_url":"https://api.github.com/users/pwittrock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwittrock/subscriptions","organizations_url":"https://api.github.com/users/pwittrock/orgs","repos_url":"https://api.github.com/users/pwittrock/repos","events_url":"https://api.github.com/users/pwittrock/events{/privacy}","received_events_url":"https://api.github.com/users/pwittrock/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":1153,"state":"open","created_at":"2017-01-19T17:41:27Z","updated_at":"2017-07-25T05:12:25Z","due_on":"2017-06-28T07:00:00Z","closed_at":null},"comments":18,"created_at":"2017-07-10T13:02:39Z","updated_at":"2017-07-25T06:50:27Z","closed_at":null,"body":"I just upgraded a cluster from 1.5.2 to 1.7.0, and it's working fine, although `kubelet` keeps trying to reclaim nodefs. Not sure why. Any thoughts?\r\n\r\nMy cluster runs in an OpenStack environment, it's provisioned by Openstack Magnum. I upgraded it by replacing all the binaries and services to 1.7.0, and it seems to be working fine.\r\n\r\nDescribing the master node (pay attention to the events below - 1200+ occurrencies of the warning):\r\n\r\n```shell\r\n$ k describe no kube-master\r\nName:\t\t\tkube-master\r\nRole:\t\t\t\r\nLabels:\t\t\tbeta.kubernetes.io/arch=amd64\r\n\t\t\tbeta.kubernetes.io/os=linux\r\n\t\t\tkubernetes.io/hostname=kube-master\r\nTaints:\t\t\t<none>\r\nCreationTimestamp:\tFri, 07 Jul 2017 08:06:53 +0000\r\nPhase:\t\t\t\r\nConditions:\r\n  Type\t\t\tStatus\tLastHeartbeatTime\t\t\tLastTransitionTime\t\t\tReason\t\t\t\tMessage\r\n  ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r\n  OutOfDisk \t\tFalse \tMon, 10 Jul 2017 12:48:23 +0000 \tMon, 10 Jul 2017 07:32:04 +0000 \tKubeletHasSufficientDisk \tkubelet has sufficient disk space available\r\n  MemoryPressure \tFalse \tMon, 10 Jul 2017 12:48:23 +0000 \tMon, 10 Jul 2017 07:32:04 +0000 \tKubeletHasSufficientMemory \tkubelet has sufficient memory available\r\n  DiskPressure \t\tTrue \tMon, 10 Jul 2017 12:48:23 +0000 \tMon, 10 Jul 2017 07:32:14 +0000 \tKubeletHasDiskPressure \t\tkubelet has disk pressure\r\n  Ready \t\tTrue \tMon, 10 Jul 2017 12:48:23 +0000 \tMon, 10 Jul 2017 07:32:14 +0000 \tKubeletReady \t\t\tkubelet is posting ready status\r\nAddresses:\t\tkube-master,kube-master\r\nCapacity:\r\n cpu:\t\t2\r\n memory:\t3696784Ki\r\n pods:\t\t110\r\nAllocatable:\r\n cpu:\t\t2\r\n memory:\t3594384Ki\r\n pods:\t\t110\r\nSystem Info:\r\n Machine ID:\t\t\t7263b14b412d402a8127bc5f461fc1ec\r\n System UUID:\t\t\t7263B14B-412D-402A-8127-BC5F461FC1EC\r\n Boot ID:\t\t\t08174161-a67c-4a43-abee-1751dc7f203a\r\n Kernel Version:\t\t4.9.6-200.fc25.x86_64\r\n OS Image:\t\t\tFedora 25 (Atomic Host)\r\n Operating System:\t\tlinux\r\n Architecture:\t\t\tamd64\r\n Container Runtime Version:\tdocker://1.12.6\r\n Kubelet Version:\t\tv1.7.0\r\n Kube-Proxy Version:\t\tv1.7.0\r\nExternalID:\t\t\tkube-master\r\nNon-terminated Pods:\t\t(0 in total)\r\n  Namespace\t\t\tName\t\tCPU Requests\tCPU Limits\tMemory Requests\tMemory Limits\r\n  ---------\t\t\t----\t\t------------\t----------\t---------------\t-------------\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.\r\n  CPU Requests\tCPU Limits\tMemory Requests\tMemory Limits\r\n  ------------\t----------\t---------------\t-------------\r\n  0 (0%)\t0 (0%)\t\t0 (0%)\t\t0 (0%)\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\t\tSubObjectPath\tType\t\tReason\t\t\tMessage\r\n  ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r\n  5h\t\t20s\t\t1271\t{kubelet kube-master}\t\t\tWarning\t\tEvictionThresholdMet\tAttempting to reclaim nodefs\r\n```\r\nLogs from kubelet:\r\n\r\n```shell\r\njul 10 12:09:11 master.cern.ch kubelet[21724]: W0710 12:09:11.316491   21724 eviction_manager.go:331] eviction manager: attempting to reclaim nodefs\r\njul 10 12:09:11 master.cern.ch kubelet[21724]: I0710 12:09:11.316826   21724 eviction_manager.go:345] eviction manager: must evict pod(s) to reclaim nodefs\r\njul 10 12:09:11 master.cern.ch kubelet[21724]: E0710 12:09:11.317073   21724 eviction_manager.go:356] eviction manager: eviction thresholds have been met, but no pods are active to evict\r\njul 10 12:09:21 master.cern.ch kubelet[21724]: W0710 12:09:21.388922   21724 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\n```\r\ncc @kubernetes/sig-scheduling-misc @kubernetes/sig-openstack-misc"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/317646328","html_url":"https://github.com/kubernetes/kubernetes/issues/48703#issuecomment-317646328","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48703","id":317646328,"user":{"login":"lukmanulhakimd","id":16384543,"avatar_url":"https://avatars0.githubusercontent.com/u/16384543?v=4","gravatar_id":"","url":"https://api.github.com/users/lukmanulhakimd","html_url":"https://github.com/lukmanulhakimd","followers_url":"https://api.github.com/users/lukmanulhakimd/followers","following_url":"https://api.github.com/users/lukmanulhakimd/following{/other_user}","gists_url":"https://api.github.com/users/lukmanulhakimd/gists{/gist_id}","starred_url":"https://api.github.com/users/lukmanulhakimd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukmanulhakimd/subscriptions","organizations_url":"https://api.github.com/users/lukmanulhakimd/orgs","repos_url":"https://api.github.com/users/lukmanulhakimd/repos","events_url":"https://api.github.com/users/lukmanulhakimd/events{/privacy}","received_events_url":"https://api.github.com/users/lukmanulhakimd/received_events","type":"User","site_admin":false},"created_at":"2017-07-25T06:50:27Z","updated_at":"2017-07-25T06:50:27Z","body":"I'm facing the same problem here.\r\nKubernetes 1.7.2\r\nDocker 1.11.2\r\n\r\n```\r\nroot@vm-r3u2b-kube17-worker0:/home/syseng# journalctl -fu kubelet\r\n-- Logs begin at Thu 2017-07-20 13:38:21 WIB. --\r\nJul 25 13:48:21 vm-r3u2b-kube17-worker0 kubelet[30234]: I0725 13:48:21.553475   30234 qos_container_manager_linux.go:286] [ContainerManager]: Updated QoS cgroup configuration\r\nJul 25 13:48:24 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:48:24.254064   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:48:34 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:48:34.285573   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:48:44 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:48:44.318364   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:48:54 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:48:54.352991   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:49:04 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:49:04.383762   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:49:14 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:49:14.419990   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:49:21 vm-r3u2b-kube17-worker0 kubelet[30234]: I0725 13:49:21.553766   30234 qos_container_manager_linux.go:286] [ContainerManager]: Updated QoS cgroup configuration\r\nJul 25 13:49:24 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:49:24.454277   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:49:34 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:49:34.487554   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\nJul 25 13:49:44 vm-r3u2b-kube17-worker0 kubelet[30234]: W0725 13:49:44.523247   30234 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available\r\n```"}}
{"actor":{"display_login":"lukmanulhakimd"}}