{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31123","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31123/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31123/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31123/events","html_url":"https://github.com/kubernetes/kubernetes/issues/31123","id":172472665,"number":31123,"title":"sleeping daemonset pods on master node after restart","user":{"login":"untoreh","id":2363212,"avatar_url":"https://avatars3.githubusercontent.com/u/2363212?v=3","gravatar_id":"","url":"https://api.github.com/users/untoreh","html_url":"https://github.com/untoreh","followers_url":"https://api.github.com/users/untoreh/followers","following_url":"https://api.github.com/users/untoreh/following{/other_user}","gists_url":"https://api.github.com/users/untoreh/gists{/gist_id}","starred_url":"https://api.github.com/users/untoreh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/untoreh/subscriptions","organizations_url":"https://api.github.com/users/untoreh/orgs","repos_url":"https://api.github.com/users/untoreh/repos","events_url":"https://api.github.com/users/untoreh/events{/privacy}","received_events_url":"https://api.github.com/users/untoreh/received_events","type":"User","site_admin":false},"labels":[{"id":136601472,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/controller-manager","name":"area/controller-manager","color":"0052cc","default":false},{"id":244613097,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane%20(deprecated%20-%20do%20not%20use)","name":"team/control-plane (deprecated - do not use)","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-22T14:50:42Z","updated_at":"2017-04-03T09:52:12Z","closed_at":null,"body":"``` bash\nClient Version: version.Info{Major:\"1\", Minor:\"3\", GitVersion:\"v1.3.5+coreos.0\", GitCommit:\"d7a04b1c6044647f5919fadf3cecb9ee70c10fc5\", GitTreeState:\"clean\", BuildDate:\"2016-08-15T21:01:42Z\", GoVersion:\"go1.6.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\nServer Version: version.Info{Major:\"1\", Minor:\"3\", GitVersion:\"v1.3.5+coreos.0\", GitCommit:\"d7a04b1c6044647f5919fadf3cecb9ee70c10fc5\", GitTreeState:\"clean\", BuildDate:\"2016-08-15T21:01:42Z\", GoVersion:\"go1.6.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\nrkt Version: 1.7.0\nappc Version: 0.8.4\nGo Version: go1.6.1\nGo OS/Arch: linux/amd64\nFeatures: -TPM\nNAME=CoreOS\nID=coreos\nVERSION=1068.9.0\nVERSION_ID=1068.9.0\nBUILD_ID=2016-08-05-0455\nPRETTY_NAME=\"CoreOS 1068.9.0 (MoreOS)\"\nANSI_COLOR=\"1;32\"\nHOME_URL=\"https://coreos.com/\"\nBUG_REPORT_URL=\"https://github.com/coreos/bugs/issues\"\nLinux coreos1 4.6.3-coreos #2 SMP Fri Aug 5 04:51:16 UTC 2016 x86_64 QEMU Virtual CPU version 1.5.3 GenuineIntel GNU/Linux\n```\n\nSometimes after reboot pods scheduled from a daemonset do not run  on the master node. \n\n``` bash\nkubectl describe daemonsets ceph-daemon-1 --namespace=ceph\nName:       ceph-daemon-1\nImage(s):   ...\nSelector:   app=ceph-daemon-1\nNode-Selector:  nodenum=1\nLabels:     app=ceph-daemon-1\nDesired Number of Nodes Scheduled: 1\nCurrent Number of Nodes Scheduled: 1\nNumber of Nodes Misscheduled: 0\nPods Status:    0 Running / 0 Waiting / 0 Succeeded / 1 Failed\nEvents:\n  FirstSeen LastSeen    Count   From        SubobjectPath   Type    Reason          Message\n  --------- --------    -----   ----        -------------   --------    ------          -------\n  50m       50m     1   {daemon-set }           NormalSuccessfulCreate  Created pod: ceph-daemon-1-z6t5n\n```\n\n``` bash\nkubectl get pods --namespace=ceph\nNAME                  READY     STATUS    RESTARTS   AGE\nceph-daemon-2-ev1g3   3/3       Running   18         22h\nceph-daemon-3-rj22g   3/3       Running   9          22h\n```\n\nno containers are created\n\n``` bash\ndocker ps | grep ceph\n```\n\nbefore json manifest dump\n\n```\nkubectl logs kube-controller-manager --namespace=kube-system | grep ceph-daemon-1\nI0822 12:43:50.048854       1 controller.go:147] Adding daemon set ceph-daemon-1\nI0822 12:43:50.056660       1 replica_set.go:298] Pod ceph-daemon-1-t6dkf created: kind:\"\" apiVersion:\"\" .\nI0822 12:43:50.056699       1 replica_set.go:244] No ReplicaSets found for pod ceph-daemon-1-t6dkf, ReplicaSet controller will avoid syncing\nI0822 12:43:50.062149       1 deployment_controller.go:319] Pod ceph-daemon-1-t6dkf created: kind:\"\" apiVersion:\"\" .\nI0822 12:43:50.062170       1 deployment_controller.go:300] Error: could not find ReplicaSet for pod ceph-daemon-1-t6dkf in namespace ceph with labels: map[app:ceph-daemon-1]. No ReplicaSets found for pod ceph-daemon-1-t6dkf, deployment controller will avoid syncing.\nI0822 12:43:50.092647       1 controller.go:356] Pod ceph-daemon-1-t6dkf added.\nI0822 12:43:50.094206       1 controller.go:166] No jobs found for pod ceph-daemon-1-t6dkf, job controller will avoid syncing\nI0822 12:43:50.094374       1 replication_controller.go:275] No controllers found for pod ceph-daemon-1-t6dkf, replication manager will avoid syncing\nI0822 12:43:50.095742       1 pet_set.go:158] Pod ceph-daemon-1-t6dkf created, labels: map[app:ceph-daemon-1]\nI0822 12:43:50.095761       1 pet_set.go:234] No PetSets found for pod ceph-daemon-1-t6dkf, PetSet controller will avoid syncing\nI0822 12:43:50.113455       1 endpoints_controller.go:322] Waiting for pods controller to sync, requeuing service ceph/ceph-daemon-1\nI0822 12:43:50.113479       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (101.210897ms)\nI0822 12:43:50.154928       1 controller_utils.go:161] Controller ceph/ceph-daemon-1 either never recorded expectations, or the ttl expired.\nI0822 12:43:50.155645       1 controller_utils.go:178] Setting expectations &controller.ControlleeExpectations{add:0, del:0, key:\"ceph/ceph-daemon-1\", timestamp:time.Time{sec:63607466630, nsec:155641687, loc:(*time.Location)(0x588a4e0)}}\nI0822 12:43:50.155677       1 controller.go:538] Nodes needing daemon pods for daemon set ceph-daemon-1: [], creating 0\nI0822 12:43:50.155691       1 controller.go:553] Pods to delete for daemon set ceph-daemon-1: [], deleting 0\nI0822 12:43:50.156165       1 controller.go:635] Finished syncing daemon set \"ceph/ceph-daemon-1\" (1.246865ms)\nI0822 12:43:50.161059       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (7.092245ms)\n```\n\nend of the json manifest dump that shows node selector fails (still with `grep ceph-daemon-1`)\n\n```\nStatus:api.PodStatus{Phase:\"Failed\", Conditions:[]api.PodCondition(nil), Message:\"Pod Predicate MatchNodeSelector failed\", Reason:\"MatchNodeSelector\", HostIP:\"\", PodIP:\"\", StartTime:(*unversioned.Time)(0xc820e6b520), InitContainerStatuses:[]api.ContainerStatus(nil), ContainerStatuses:[]api.ContainerStatus(nil)}}.\nI0822 12:43:58.091310       1 deployment_controller.go:300] Error: could not find ReplicaSet for pod ceph-daemon-1-t6dkf in namespace ceph with labels: map[app:ceph-daemon-1]. No ReplicaSets found for pod ceph-daemon-1-t6dkf, deployment controller will avoid syncing.\nI0822 12:43:58.091333       1 deployment_controller.go:300] Error: could not find ReplicaSet for pod ceph-daemon-1-t6dkf in namespace ceph with labels: map[app:ceph-daemon-1]. No ReplicaSets found for pod ceph-daemon-1-t6dkf, deployment controller will avoid syncing.\nI0822 12:43:58.091367       1 controller.go:166] No jobs found for pod ceph-daemon-1-t6dkf, job controller will avoid syncing\nI0822 12:43:58.091634       1 controller_utils.go:178] Setting expectations &controller.ControlleeExpectations{add:0, del:0, key:\"ceph/ceph-daemon-1\", timestamp:time.Time{sec:63607466638, nsec:91630726, loc:(*time.Location)(0x588a4e0)}}\nI0822 12:43:58.091655       1 controller.go:538] Nodes needing daemon pods for daemon set ceph-daemon-1: [], creating 0\nI0822 12:43:58.091663       1 controller.go:553] Pods to delete for daemon set ceph-daemon-1: [], deleting 0\nI0822 12:43:58.091764       1 replica_set.go:329] Pod ceph-daemon-1-t6dkf updated, objectMeta {Name:ceph-daemon-1-t6dkf GenerateName:ceph-daemon-1- Namespace:ceph SelfLink:/api/v1/namespaces/ceph/pods/ceph-daemon-1-t6dkf UID:6e81bfde-67d6-11e6-a00f-00163c830412 ResourceVersion:19865048 Generation:0 CreationTimestamp:{Time:2016-08-21 19:35:35 +0000 UTC} DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[app:ceph-daemon-1] Annotations:map[kubernetes.io/created-by:{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"DaemonSet\",\"namespace\":\"ceph\",\"name\":\"ceph-daemon-1\",\"uid\":\"6e7e6ec1-67d6-11e6-a00f-00163c830412\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"19865015\"}}\n] OwnerReferences:[] Finalizers:[]} -> {Name:ceph-daemon-1-t6dkf GenerateName:ceph-daemon-1- Namespace:ceph SelfLink:/api/v1/namespaces/ceph/pods/ceph-daemon-1-t6dkf UID:6e81bfde-67d6-11e6-a00f-00163c830412 ResourceVersion:20010907 Generation:0 CreationTimestamp:{Time:2016-08-21 19:35:35 +0000 UTC} DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[app ceph-daemon-1] Annotations:map[kubernetes.io/created-by:{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"DaemonSet\",\"namespace\":\"ceph\",\"name\":\"ceph-daemon-1\",\"uid\":\"6e7e6ec1-67d6-11e6-a00f-00163c830412\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"19865015\"}}\nI0822 12:43:58.091854       1 pet_set.go:234] No PetSets found for pod ceph-daemon-1-t6dkf, PetSet controller will avoid syncing\nI0822 12:43:58.091871       1 replica_set.go:244] No ReplicaSets found for pod ceph-daemon-1-t6dkf, ReplicaSet controller will avoid syncing\nI0822 12:43:58.091969       1 controller.go:635] Finished syncing daemon set \"ceph/ceph-daemon-1\" (745.643µs)\nI0822 12:43:58.103371       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (12.603004ms)\nI0822 12:44:19.991972       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (9.311024ms)\nI0822 12:44:20.035513       1 controller.go:168] Updating daemon set ceph-daemon-1\nI0822 12:44:20.050332       1 controller_utils.go:178] Setting expectations &controller.ControlleeExpectations{add:0, del:0, key:\"ceph/ceph-daemon-1\", timestamp:time.Time{sec:63607466660, nsec:50328049, loc:(*time.Location)(0x588a4e0)}}\nI0822 12:44:20.050373       1 controller.go:538] Nodes needing daemon pods for daemon set ceph-daemon-1: [], creating 0\nI0822 12:44:20.050411       1 controller.go:553] Pods to delete for daemon set ceph-daemon-1: [], deleting 0\nI0822 12:44:20.050816       1 controller.go:635] Finished syncing daemon set \"ceph/ceph-daemon-1\" (979.7µs)\nI0822 12:44:49.998274       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (9.794818ms)\nI0822 12:44:50.035498       1 controller.go:168] Updating daemon set ceph-daemon-1\nI0822 12:44:50.036299       1 controller_utils.go:178] Setting expectations &controller.ControlleeExpectations{add:0, del:0, key:\"ceph/ceph-daemon-1\", timestamp:time.Time{sec:63607466690, nsec:36292015, loc:(*time.Location)(0x588a4e0)}}\nI0822 12:44:50.036344       1 controller.go:538] Nodes needing daemon pods for daemon set ceph-daemon-1: [], creating 0\nI0822 12:44:50.036359       1 controller.go:553] Pods to delete for daemon set ceph-daemon-1: [], deleting 0\nI0822 12:44:50.036628       1 controller.go:635] Finished syncing daemon set \"ceph/ceph-daemon-1\" (978.748µs)\nI0822 12:45:20.002096       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (18.636798ms)\nI0822 12:45:20.036849       1 controller.go:168] Updating daemon set ceph-daemon-1\nI0822 12:45:20.038182       1 controller_utils.go:178] Setting expectations &controller.ControlleeExpectations{add:0, del:0, key:\"ceph/ceph-daemon-1\", timestamp:time.Time{sec:63607466720, nsec:38171858, loc:(*time.Location)(0x588a4e0)}}\nI0822 12:45:20.038235       1 controller.go:538] Nodes needing daemon pods for daemon set ceph-daemon-1: [], creating 0\nI0822 12:45:20.038276       1 controller.go:553] Pods to delete for daemon set ceph-daemon-1: [], deleting 0\nI0822 12:45:20.038938       1 controller.go:635] Finished syncing daemon set \"ceph/ceph-daemon-1\" (1.869225ms)\nI0822 12:45:49.996657       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (6.353948ms)\nI0822 12:45:50.036997       1 controller.go:168] Updating daemon set ceph-daemon-1\nI0822 12:45:50.037926       1 controller_utils.go:178] Setting expectations &controller.ControlleeExpectations{add:0, del:0, key:\"ceph/ceph-daemon-1\", timestamp:time.Time{sec:63607466750, nsec:37918753, loc:(*time.Location)(0x588a4e0)}}\nI0822 12:45:50.037961       1 controller.go:538] Nodes needing daemon pods for daemon set ceph-daemon-1: [], creating 0\nI0822 12:45:50.037974       1 controller.go:553] Pods to delete for daemon set ceph-daemon-1: [], deleting 0\nI0822 12:45:50.038817       1 controller.go:635] Finished syncing daemon set \"ceph/ceph-daemon-1\" (1.62769ms)\nI0822 12:46:20.017322       1 endpoints_controller.go:316] Finished syncing service \"ceph/ceph-daemon-1\" endpoints. (24.107295ms)\nI0822 12:46:20.037628       1 controller.go:168] Updating daemon set ceph-daemon-1\n```\n\ndeleting the orphaned pod manually is enough to reschedule and run another one successfully\n`kubectl delete pods ceph-daemon-1-t6dkf --namespace=ceph`\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/291098016","html_url":"https://github.com/kubernetes/kubernetes/issues/31123#issuecomment-291098016","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31123","id":291098016,"user":{"login":"fabiand","id":83456,"avatar_url":"https://avatars2.githubusercontent.com/u/83456?v=3","gravatar_id":"","url":"https://api.github.com/users/fabiand","html_url":"https://github.com/fabiand","followers_url":"https://api.github.com/users/fabiand/followers","following_url":"https://api.github.com/users/fabiand/following{/other_user}","gists_url":"https://api.github.com/users/fabiand/gists{/gist_id}","starred_url":"https://api.github.com/users/fabiand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabiand/subscriptions","organizations_url":"https://api.github.com/users/fabiand/orgs","repos_url":"https://api.github.com/users/fabiand/repos","events_url":"https://api.github.com/users/fabiand/events{/privacy}","received_events_url":"https://api.github.com/users/fabiand/received_events","type":"User","site_admin":false},"created_at":"2017-04-03T09:52:12Z","updated_at":"2017-04-03T09:52:12Z","body":"@resouer we are also seeing this sometimes. Are you saying this is not a bug?\r\nIt rather needs to be solved by ensuring that the labels are settled when the daemonset gets defined?"}}
{"actor":{"display_login":"fabiand"}}