{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/41007","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/41007/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/41007/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/41007/events","html_url":"https://github.com/kubernetes/kubernetes/issues/41007","id":205545546,"number":41007,"title":"\"Failed to register layer\" after pulling docker images","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"id":114528223,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/33","html_url":"https://github.com/kubernetes/kubernetes/milestone/33","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/33/labels","id":2265498,"number":33,"title":"1.6","description":null,"creator":{"login":"timothysc","id":169553,"avatar_url":"https://avatars.githubusercontent.com/u/169553?v=3","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":22,"state":"open","created_at":"2017-01-19T21:27:19Z","updated_at":"2017-02-06T10:43:10Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2017-02-06T10:43:09Z","updated_at":"2017-02-06T10:43:17Z","closed_at":null,"body":"On saturday, in large cluster we started observing runs (all runs since then failed due to the same reason), in which not all system pods come up. \r\nIn particular, always some number of fluentd are not starting. I looked into why those are not starting and all of them are failing with logs like these:\r\n\r\n```\r\n...\r\nE0205 01:18:07.691491    1508 pod_workers.go:182] Error syncing pod 58fff48e-eb40-11e6-bd1d-42010af00002 (\"fluentd-gcp-v1.35-zhn1k_kube-system(58fff48e-eb40-11e6-bd1d-42010af00002)\"), skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR]\r\nI0205 01:18:21.991789    1508 docker_manager.go:2002] Need to restart pod infra container for \"fluentd-gcp-v1.35-zhn1k_kube-system(58fff48e-eb40-11e6-bd1d-42010af00002)\" because it is not found\r\nI0205 01:18:22.350163    1508 docker_manager.go:2263] Determined pod ip after infra change: \"fluentd-gcp-v1.35-zhn1k_kube-system(58fff48e-eb40-11e6-bd1d-42010af00002)\": \"10.215.107.3\"\r\nI0205 01:18:22.913070    1508 kubelet.go:1805] SyncLoop (PLEG): \"fluentd-gcp-v1.35-zhn1k_kube-system(58fff48e-eb40-11e6-bd1d-42010af00002)\", event: &pleg.PodLifecycleEvent{ID:\"58fff48e-eb40-11e6-bd1d-42010af00002\", Type:\"ContainerStarted\", Data:\"8c62da43a6e72a5e5522df780b2cdf4807d16cc06b63e64c1de826c6862a6b6b\"}\r\nI0205 01:18:32.626675    1508 kube_docker_client.go:326] Pulling image \"gcr.io/google_containers/fluentd-gcp:1.35\": \"a3a9a5587906: Extracting [==================================================>] 73.72 MB/73.72 MB\"\r\nE0205 01:18:35.694647    1508 pod_workers.go:182] Error syncing pod 58fff48e-eb40-11e6-bd1d-42010af00002 (\"fluentd-gcp-v1.35-zhn1k_kube-system(58fff48e-eb40-11e6-bd1d-42010af00002)\"), skipping: failed to \"StartContainer\" for \"fluentd-gcp\" with ErrImagePull: \"failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-889925918 /var/lib/docker/image/overlay/layerdb/sha256/09995c36eaaa0805fbabd4d5d2b547ef197a9d8a5870b61fe3a8c6de0e73ca27: directory not empty\"\r\n```\r\n\r\nExample failures of this are in:\r\nhttp://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/58/artifacts/gke-bootstrap-e2e-default-pool-0-0d038807-36tc/\r\nhttp://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/59/artifacts/gke-bootstrap-e2e-default-pool-0-648c74ed-knzl/\r\n\r\n@kubernetes/sig-node-bugs @dchen1107 @yujuhong "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/277646239","html_url":"https://github.com/kubernetes/kubernetes/issues/41007#issuecomment-277646239","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/41007","id":277646239,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2017-02-06T10:43:17Z","updated_at":"2017-02-06T10:43:17Z","body":"@gmarek "}}
{"actor":{"display_login":"wojtek-t"}}