{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39028","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39028/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39028/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39028/events","html_url":"https://github.com/kubernetes/kubernetes/issues/39028","id":196705705,"number":39028,"title":"RunContainerError: \"runContainer: operation timeout: context deadline exceeded\"","user":{"login":"george-angel","id":1801067,"avatar_url":"https://avatars3.githubusercontent.com/u/1801067?v=3","gravatar_id":"","url":"https://api.github.com/users/george-angel","html_url":"https://github.com/george-angel","followers_url":"https://api.github.com/users/george-angel/followers","following_url":"https://api.github.com/users/george-angel/following{/other_user}","gists_url":"https://api.github.com/users/george-angel/gists{/gist_id}","starred_url":"https://api.github.com/users/george-angel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/george-angel/subscriptions","organizations_url":"https://api.github.com/users/george-angel/orgs","repos_url":"https://api.github.com/users/george-angel/repos","events_url":"https://api.github.com/users/george-angel/events{/privacy}","received_events_url":"https://api.github.com/users/george-angel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-12-20T15:45:51Z","updated_at":"2017-03-23T23:49:15Z","closed_at":null,"body":"Kubernetes version:\r\n\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"5\", GitVersion:\"v1.5.1\", GitCommit:\"82450d03cb057bab0950214ef122b67c83fb11df\", GitTreeState:\"clean\", BuildDate:\"2016-12-14T00:57:05Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"5\", GitVersion:\"v1.5.1+coreos.0\", GitCommit:\"cc65f5321f9230bf9a3fa171155c1213d6e3480e\", GitTreeState:\"clean\", BuildDate:\"2016-12-14T04:08:28Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\nOS Version: `CoreOS stable (1185.5.0)`\r\n\r\nFrom the kubelet logs:\r\n\r\n```\r\nDec 20 15:40:30 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: I1220 15:40:30.077807    1401 operation_executor.go:917] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/75207ba5-c6c6-11e6-8758-06099f2a458d-default-token-3nexu\" (spec.Name: \"default-token-3nexu\") pod \"75207ba5-c6c6-11e6-8758-06099f2a458d\" (UID: \"75207ba5-c6c6-11e6-8758-06099f2a458d\").\r\nDec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: E1220 15:40:31.616962    1401 docker_manager.go:2324] container start failed: RunContainerError: runContainer: operation timeout: context deadline exceeded\r\nDec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: E1220 15:40:31.617011    1401 pod_workers.go:184] Error syncing pod 3d09fff2-c6c7-11e6-8758-06099f2a458d, skipping: failed to \"StartContainer\" for \"prometheus\" with RunContainerError: \"runContainer: operation timeout: context deadline exceeded\"\r\nDec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: I1220 15:40:31.880783    1401 operation_executor.go:917] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/3d09fff2-c6c7-11e6-8758-06099f2a458d-default-token-0shxu\" (spec.Name: \"default-token-0shxu\") pod \"3d09fff2-c6c7-11e6-8758-06099f2a458d\" (UID: \"3d09fff2-c6c7-11e6-8758-06099f2a458d\").\r\nDec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: I1220 15:40:31.881996    1401 operation_executor.go:917] MountVolume.SetUp succeeded for volume \"kubernetes.io/configmap/3d09fff2-c6c7-11e6-8758-06099f2a458d-config-volume\" (spec.Name: \"config-volume\") pod \"3d09fff2-c6c7-11e6-8758-06099f2a458d\" (UID: \"3d09fff2-c6c7-11e6-8758-06099f2a458d\").\r\n```\r\n\r\ndocker ps:\r\n\r\n```\r\n8c2ea1120e4a        prom/prometheus:v1.2.1                                                        \"/bin/prometheus -sto\"   10 seconds ago       Created                                             k8s_prometheus.edf8dc49_prometheus-227300114-v7z1h_system_3d09fff2-c6c7-11e6-8758-06099f2a458d_af434bd8\r\n```\r\n\r\nNo logs on the container:\r\n\r\n```\r\ncore@ip-10-66-21-87 ~ $ docker logs 8c2ea1120e4a\r\ncore@ip-10-66-21-87 ~ $\r\n```\r\n\r\ndocker inspect:\r\n\r\n```\r\n[\r\n    {\r\n        \"Id\": \"8c2ea1120e4a8b41ca2a0c4ef9afca5d19d790c633e1b965434a0ec9fca9cd04\",\r\n        \"Created\": \"2016-12-20T15:42:32.840639703Z\",\r\n        \"Path\": \"/bin/prometheus\",\r\n        \"Args\": [\r\n            \"-storage.local.retention=336h\",\r\n            \"-storage.local.memory-chunks=262144\",\r\n            \"-config.file=/etc/prometheus/prometheus.yml\",\r\n            \"-alertmanager.url=http://alertmanager:9093\",\r\n            \"-web.external-url=https://prometheus.dev.uw.systems\"\r\n        ],\r\n        \"State\": {\r\n            \"Status\": \"created\",\r\n            \"Running\": false,\r\n            \"Paused\": false,\r\n            \"Restarting\": false,\r\n            \"OOMKilled\": false,\r\n            \"Dead\": false,\r\n            \"Pid\": 0,\r\n            \"ExitCode\": 0,\r\n            \"Error\": \"\",\r\n            \"StartedAt\": \"0001-01-01T00:00:00Z\",\r\n            \"FinishedAt\": \"0001-01-01T00:00:00Z\"\r\n        },\r\n        \"Image\": \"sha256:18eb3e2914849e994e7a22ad84e1ce3c7d41a51eec2ce68584728f36181900bc\",\r\n        \"ResolvConfPath\": \"\",\r\n        \"HostnamePath\": \"\",\r\n        \"HostsPath\": \"\",\r\n        \"LogPath\": \"\",\r\n        \"Name\": \"/k8s_prometheus.edf8dc49_prometheus-227300114-v7z1h_system_3d09fff2-c6c7-11e6-8758-06099f2a458d_af434bd8\",\r\n        \"RestartCount\": 0,\r\n        \"Driver\": \"overlay\",\r\n        \"MountLabel\": \"system_u:object_r:svirt_lxc_file_t:s0:c241,c498\",\r\n        \"ProcessLabel\": \"system_u:system_r:svirt_lxc_net_t:s0:c241,c498\",\r\n        \"AppArmorProfile\": \"\",\r\n        \"ExecIDs\": null,\r\n        \"HostConfig\": {\r\n            \"Binds\": null,\r\n            \"ContainerIDFile\": \"\",\r\n            \"LogConfig\": {\r\n                \"Type\": \"json-file\",\r\n                \"Config\": {}\r\n            },\r\n            \"NetworkMode\": \"\",\r\n            \"PortBindings\": null,\r\n            \"RestartPolicy\": {\r\n                \"Name\": \"\",\r\n                \"MaximumRetryCount\": 0\r\n            },\r\n            \"AutoRemove\": false,\r\n            \"VolumeDriver\": \"\",\r\n            \"VolumesFrom\": null,\r\n            \"CapAdd\": null,\r\n            \"CapDrop\": null,\r\n            \"Dns\": null,\r\n            \"DnsOptions\": null,\r\n            \"DnsSearch\": null,\r\n            \"ExtraHosts\": null,\r\n            \"GroupAdd\": null,\r\n            \"IpcMode\": \"\",\r\n            \"Cgroup\": \"\",\r\n            \"Links\": null,\r\n            \"OomScoreAdj\": 0,\r\n            \"PidMode\": \"\",\r\n            \"Privileged\": false,\r\n            \"PublishAllPorts\": false,\r\n            \"ReadonlyRootfs\": false,\r\n            \"SecurityOpt\": null,\r\n            \"StorageOpt\": null,\r\n            \"UTSMode\": \"\",\r\n            \"UsernsMode\": \"\",\r\n            \"ShmSize\": 0,\r\n            \"ConsoleSize\": [\r\n                0,\r\n                0\r\n            ],\r\n            \"Isolation\": \"\",\r\n            \"CpuShares\": 0,\r\n            \"Memory\": 0,\r\n            \"CgroupParent\": \"\",\r\n            \"BlkioWeight\": 0,\r\n            \"BlkioWeightDevice\": null,\r\n            \"BlkioDeviceReadBps\": null,\r\n            \"BlkioDeviceWriteBps\": null,\r\n            \"BlkioDeviceReadIOps\": null,\r\n            \"BlkioDeviceWriteIOps\": null,\r\n            \"CpuPeriod\": 0,\r\n            \"CpuQuota\": 0,\r\n            \"CpusetCpus\": \"\",\r\n            \"CpusetMems\": \"\",\r\n            \"Devices\": null,\r\n            \"DiskQuota\": 0,\r\n            \"KernelMemory\": 0,\r\n            \"MemoryReservation\": 0,\r\n            \"MemorySwap\": 0,\r\n            \"MemorySwappiness\": null,\r\n            \"OomKillDisable\": null,\r\n            \"PidsLimit\": 0,\r\n            \"Ulimits\": null,\r\n            \"CpuCount\": 0,\r\n            \"CpuPercent\": 0,\r\n            \"BlkioIOps\": 0,\r\n            \"BlkioBps\": 0,\r\n            \"SandboxSize\": 0\r\n        },\r\n        \"GraphDriver\": {\r\n            \"Name\": \"overlay\",\r\n            \"Data\": {\r\n                \"LowerDir\": \"/var/lib/docker/overlay/a84216d9ef3e8348e9296b791fa044617d02ede93add2eaf4564effa5c31644d/root\",\r\n                \"MergedDir\": \"/var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/merged\",\r\n                \"UpperDir\": \"/var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/upper\",\r\n                \"WorkDir\": \"/var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/work\"\r\n            }\r\n        },\r\n        \"Mounts\": [],\r\n        \"Config\": {\r\n            \"Hostname\": \"8c2ea1120e4a\",\r\n            \"Domainname\": \"\",\r\n            \"User\": \"\",\r\n            \"AttachStdin\": false,\r\n            \"AttachStdout\": false,\r\n            \"AttachStderr\": false,\r\n            \"ExposedPorts\": {\r\n                \"9090/tcp\": {}\r\n            },\r\n            \"Tty\": false,\r\n            \"OpenStdin\": false,\r\n            \"StdinOnce\": false,\r\n            \"Env\": [\r\n                \"PROMETHEUS_PORT_9090_TCP_PROTO=tcp\",\r\n                \"ALERTMANAGER_PORT_9093_TCP=tcp://10.3.0.53:9093\",\r\n                \"ALERTMANAGER_PORT_9093_TCP_PORT=9093\",\r\n                \"GITLAB_PORT_80_TCP_ADDR=10.3.0.28\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_80_TCP_PORT=80\",\r\n                \"PROM_PUSHGATEWAY_PORT_80_TCP_ADDR=10.3.0.251\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_8080_TCP_PROTO=tcp\",\r\n                \"PROMETHEUS_SERVICE_PORT=9090\",\r\n                \"GITLAB_PORT_80_TCP_PROTO=tcp\",\r\n                \"EXTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP_ADMIN=8080\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP_PROTO=tcp\",\r\n                \"INTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP=80\",\r\n                \"INTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP_ADMIN=8080\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP_PROTO=tcp\",\r\n                \"PROM_PUSHGATEWAY_SERVICE_PORT_PROMETHEUS=80\",\r\n                \"PROM_PUSHGATEWAY_PORT=tcp://10.3.0.251:80\",\r\n                \"GITLAB_SERVICE_PORT_HTTPS=443\",\r\n                \"GITLAB_PORT_443_TCP_PROTO=tcp\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP_PORT=80\",\r\n                \"PROMETHEUS_SERVICE_HOST=10.3.0.110\",\r\n                \"PROMETHEUS_PORT_9090_TCP_PORT=9090\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP_PORT=6379\",\r\n                \"PROMETHEUS_SERVICE_PORT_PROMETHEUS=9090\",\r\n                \"PROMETHEUS_PORT_9090_TCP_ADDR=10.3.0.110\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP=tcp://10.3.0.44:80\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP_PROTO=tcp\",\r\n                \"BLACKBOX_EXPORTER_SERVICE_PORT_HTTP=80\",\r\n                \"GRAFANA_PORT_3000_TCP=tcp://10.3.0.113:3000\",\r\n                \"GITLAB_PORT_443_TCP=tcp://10.3.0.28:443\",\r\n                \"GITLAB_PORT_22_TCP_PROTO=tcp\",\r\n                \"PROM_PUSHGATEWAY_PORT_80_TCP_PORT=80\",\r\n                \"GITLAB_SERVICE_HOST=10.3.0.28\",\r\n                \"GITLAB_PORT_22_TCP=tcp://10.3.0.28:22\",\r\n                \"GITHUB_SSHKEY_PROVIDER_SERVICE_PORT_REDIS=6379\",\r\n                \"GRAFANA_SERVICE_HOST=10.3.0.113\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP_ADDR=10.3.0.44\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP_ADDR=10.3.0.161\",\r\n                \"KUBERNETES_PORT_443_TCP_ADDR=10.3.0.1\",\r\n                \"GITLAB_PORT_443_TCP_ADDR=10.3.0.28\",\r\n                \"INTERNAL_INGRESS_PROXY_SERVICE_PORT=80\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_8080_TCP=tcp://10.3.0.176:8080\",\r\n                \"PROM_PUSHGATEWAY_SERVICE_HOST=10.3.0.251\",\r\n                \"KUBERNETES_PORT_443_TCP_PORT=443\",\r\n                \"KUBERNETES_SERVICE_PORT=443\",\r\n                \"GITLAB_SERVICE_PORT=80\",\r\n                \"BLACKBOX_EXPORTER_SERVICE_PORT=80\",\r\n                \"GRAFANA_PORT_3000_TCP_ADDR=10.3.0.113\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP=tcp://10.3.0.44:6379\",\r\n                \"EXTERNAL_INGRESS_PROXY_SERVICE_HOST=10.3.0.161\",\r\n                \"EXTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP=80\",\r\n                \"KUBERNETES_PORT_443_TCP=tcp://10.3.0.1:443\",\r\n                \"GITLAB_PORT_22_TCP_PORT=22\",\r\n                \"ALERTMANAGER_PORT_9093_TCP_PROTO=tcp\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_8080_TCP_PORT=8080\",\r\n                \"PROMETHEUS_PORT_9090_TCP=tcp://10.3.0.110:9090\",\r\n                \"KUBERNETES_PORT_443_TCP_PROTO=tcp\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_80_TCP=tcp://10.3.0.161:80\",\r\n                \"ALERTMANAGER_SERVICE_HOST=10.3.0.53\",\r\n                \"BLACKBOX_EXPORTER_PORT_80_TCP_PORT=80\",\r\n                \"PROM_PUSHGATEWAY_SERVICE_PORT=80\",\r\n                \"PROMETHEUS_PORT=tcp://10.3.0.110:9090\",\r\n                \"KUBERNETES_SERVICE_HOST=10.3.0.1\",\r\n                \"EXTERNAL_INGRESS_PROXY_SERVICE_PORT=80\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT=tcp://10.3.0.161:80\",\r\n                \"GRAFANA_PORT_3000_TCP_PORT=3000\",\r\n                \"PROM_PUSHGATEWAY_PORT_80_TCP=tcp://10.3.0.251:80\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_80_TCP_PORT=80\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_8080_TCP_ADDR=10.3.0.176\",\r\n                \"PROM_PUSHGATEWAY_PORT_80_TCP_PROTO=tcp\",\r\n                \"BLACKBOX_EXPORTER_SERVICE_HOST=10.3.0.3\",\r\n                \"BLACKBOX_EXPORTER_PORT_80_TCP=tcp://10.3.0.3:80\",\r\n                \"ALERTMANAGER_PORT=tcp://10.3.0.53:9093\",\r\n                \"GRAFANA_PORT=tcp://10.3.0.113:3000\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT=tcp://10.3.0.176:80\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_80_TCP_PROTO=tcp\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP_PORT=8080\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_80_TCP_ADDR=10.3.0.161\",\r\n                \"GRAFANA_SERVICE_PORT=3000\",\r\n                \"ALERTMANAGER_PORT_9093_TCP_ADDR=10.3.0.53\",\r\n                \"GITLAB_PORT_443_TCP_PORT=443\",\r\n                \"GITLAB_PORT_22_TCP_ADDR=10.3.0.28\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT=tcp://10.3.0.44:80\",\r\n                \"INTERNAL_INGRESS_PROXY_SERVICE_HOST=10.3.0.176\",\r\n                \"KUBERNETES_SERVICE_PORT_HTTPS=443\",\r\n                \"KUBERNETES_PORT=tcp://10.3.0.1:443\",\r\n                \"BLACKBOX_EXPORTER_PORT=tcp://10.3.0.3:80\",\r\n                \"ALERTMANAGER_SERVICE_PORT_ALERTMANAGER=9093\",\r\n                \"GITLAB_SERVICE_PORT_SSH=22\",\r\n                \"GITLAB_PORT=tcp://10.3.0.28:80\",\r\n                \"GITHUB_SSHKEY_PROVIDER_SERVICE_HOST=10.3.0.44\",\r\n                \"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP_ADDR=10.3.0.44\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_80_TCP=tcp://10.3.0.176:80\",\r\n                \"GITLAB_PORT_80_TCP_PORT=80\",\r\n                \"GITHUB_SSHKEY_PROVIDER_SERVICE_PORT=80\",\r\n                \"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP=tcp://10.3.0.161:8080\",\r\n                \"GRAFANA_PORT_3000_TCP_PROTO=tcp\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_80_TCP_ADDR=10.3.0.176\",\r\n                \"INTERNAL_INGRESS_PROXY_PORT_80_TCP_PROTO=tcp\",\r\n                \"GITLAB_SERVICE_PORT_HTTP=80\",\r\n                \"GITLAB_PORT_80_TCP=tcp://10.3.0.28:80\",\r\n                \"BLACKBOX_EXPORTER_PORT_80_TCP_PROTO=tcp\",\r\n                \"BLACKBOX_EXPORTER_PORT_80_TCP_ADDR=10.3.0.3\",\r\n                \"ALERTMANAGER_SERVICE_PORT=9093\",\r\n                \"GITHUB_SSHKEY_PROVIDER_SERVICE_PORT_WEB=80\",\r\n                \"GRAFANA_SERVICE_PORT_GRAFANA=3000\",\r\n                \"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"\r\n            ],\r\n            \"Cmd\": [\r\n                \"-storage.local.retention=336h\",\r\n                \"-storage.local.memory-chunks=262144\",\r\n                \"-config.file=/etc/prometheus/prometheus.yml\",\r\n                \"-alertmanager.url=http://alertmanager:9093\",\r\n                \"-web.external-url=https://prometheus.dev.uw.systems\"\r\n            ],\r\n            \"Image\": \"prom/prometheus:v1.2.1\",\r\n            \"Volumes\": {\r\n                \"/prometheus\": {}\r\n            },\r\n            \"WorkingDir\": \"/prometheus\",\r\n            \"Entrypoint\": [\r\n                \"/bin/prometheus\"\r\n            ],\r\n            \"OnBuild\": null,\r\n            \"Labels\": {\r\n                \"io.kubernetes.container.hash\": \"edf8dc49\",\r\n                \"io.kubernetes.container.name\": \"prometheus\",\r\n                \"io.kubernetes.container.ports\": \"[{\\\"name\\\":\\\"web\\\",\\\"containerPort\\\":9090,\\\"protocol\\\":\\\"TCP\\\"}]\",\r\n                \"io.kubernetes.container.restartCount\": \"11\",\r\n                \"io.kubernetes.container.terminationMessagePath\": \"/dev/termination-log\",\r\n                \"io.kubernetes.pod.name\": \"prometheus-227300114-v7z1h\",\r\n                \"io.kubernetes.pod.namespace\": \"system\",\r\n                \"io.kubernetes.pod.terminationGracePeriod\": \"30\",\r\n                \"io.kubernetes.pod.uid\": \"3d09fff2-c6c7-11e6-8758-06099f2a458d\"\r\n            }\r\n        },\r\n        \"NetworkSettings\": {\r\n            \"Bridge\": \"\",\r\n            \"SandboxID\": \"\",\r\n            \"HairpinMode\": false,\r\n            \"LinkLocalIPv6Address\": \"\",\r\n            \"LinkLocalIPv6PrefixLen\": 0,\r\n            \"Ports\": null,\r\n            \"SandboxKey\": \"\",\r\n            \"SecondaryIPAddresses\": null,\r\n            \"SecondaryIPv6Addresses\": null,\r\n            \"EndpointID\": \"\",\r\n            \"Gateway\": \"\",\r\n            \"GlobalIPv6Address\": \"\",\r\n            \"GlobalIPv6PrefixLen\": 0,\r\n            \"IPAddress\": \"\",\r\n            \"IPPrefixLen\": 0,\r\n            \"IPv6Gateway\": \"\",\r\n            \"MacAddress\": \"\",\r\n            \"Networks\": null\r\n        }\r\n    }\r\n]\r\n```\r\n\r\nkubectl describe pod:\r\n\r\n```\r\nName:\t\tprometheus-227300114-v7z1h\r\nNamespace:\tsystem\r\nNode:\t\tip-10-66-21-87.eu-west-1.compute.internal/10.66.21.87\r\nStart Time:\tTue, 20 Dec 2016 15:16:10 +0000\r\nLabels:\t\tapp=prometheus\r\n\t\tpod-template-hash=227300114\r\nStatus:\t\tPending\r\nIP:\t\t10.2.56.5\r\nControllers:\tReplicaSet/prometheus-227300114\r\nContainers:\r\n  prometheus:\r\n    Container ID:\tdocker://8c2ea1120e4a8b41ca2a0c4ef9afca5d19d790c633e1b965434a0ec9fca9cd04\r\n    Image:\t\tprom/prometheus:v1.2.1\r\n    Image ID:\t\tdocker://sha256:18eb3e2914849e994e7a22ad84e1ce3c7d41a51eec2ce68584728f36181900bc\r\n    Port:\t\t9090/TCP\r\n    Args:\r\n      -storage.local.retention=336h\r\n      -storage.local.memory-chunks=262144\r\n      -config.file=/etc/prometheus/prometheus.yml\r\n      -alertmanager.url=http://alertmanager:9093\r\n      -web.external-url=https://prometheus.dev.uw.systems\r\n    Limits:\r\n      cpu:\t1500m\r\n      memory:\t6000Mi\r\n    Requests:\r\n      cpu:\t\t250m\r\n      memory:\t\t2000Mi\r\n    State:\t\tWaiting\r\n      Reason:\t\tRunContainerError\r\n    Ready:\t\tFalse\r\n    Restart Count:\t11\r\n    Volume Mounts:\r\n      /etc/kubernetes/ssl from ssl (ro)\r\n      /etc/prometheus from config-volume (rw)\r\n      /prometheus from data (rw)\r\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-0shxu (ro)\r\n    Environment Variables:\t<none>\r\nConditions:\r\n  Type\t\tStatus\r\n  Initialized \tTrue \r\n  Ready \tFalse \r\n  PodScheduled \tTrue \r\nVolumes:\r\n  config-volume:\r\n    Type:\tConfigMap (a volume populated by a ConfigMap)\r\n    Name:\tprometheus\r\n  data:\r\n    Type:\tPersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\r\n    ClaimName:\tprometheus-ebs-pvc\r\n    ReadOnly:\tfalse\r\n  ssl:\r\n    Type:\tHostPath (bare host directory volume)\r\n    Path:\t/etc/kubernetes/ssl\r\n  default-token-0shxu:\r\n    Type:\tSecret (a volume populated by a Secret)\r\n    SecretName:\tdefault-token-0shxu\r\nQoS Class:\tBurstable\r\nTolerations:\t<none>\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\tSubObjectPath\tType\t\tReason\t\tMessage\r\n  ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r\n  29m\t\t29m\t\t1\t{default-scheduler }\t\t\tNormal\t\tScheduled\tSuccessfully assigned prometheus-227300114-v7z1h to ip-10-66-21-87.eu-west-1.compute.internal\r\n  29m\t\t27m\t\t8\t{controller-manager }\t\t\tWarning\t\tFailedMount\tFailed to attach volume \"pvc-64b14071-c5db-11e6-8758-06099f2a458d\" on node \"ip-10-66-21-87.eu-west-1.compute.internal\" with: Error attaching EBS volume \"vol-02ff0c2158d03e018\" to instance \"i-01fb7ff93bad20e38\": VolumeInUse: vol-02ff0c2158d03e018 is already attached to an instance\r\n\t\tstatus code: 400, request id: \r\n  27m\t\t27m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\t\t\t\t\tWarning\tFailedMount\tUnable to mount volumes for pod \"prometheus-227300114-v7z1h_system(3d09fff2-c6c7-11e6-8758-06099f2a458d)\": timeout expired waiting for volumes to attach/mount for pod \"prometheus-227300114-v7z1h\"/\"system\". list of unattached/unmounted volumes=[data]\r\n  27m\t\t27m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\t\t\t\t\tWarning\tFailedSync\tError syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \"prometheus-227300114-v7z1h\"/\"system\". list of unattached/unmounted volumes=[data]\r\n  24m\t\t24m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tNormal\tPulling\t\tpulling image \"prom/prometheus:v1.2.1\"\r\n  24m\t\t24m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tNormal\tPulled\t\tSuccessfully pulled image \"prom/prometheus:v1.2.1\"\r\n  22m\t\t43s\t12\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tWarning\tFailed\t\tFailed to create docker container \"prometheus\" of pod \"prometheus-227300114-v7z1h_system(3d09fff2-c6c7-11e6-8758-06099f2a458d)\" with error: operation timeout: context deadline exceeded\r\n  22m\t\t43s\t12\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\t\t\t\t\tWarning\tFailedSync\tError syncing pod, skipping: failed to \"StartContainer\" for \"prometheus\" with RunContainerError: \"runContainer: operation timeout: context deadline exceeded\"\r\n\r\n  22m\t42s\t12\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tNormal\tPulled\tContainer image \"prom/prometheus:v1.2.1\" already present on machine\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/288894277","html_url":"https://github.com/kubernetes/kubernetes/issues/39028#issuecomment-288894277","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39028","id":288894277,"user":{"login":"guoshimin","id":11787293,"avatar_url":"https://avatars1.githubusercontent.com/u/11787293?v=3","gravatar_id":"","url":"https://api.github.com/users/guoshimin","html_url":"https://github.com/guoshimin","followers_url":"https://api.github.com/users/guoshimin/followers","following_url":"https://api.github.com/users/guoshimin/following{/other_user}","gists_url":"https://api.github.com/users/guoshimin/gists{/gist_id}","starred_url":"https://api.github.com/users/guoshimin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guoshimin/subscriptions","organizations_url":"https://api.github.com/users/guoshimin/orgs","repos_url":"https://api.github.com/users/guoshimin/repos","events_url":"https://api.github.com/users/guoshimin/events{/privacy}","received_events_url":"https://api.github.com/users/guoshimin/received_events","type":"User","site_admin":false},"created_at":"2017-03-23T23:49:15Z","updated_at":"2017-03-23T23:49:15Z","body":"We are also hitting this issue after upgrading to v1.5.4 from v1.4.7. We noticed that in v1.5.4, kubelet adds a `:Z` to each bind mount spec if SELinux is enabled on the system. We patched kubelet to disable this behavior and the problem went away.\r\n\r\nOur running theory is that this causes libcontainer to try to recursively label all the files in the prometheus volume, which has a lot of files, and the container create request times out."}}
{"actor":{"display_login":"guoshimin"}}