{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/9882","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/9882/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/9882/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/9882/events","html_url":"https://github.com/rancher/rancher/issues/9882","id":257848041,"number":9882,"title":"Unable to start pod infra images in k8s","user":{"login":"naveensrinivasan","id":172697,"avatar_url":"https://avatars3.githubusercontent.com/u/172697?v=4","gravatar_id":"","url":"https://api.github.com/users/naveensrinivasan","html_url":"https://github.com/naveensrinivasan","followers_url":"https://api.github.com/users/naveensrinivasan/followers","following_url":"https://api.github.com/users/naveensrinivasan/following{/other_user}","gists_url":"https://api.github.com/users/naveensrinivasan/gists{/gist_id}","starred_url":"https://api.github.com/users/naveensrinivasan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/naveensrinivasan/subscriptions","organizations_url":"https://api.github.com/users/naveensrinivasan/orgs","repos_url":"https://api.github.com/users/naveensrinivasan/repos","events_url":"https://api.github.com/users/naveensrinivasan/events{/privacy}","received_events_url":"https://api.github.com/users/naveensrinivasan/received_events","type":"User","site_admin":false},"labels":[{"id":256070545,"url":"https://api.github.com/repos/rancher/rancher/labels/area/kubernetes","name":"area/kubernetes","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":{"login":"galal-hussein","id":3309968,"avatar_url":"https://avatars1.githubusercontent.com/u/3309968?v=4","gravatar_id":"","url":"https://api.github.com/users/galal-hussein","html_url":"https://github.com/galal-hussein","followers_url":"https://api.github.com/users/galal-hussein/followers","following_url":"https://api.github.com/users/galal-hussein/following{/other_user}","gists_url":"https://api.github.com/users/galal-hussein/gists{/gist_id}","starred_url":"https://api.github.com/users/galal-hussein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galal-hussein/subscriptions","organizations_url":"https://api.github.com/users/galal-hussein/orgs","repos_url":"https://api.github.com/users/galal-hussein/repos","events_url":"https://api.github.com/users/galal-hussein/events{/privacy}","received_events_url":"https://api.github.com/users/galal-hussein/received_events","type":"User","site_admin":false},"assignees":[{"login":"galal-hussein","id":3309968,"avatar_url":"https://avatars1.githubusercontent.com/u/3309968?v=4","gravatar_id":"","url":"https://api.github.com/users/galal-hussein","html_url":"https://github.com/galal-hussein","followers_url":"https://api.github.com/users/galal-hussein/followers","following_url":"https://api.github.com/users/galal-hussein/following{/other_user}","gists_url":"https://api.github.com/users/galal-hussein/gists{/gist_id}","starred_url":"https://api.github.com/users/galal-hussein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galal-hussein/subscriptions","organizations_url":"https://api.github.com/users/galal-hussein/orgs","repos_url":"https://api.github.com/users/galal-hussein/repos","events_url":"https://api.github.com/users/galal-hussein/events{/privacy}","received_events_url":"https://api.github.com/users/galal-hussein/received_events","type":"User","site_admin":false}],"milestone":null,"comments":10,"created_at":"2017-09-14T20:34:12Z","updated_at":"2017-11-10T05:41:45Z","closed_at":"2017-09-21T21:06:55Z","author_association":"NONE","body":"**Rancher versions:**\r\nrancher/server: v1.7.2\r\nrancher/agent:\r\n\r\n**Infrastructure Stack versions:**\r\nhealthcheck:\r\nipsec:\r\nnetwork-services:\r\nscheduler:\r\nkubernetes (if applicable): k8s\r\n\r\n**Docker version: (`docker version`,`docker info` preferred)** 1.12\r\n\r\n**Operating system and kernel: (`cat /etc/os-release`, `uname -r` preferred)** ubuntu xenial\r\n\r\n**Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)** bare-metal\r\n\r\n**Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)** single node\r\n\r\n**Environment Template: (Cattle/Kubernetes/Swarm/Mesos)** k8s\r\n\r\n**Steps to Reproduce:**\r\nWe are running k8s with private docker registry.\r\nWe want all the pods to run from the private docker registry so that it can be air-gapped.\r\n\r\nHere is our k8s version v1.7.2\r\n\r\nHere is my configuration for pod infra images\r\n[https://discourse-cdn-sjc1.com/standard/uploads/rancher/original/2X/f/f8481bfd59a2937aa5d88e1b325f24ab4e43da9b.png](https://discourse-cdn-sjc1.com/standard/uploads/rancher/original/2X/f/f8481bfd59a2937aa5d88e1b325f24ab4e43da9b.png)\r\nMy registry is set to ip-10-0-0-160:5000/gcr.io without a trailing slash. Learnt it the hard way the trailing slash causes issues.\r\n\r\nHere is my log on docker pull\r\n```\r\ndocker pull ip-10-0-0-160:5000/gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2\r\n1.14.2: Pulling from gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64\r\n\r\nDigest: sha256:70b157a6695e5dddc5a3741c401c0507f0fd9326376c77162c690f5f5e82a264\r\nStatus: Image is up to date for ip-10-0-0-160:5000/gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2\r\n```\r\nThe kube dns does not comeup.\r\n```\r\nNAME                                   READY     STATUS             RESTARTS   AGE\r\nheapster-1584063577-n56m2              1/1       Running            0          16h\r\nkube-dns-1842970385-ww5vm              0/3       Pending            0          16h\r\nkube-dns-2943014524-gsskt              0/3       InvalidImageName   0          14m\r\nkubernetes-dashboard-96113950-48pz7    1/1       Running            0          16h\r\nmonitoring-grafana-1854198434-ngjt4    1/1       Running            0          16h\r\nmonitoring-influxdb-2450037842-wwt38   1/1       Running            0          16h\r\ntiller-deploy-3765135481-f9dg8         1/1       Running            0          16h\r\n```\r\nAnd this is what is in the logs in kubectl describe\r\n```\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\t\t\t\t\tSubObjectPath\t\t\tType\t\tReason\t\t\tMessage\r\n  ---------\t--------\t-----\t----\t\t\t\t\t\t\t-------------\t\t\t--------\t------\t\t\t-------\r\n  14m\t\t14m\t\t6\tdefault-scheduler\t\t\t\t\t\t\t\t\tWarning\t\tFailedScheduling\tNo nodes are available that match all of the following predicates:: MatchInterPodAffinity (1).\r\n  14m\t\t14m\t\t1\tdefault-scheduler\t\t\t\t\t\t\t\t\tNormal\t\tScheduled\t\tSuccessfully assigned kube-dns-2943014524-gsskt to ip-10-0-0-160.us-west-1.compute.internal\r\n  14m\t\t14m\t\t1\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\t\t\t\t\tNormal\t\tSuccessfulMountVolume\tMountVolume.SetUp succeeded for volume \"kube-dns-config\"\r\n  14m\t\t14m\t\t1\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\t\t\t\t\tNormal\t\tSuccessfulMountVolume\tMountVolume.SetUp succeeded for volume \"kube-dns-token-p41d6\"\r\n  14m\t\t13m\t\t3\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\t\t\t\t\tNormal\t\tSandboxChanged\t\tPod sandbox changed, it will be killed and re-created.\r\n  14m\t\t4s\t\t68\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\t\t\t\t\tWarning\t\tFailedSync\t\tError syncing pod\r\n  13m\t\t4s\t\t65\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\tspec.containers{kubedns}\tWarning\t\tInspectFailed\t\tFailed to apply default image tag \"ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-kube-dns-amd64:1.14.2\": couldn't parse image reference \"ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-kube-dns-amd64:1.14.2\": invalid reference format\r\n  13m\t\t4s\t\t65\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\tspec.containers{dnsmasq}\tWarning\t\tInspectFailed\t\tFailed to apply default image tag \"ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2\": couldn't parse image reference \"ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2\": invalid reference format\r\n  13m\t\t4s\t\t65\tkubelet, ip-10-0-0-160.us-west-1.compute.internal\tspec.containers{sidecar}\tWarning\t\tInspectFailed\t\tFailed to apply default image tag \"ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-sidecar-amd64:1.14.2\": couldn't parse image reference \"ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-sidecar-amd64:1.14.2\": invalid reference format\r\n```\r\n\r\n\r\n**Results:**\r\nThe issue is the code seems to be adding additional trailing slash ip-10-0-0-160:5000/gcr.io//google_containers/k8s-dns-kube-dns-amd64:1.14.2\"\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/343380146","html_url":"https://github.com/rancher/rancher/issues/9882#issuecomment-343380146","issue_url":"https://api.github.com/repos/rancher/rancher/issues/9882","id":343380146,"user":{"login":"anjia0532","id":15098916,"avatar_url":"https://avatars1.githubusercontent.com/u/15098916?v=4","gravatar_id":"","url":"https://api.github.com/users/anjia0532","html_url":"https://github.com/anjia0532","followers_url":"https://api.github.com/users/anjia0532/followers","following_url":"https://api.github.com/users/anjia0532/following{/other_user}","gists_url":"https://api.github.com/users/anjia0532/gists{/gist_id}","starred_url":"https://api.github.com/users/anjia0532/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anjia0532/subscriptions","organizations_url":"https://api.github.com/users/anjia0532/orgs","repos_url":"https://api.github.com/users/anjia0532/repos","events_url":"https://api.github.com/users/anjia0532/events{/privacy}","received_events_url":"https://api.github.com/users/anjia0532/received_events","type":"User","site_admin":false},"created_at":"2017-11-10T05:41:45Z","updated_at":"2017-11-10T05:41:45Z","author_association":"NONE","body":"hei , in the china ,i can not access gcr.io.\r\n\r\nso i use new york vps pull the gcr.io image and push to docker hub.\r\n\r\n\r\n\r\n## get rancher-k8s original version\r\n\r\n// copy to k8s cli and run it to get gcr.io images version\r\n```bash\r\nk8s=(\r\n    heapster\r\n    kube-dns\r\n    kubernetes-dashboard\r\n    monitoring-grafana\r\n    monitoring-influxdb\r\n    tiller-deploy\r\n)\r\nfor imageName in ${k8s[@]} ; do\r\n    for t in $(kubectl --namespace=kube-system describe deployment $imageName | grep gcr | awk '{print $2}') ; do\r\n        echo $t\r\n    done\r\ndone\r\n```\r\n// output ,copy it\r\n```\r\ngcr.io/google_containers/heapster-amd64:v1.3.0-beta.1\r\ngcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5\r\ngcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5\r\ngcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5\r\ngcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1\r\ngcr.io/google_containers/heapster-grafana-amd64:v4.0.2\r\ngcr.io/google_containers/heapster-influxdb-amd64:v1.3.3\r\ngcr.io/kubernetes-helm/tiller:v2.3.0\r\n```\r\n\r\n\r\n\r\n## pull gcr.io image and push to docker hub\r\n\r\n```bash\r\nhubName=anjia0532\r\n\r\nimages=(\r\n    gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1\r\n    gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5\r\n    gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5\r\n    gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5\r\n    gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1\r\n    gcr.io/google_containers/heapster-grafana-amd64:v4.0.2\r\n    gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3\r\n    gcr.io/kubernetes-helm/tiller:v2.3.0\r\n)\r\n\r\nfor imageName in ${images[@]} ; do\r\n    imgName=$(echo ${imageName} | cut -d\"/\" -f3)\r\n    docker pull $imageName\r\n    docker tag $imageName $hubName/$imgName\r\n    docker push $hubName/$imgName\r\ndone\r\n```\r\n\r\n## change rancher-k8s namespace=kube-system's res image source\r\n\r\n```bash\r\nkubectl --namespace=kube-system edit deployment  heapster\r\nkubectl --namespace=kube-system edit deployment  kube-dns\r\nkubectl --namespace=kube-system edit deployment  kubernetes-dashboard\r\nkubectl --namespace=kube-system edit deployment  monitoring-grafana\r\nkubectl --namespace=kube-system edit deployment  monitoring-influxdb\r\n\r\n:%s#gcr.io/google_containers#anjia0532#g\r\n:wq!\r\n\r\nkubectl --namespace=kube-system edit deployment  tiller-deploy\r\n\r\n:%s#gcr.io/kubernetes-helm#anjia0532#g\r\n:wq!\r\n```\r\n\r\n## but, rancher-k8s is keep pending\r\n\r\n```bash\r\n$ kubectl --namespace=kube-system  get pods\r\nNAME                                    READY     STATUS              RESTARTS   AGE\r\nheapster-2407085140-hgddj               0/1       ContainerCreating   0          48m\r\nkube-dns-570853077-hcqzg                0/3       Pending             0          1h\r\nkube-dns-638003847-8vps9                0/3       ContainerCreating   0          2h\r\nkubernetes-dashboard-3888044391-wm3s3   0/1       ContainerCreating   0          14m\r\nmonitoring-grafana-3847008717-06988     0/1       ContainerCreating   0          14m\r\nmonitoring-influxdb-3527312529-n3xxw    0/1       ContainerCreating   0          14m\r\ntiller-deploy-402017509-jkw7n           0/1       ContainerCreating   0          13m\r\n\r\n```"}}
{"actor":{"display_login":"anjia0532"}}