{"action":"closed","issue":{"url":"https://api.github.com/repos/kubernetes-incubator/kubespray/issues/1565","repository_url":"https://api.github.com/repos/kubernetes-incubator/kubespray","labels_url":"https://api.github.com/repos/kubernetes-incubator/kubespray/issues/1565/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes-incubator/kubespray/issues/1565/comments","events_url":"https://api.github.com/repos/kubernetes-incubator/kubespray/issues/1565/events","html_url":"https://github.com/kubernetes-incubator/kubespray/issues/1565","id":252513045,"number":1565,"title":"kube-controller-manager not working on Azure","user":{"login":"saptaktakalkar","id":17902642,"avatar_url":"https://avatars3.githubusercontent.com/u/17902642?v=4","gravatar_id":"","url":"https://api.github.com/users/saptaktakalkar","html_url":"https://github.com/saptaktakalkar","followers_url":"https://api.github.com/users/saptaktakalkar/followers","following_url":"https://api.github.com/users/saptaktakalkar/following{/other_user}","gists_url":"https://api.github.com/users/saptaktakalkar/gists{/gist_id}","starred_url":"https://api.github.com/users/saptaktakalkar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saptaktakalkar/subscriptions","organizations_url":"https://api.github.com/users/saptaktakalkar/orgs","repos_url":"https://api.github.com/users/saptaktakalkar/repos","events_url":"https://api.github.com/users/saptaktakalkar/events{/privacy}","received_events_url":"https://api.github.com/users/saptaktakalkar/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-08-24T07:35:44Z","updated_at":"2017-09-25T07:05:28Z","closed_at":"2017-09-25T07:05:28Z","author_association":"NONE","body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** (choose one): \r\n`BUG REPORT`\r\n\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration:** \r\n`Azure`\r\n\r\n- **OS (`printf \"$(uname -srm)\\n$(cat /etc/os-release)\\n\"`):**\r\n```\r\nNAME=\"Ubuntu\"\r\nVERSION=\"16.04.1 LTS (Xenial Xerus)\"\r\nID=ubuntu\r\nID_LIKE=debian\r\nPRETTY_NAME=\"Ubuntu 16.04.1 LTS\"\r\nVERSION_ID=\"16.04\"\r\nHOME_URL=\"http://www.ubuntu.com/\"\r\nSUPPORT_URL=\"http://help.ubuntu.com/\"\r\nBUG_REPORT_URL=\"http://bugs.launchpad.net/ubuntu/\"\r\nVERSION_CODENAME=xenial\r\nUBUNTU_CODENAME=xenial \r\n```\r\n\r\n- **Version of Ansible** (`ansible --version`): `ansible 2.3.2.0`\r\n\r\n\r\n**Kubespray version (commit) (`git rev-parse --short HEAD`):**  `e91c04f`\r\n\r\n\r\n**Network plugin used**: `calico`\r\n\r\n\r\n**Copy of your inventory file:**\r\n```\r\nbastion ansible_ssh_host=52.176.55.xx ip=30.241.2.4\r\nmaster-0 ansible_ssh_host=30.0.4.4\r\nmaster-1 ansible_ssh_host=30.0.4.5\r\nmaster-2 ansible_ssh_host=30.0.4.6\r\nminion-0 ansible_ssh_host=30.240.0.6\r\nminion-1 ansible_ssh_host=30.240.0.5\r\nminion-2 ansible_ssh_host=30.240.0.4\r\n\r\n[kube-master]\r\nmaster-0\r\nmaster-1\r\nmaster-2\r\n\r\n[etcd]\r\nmaster-0\r\nmaster-1\r\nmaster-2\r\n\r\n[kube-node]\r\nminion-0\r\nminion-1\r\nminion-2\r\n\r\n[k8s-cluster:children]\r\nkube-node\r\nkube-master\r\n```\r\n\r\n**Command used to invoke ansible**:\r\n`ansible-playbook -i contrib/azurerm/inventory -u vagrant --become -e \"@inventory/group_vars/all.yml\" cluster.yml`\r\n\r\n\r\n**Anything else do we need to know**:\r\n<!-- By running scripts/collect-info.yaml you can get a lot of useful informations.\r\nScript can be started by:\r\nansible-playbook -i <inventory_file_path> -u <ssh_user> -e ansible_ssh_user=<ssh_user> -b --become-user=root -e dir=`pwd` scripts/collect-info.yaml\r\n(If you using CoreOS remember to add '-e ansible_python_interpreter=/opt/bin/python').\r\nAfter running this command you can find logs in `pwd`/logs.tar.gz. You can even upload somewhere entire file and paste link here.-->\r\n\r\nI have followed all the steps given at https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/azurerm to create k8s cluster on Azure.\r\n\r\nAfter executing `cluster.yml`, when I log into master-0 and run `kubectl get pods --all-namespaces`, I get the following output:\r\n\r\n```\r\n2017-08-24 07:29:30.636962 I | proto: duplicate proto type registered: google.protobuf.Any\r\n2017-08-24 07:29:30.637037 I | proto: duplicate proto type registered: google.protobuf.Duration\r\n2017-08-24 07:29:30.637056 I | proto: duplicate proto type registered: google.protobuf.Timestamp\r\nNAMESPACE     NAME                               READY     STATUS             RESTARTS   AGE\r\nkube-system   kube-apiserver-master-0            1/1       Running            0          52m\r\nkube-system   kube-apiserver-master-1            1/1       Running            0          52m\r\nkube-system   kube-apiserver-master-2            1/1       Running            0          52m\r\nkube-system   kube-controller-manager-master-0   0/1       CrashLoopBackOff   14         52m\r\nkube-system   kube-controller-manager-master-1   0/1       CrashLoopBackOff   14         52m\r\nkube-system   kube-controller-manager-master-2   1/1       Running            14         52m\r\nkube-system   kube-proxy-master-0                1/1       Running            0          52m\r\nkube-system   kube-proxy-master-1                1/1       Running            0          52m\r\nkube-system   kube-proxy-master-2                1/1       Running            0          52m\r\nkube-system   kube-proxy-minion-0                1/1       Running            0          53m\r\nkube-system   kube-proxy-minion-1                1/1       Running            0          52m\r\nkube-system   kube-proxy-minion-2                1/1       Running            0          52m\r\nkube-system   kube-scheduler-master-0            1/1       Running            0          52m\r\nkube-system   kube-scheduler-master-1            1/1       Running            0          52m\r\nkube-system   kube-scheduler-master-2            1/1       Running            0          52m\r\nkube-system   nginx-proxy-minion-0               1/1       Running            0          52m\r\nkube-system   nginx-proxy-minion-1               1/1       Running            0          52m\r\nkube-system   nginx-proxy-minion-2               1/1       Running            0          52m\r\n```\r\n\r\nAs you can see, STATUS of all `kube-controller-manager-master` nodes is `CrashLoopBackOff` and they are not getting up properly. \r\n\r\nHere are the logs:\r\nCommand: `kubectl logs kube-controller-manager-master-2 -n kube-system`\r\nOutput: \r\n```\r\n2017-08-24 07:32:17.291470 I | proto: duplicate proto type registered: google.protobuf.Any\r\n2017-08-24 07:32:17.291669 I | proto: duplicate proto type registered: google.protobuf.Duration\r\n2017-08-24 07:32:17.291733 I | proto: duplicate proto type registered: google.protobuf.Timestamp\r\n2017-08-24 07:29:22.000005 I | proto: duplicate proto type registered: google.protobuf.Any\r\n2017-08-24 07:29:22.000332 I | proto: duplicate proto type registered: google.protobuf.Duration\r\n2017-08-24 07:29:22.004515 I | proto: duplicate proto type registered: google.protobuf.Timestamp\r\nI0824 07:29:22.489862     158 controllermanager.go:107] Version: v1.7.3+coreos.0\r\nI0824 07:29:22.590421     158 leaderelection.go:179] attempting to acquire leader lease...\r\nI0824 07:29:37.810149     158 leaderelection.go:189] successfully acquired lease kube-system/kube-controller-manager\r\nI0824 07:29:37.812605     158 event.go:218] Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"kube-system\", Name:\"kube-controller-manager\", UID:\"af22bc1d-8896-11e7-90d6-000d3a900fab\", APIVersion:\"v1\", ResourceVersion:\"6790\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' master-2 became leader\r\nF0824 07:29:37.890232     158 controllermanager.go:176] error building controller context: cloud provider could not be initialized: could not init cloud provider \"azure\": AzureDisk -  failed to init Blob Disk Controller with error (azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/xxxx/providers/Microsoft.Storage/storageAccounts?api-version=2016-12-01: StatusCode=0 -- Original Error: adal: Failed to execute the refresh request. Error = 'Post https://login.microsoftonline.com/xxxxxx/oauth2/token?api-version=1.0: x509: failed to load system roots and no roots provided')\r\n``` \r\n\r\nI will appreciate if anyone can help me get all `kube-controller-manager` nodes up and running.\r\n\r\nThank you."}}
{"actor":{"display_login":"mattymo"}}