{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/691","repository_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws","labels_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/691/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/691/comments","events_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/691/events","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/691","id":233225552,"number":691,"title":"ETCd crash after few days","user":{"login":"Vrtak-CZ","id":112567,"avatar_url":"https://avatars2.githubusercontent.com/u/112567?v=3","gravatar_id":"","url":"https://api.github.com/users/Vrtak-CZ","html_url":"https://github.com/Vrtak-CZ","followers_url":"https://api.github.com/users/Vrtak-CZ/followers","following_url":"https://api.github.com/users/Vrtak-CZ/following{/other_user}","gists_url":"https://api.github.com/users/Vrtak-CZ/gists{/gist_id}","starred_url":"https://api.github.com/users/Vrtak-CZ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vrtak-CZ/subscriptions","organizations_url":"https://api.github.com/users/Vrtak-CZ/orgs","repos_url":"https://api.github.com/users/Vrtak-CZ/repos","events_url":"https://api.github.com/users/Vrtak-CZ/events{/privacy}","received_events_url":"https://api.github.com/users/Vrtak-CZ/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-06-02T15:49:36Z","updated_at":"2017-06-07T13:43:46Z","closed_at":null,"body":"Hello,\r\n\r\nwe experience crash 2 of 3 etcd servers see logs bellow. But what is really bad we have no access to whole k8s cluster due etcd errors (timeout).\r\n\r\n```\r\ncore@ip-10-101-3-236 ~ $ journalctl -u etcdadm-check.service -f\r\n\r\n-- Logs begin at Wed 2017-05-31 22:08:13 UTC. --\r\nJun 02 15:38:09 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: etcdadm-check.service: Unit entered failed state.\r\nJun 02 15:38:09 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: etcdadm-check.service: Failed with result 'timeout'.\r\nJun 02 15:38:21 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: Starting etcd health check...\r\nJun 02 15:38:24 ip-10-101-3-236.eu-west-1.compute.internal systemctl[11792]: active\r\nJun 02 15:38:27 ip-10-101-3-236.eu-west-1.compute.internal etcdadm[11806]: declare -x ETCDCTL_CACERT=\"/etc/ssl/certs/ca.pem\"\r\nJun 02 15:38:27 ip-10-101-3-236.eu-west-1.compute.internal etcdadm[11806]: declare -x ETCDCTL_CA_FILE=\"/etc/ssl/certs/ca.pem\"\r\nJun 02 15:38:27 ip-10-101-3-236.eu-west-1.compute.internal etcdadm[11806]: declare -x ETCDCTL_CERT=\"/etc/ssl/certs/etcd-client.pem\"\r\nJun 02 15:38:27 ip-10-101-3-236.eu-west-1.compute.internal etcdadm[11806]: declare -x ETCDCTL_CERT_FILE=\"/etc/ssl/certs/etcd-client.pem\"\r\nJun 02 15:38:27 ip-10-101-3-236.eu-west-1.compute.internal etcdadm[11806]: declare -x ETCDCTL_KEY=\"/etc/ssl/certs/etcd-client-key.pem\"\r\nJun 02 15:38:27 ip-10-101-3-236.eu-west-1.compute.internal etcdadm[11806]: declare -x ETCDCTL_KEY_FILE=\"/etc/ssl/certs/etcd-client-key.pem\"\r\nJun 02 15:40:24 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: etcdadm-check.service: Start operation timed out. Terminating.\r\nJun 02 15:40:25 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: etcdadm-check.service: Main process exited, code=killed, status=15/TERM\r\nJun 02 15:41:25 ip-10-101-3-236.eu-west-1.compute.internal systemctl[11871]: active\r\nJun 02 15:40:28 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: Failed to start etcd health check.\r\nJun 02 15:40:28 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: etcdadm-check.service: Unit entered failed state.\r\nJun 02 15:40:28 ip-10-101-3-236.eu-west-1.compute.internal systemd[1]: etcdadm-check.service: Failed with result 'timeout'.\r\n```\r\n\r\nrelevant configuration:\r\n\r\n```\r\netcd:\r\n  count: 3\r\n  instanceType: t2.medium\r\n  subnets:\r\n    - name: ManagementPrivateSubnet1\r\n    - name: ManagementPrivateSubnet2\r\n    - name: ManagementPrivateSubnet3\r\n\r\n  snapshot:\r\n    automated: true\r\n  disasterRecovery:\r\n    automated: true\r\n\r\n  kmsKeyArn: 'arn:aws:kms:eu-west-1:xxxxxxxxxxxx:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\r\n  dataVolume:\r\n    encrypted: true\r\n\r\n  memberIdentityProvider: eni\r\n```\r\n\r\nused version `kube-aws version v0.9.7-rc.2`\r\n\r\nplease ping me if you want more information."},"comment":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/comments/306798818","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/691#issuecomment-306798818","issue_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/691","id":306798818,"user":{"login":"Vrtak-CZ","id":112567,"avatar_url":"https://avatars2.githubusercontent.com/u/112567?v=3","gravatar_id":"","url":"https://api.github.com/users/Vrtak-CZ","html_url":"https://github.com/Vrtak-CZ","followers_url":"https://api.github.com/users/Vrtak-CZ/followers","following_url":"https://api.github.com/users/Vrtak-CZ/following{/other_user}","gists_url":"https://api.github.com/users/Vrtak-CZ/gists{/gist_id}","starred_url":"https://api.github.com/users/Vrtak-CZ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vrtak-CZ/subscriptions","organizations_url":"https://api.github.com/users/Vrtak-CZ/orgs","repos_url":"https://api.github.com/users/Vrtak-CZ/repos","events_url":"https://api.github.com/users/Vrtak-CZ/events{/privacy}","received_events_url":"https://api.github.com/users/Vrtak-CZ/received_events","type":"User","site_admin":false},"created_at":"2017-06-07T13:43:46Z","updated_at":"2017-06-07T13:43:46Z","body":"Just little update for you. We switched our Etcd instances to `m3.medium` on Friday (2017-06-02). And this problem is back now. When I look in to monitoring ...\r\n\r\n- Etcd-1 has 1.5% CPU Utilization\r\n- Etcd-2 has 100% CPU Utilization\r\n- Etcd-3 has 100% CPU Utilization\r\n\r\nThere is CPU Utilization graph for Etcd-2\r\n\r\n<img width=\"915\" alt=\"screen shot 2017-06-07 at 15 34 02\" src=\"https://user-images.githubusercontent.com/112567/26881225-1bfebc08-4b97-11e7-9b80-0ee0c4f3cd50.png\">\r\n\r\nWe don't make any change in this cluster from the moment when we switched to m3.medium.\r\n\r\nOn this cluster we have 2 worker nodes with 2x3pods (with 2 services and 2 ingress).\r\n\r\nThere is screenshot of TOP from Etcd-3\r\n\r\n<img width=\"1131\" alt=\"screen shot 2017-06-07 at 15 39 06\" src=\"https://user-images.githubusercontent.com/112567/26881550-0b11f71a-4b98-11e7-9336-89d2c78774f3.png\">\r\n\r\n"}}
{"actor":{"display_login":"Vrtak-CZ"}}