{"action":"opened","issue":{"url":"https://api.github.com/repos/openebs/openebs/issues/186","repository_url":"https://api.github.com/repos/openebs/openebs","labels_url":"https://api.github.com/repos/openebs/openebs/issues/186/labels{/name}","comments_url":"https://api.github.com/repos/openebs/openebs/issues/186/comments","events_url":"https://api.github.com/repos/openebs/openebs/issues/186/events","html_url":"https://github.com/openebs/openebs/issues/186","id":248690296,"number":186,"title":"Percona pod crashes during demonstration of HA","user":{"login":"AmitKumarDas","id":2794362,"avatar_url":"https://avatars2.githubusercontent.com/u/2794362?v=4","gravatar_id":"","url":"https://api.github.com/users/AmitKumarDas","html_url":"https://github.com/AmitKumarDas","followers_url":"https://api.github.com/users/AmitKumarDas/followers","following_url":"https://api.github.com/users/AmitKumarDas/following{/other_user}","gists_url":"https://api.github.com/users/AmitKumarDas/gists{/gist_id}","starred_url":"https://api.github.com/users/AmitKumarDas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmitKumarDas/subscriptions","organizations_url":"https://api.github.com/users/AmitKumarDas/orgs","repos_url":"https://api.github.com/users/AmitKumarDas/repos","events_url":"https://api.github.com/users/AmitKumarDas/events{/privacy}","received_events_url":"https://api.github.com/users/AmitKumarDas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-08-08T11:54:39Z","updated_at":"2017-08-08T11:54:39Z","closed_at":null,"body":"#### Setup:\r\n\r\n- K8s 1.7.0\r\n- 1 K8s master & 3 minions\r\n- Percona app was created (ref - https://github.com/openebs/openebs/tree/master/k8s/demo/percona)\r\n- One of the minions holding above jiva replica was shutdown\r\n- Volume goes into `read only` state\r\n  - confirmed after looking into jiva controller logs\r\n- Percona app entered into `CrashLoopBackOff` state\r\n\r\n#### Workaround:\r\n\r\n- logout of iscsi session & login back\r\n- mount the iscsi volume to percona mount path\r\n\r\n#### Logs\r\n\r\n```bash\r\nubuntu@kubemaster-01:/vagrant/demo/percona$ kubectl get pod\r\nNAME                                                            READY     STATUS             RESTARTS   AGE\r\nmaya-apiserver-3516355633-m0hpl                                 1/1       Running            0          52m\r\nopenebs-provisioner-3862123968-cx1vs                            1/1       Running            1          6h\r\npercona                                                         0/1       CrashLoopBackOff   15         1h\r\npvc-cc2824ea-7c22-11e7-8562-021c6f7dbe9d-ctrl-339357979-2fb9c   1/1       Running            0          1h\r\npvc-cc2824ea-7c22-11e7-8562-021c6f7dbe9d-rep-32159078-43s3b     1/1       Running            0          1h\r\npvc-cc2824ea-7c22-11e7-8562-021c6f7dbe9d-rep-32159078-d5zkw     1/1       Running            1          1h\r\n```\r\n\r\n-  kubectl logs pvc-cc2824ea-7c22-11e7-8562-021c6f7dbe9d-ctrl-339357979-2fb9c\r\n\r\n```\r\n10.36.0.0 - - [08/Aug/2017:10:22:17 +0000] \"GET /v1/stats HTTP/1.1\" 200 376\r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Error reading from wire: EOF\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Setting replica tcp://10.36.0.1:9502 to ERR due to: EOF\" \r\ntime=\"2017-08-08T10:22:18Z\" level=info msg=\"Set replica tcp://10.36.0.1:9502 to mode ERR\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Ignoring error because tcp://10.44.0.3:9502 is mode RW: tcp://10.36.0.1:9502: EOF\" \r\ntime=\"2017-08-08T10:22:18Z\" level=info msg=\"Removing backend: tcp://10.36.0.1:9502\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"<nil>\" \r\ntime=\"2017-08-08T10:22:18Z\" level=info msg=\"Monitoring stopped tcp://10.36.0.1:9502\" \r\ntime=\"2017-08-08T10:22:18Z\" level=info msg=\"Closing: 10.36.0.1:9502\" \r\n[0 18 112 0 4 0 0 0 0 10 0 0 0 0 68 0 0 0 0 0]\r\nSENDS SENSE LENGTH =  20\r\n[0 18 112 0 4 0 0 0 0 10 0 0 0 0 68 0 0 0 0 0]\r\n[0 18 112 0 4 0 0 0 0 10 0 0 0 0 68 0 0 0 0 0]\r\nSENDS SENSE LENGTH =  20\r\n[0 18 112 0 4 0 0 0 0 10 0 0 0 0 68 0 0 0 0 0]\r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Mode: ReadOnly\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Mode: ReadOnly\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Mode: ReadOnly\" \r\ntime=\"2017-08-08T10:22:18Z\" level=warning msg=\"check condition\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Mode: ReadOnly\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Mode: ReadOnly\" \r\ntime=\"2017-08-08T10:22:18Z\" level=error msg=\"Mode: ReadOnly\" \r\ntime=\"2017-08-08T10:22:18Z\" level=warning msg=\"check condition\" \r\n```\r\n\r\n"}}
{"actor":{"display_login":"AmitKumarDas"}}