{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44372","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44372/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44372/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44372/events","html_url":"https://github.com/kubernetes/kubernetes/issues/44372","id":221121634,"number":44372,"title":"Kubernetes deleted volume unexpectedly while leaving PV/PVC state unchanged","user":{"login":"zhan849","id":3003026,"avatar_url":"https://avatars1.githubusercontent.com/u/3003026?v=3","gravatar_id":"","url":"https://api.github.com/users/zhan849","html_url":"https://github.com/zhan849","followers_url":"https://api.github.com/users/zhan849/followers","following_url":"https://api.github.com/users/zhan849/following{/other_user}","gists_url":"https://api.github.com/users/zhan849/gists{/gist_id}","starred_url":"https://api.github.com/users/zhan849/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhan849/subscriptions","organizations_url":"https://api.github.com/users/zhan849/orgs","repos_url":"https://api.github.com/users/zhan849/repos","events_url":"https://api.github.com/users/zhan849/events{/privacy}","received_events_url":"https://api.github.com/users/zhan849/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-04-12T00:57:25Z","updated_at":"2017-04-18T17:00:37Z","closed_at":null,"body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a request for help?** NO\r\n\r\n\r\n**What keywords did you search in Kubernetes issues before filing this one?** (If you have found any duplicates, you should instead reply there.):\r\n\r\nKubernetes volume deleted unexpectedly. Kubernetes volume deleted after apiserver panic\r\n---\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** (choose one):\r\nBUG REPORT\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Kubernetes version** (use `kubectl version`):\r\n```\r\nroot@2b1dfb511ac9:~# kubectl version                 \r\nClient Version: version.Info{Major:\"1\", Minor:\"4+\", GitVersion:\"v1.4.3-39+397fdc3990005b\", GitCommit:\"397fdc3990005b3823a020d001de4b8381efff89\", GitTreeState:\"clean\", BuildDate:\"2017-03-16T22:22:25Z\", GoVersion:\"go1.6.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"4+\", GitVersion:\"v1.4.3-39+397fdc3990005b\", GitCommit:\"397fdc3990005b3823a020d001de4b8381efff89\", GitTreeState:\"clean\", BuildDate:\"2017-03-16T22:13:52Z\", GoVersion:\"go1.6.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration**: aws m3.large instance\r\n- **OS** (e.g. from /etc/os-release): \r\n```\r\nroot@ip-10-128-0-9:/home/admin# cat /etc/os-release\r\nPRETTY_NAME=\"Debian GNU/Linux 8 (jessie)\"\r\nNAME=\"Debian GNU/Linux\"\r\nVERSION_ID=\"8\"\r\nVERSION=\"8 (jessie)\"\r\nID=debian\r\nHOME_URL=\"http://www.debian.org/\"\r\nSUPPORT_URL=\"http://www.debian.org/support\"\r\nBUG_REPORT_URL=\"https://bugs.debian.org/\"\r\n```\r\n- **Kernel** (e.g. `uname -a`): ```Linux ip-10-128-0-9 4.4.41-k8s #1 SMP Mon Jan 9 15:34:39 UTC 2017 x86_64 GNU/Linux```\r\n- **Install tools**: kube-up (For node setup only)\r\n- **Others**:\r\n\r\n\r\n**What happened**:\r\nkubernetes controller manager deleted aws ebs volume unintentionally (no one issued a DELETE api call), while leaving PV/PVC state unchanged. Before deleting volume, api server had couple of go routines crashed due to timeout\r\n\r\n**What you expected to happen**:\r\n1. EBS volume should not be unintentionally deleted\r\n2. Even EBS volume is deleted, PV/PVC status should reflect it\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\nCannot reproduce it stably.\r\n\r\n**Anything else we need to know**:\r\nHere are some details:\r\n\r\n\r\n\r\nPV status (After volume is deleted)\r\n```\r\nroot@2b1dfb511ac9:~# kubectl --kubeconfig /tmp/ax_kube/cluster_hz-mac-ttt93-b1c768d0-1f09-11e7-b6c5-0242ac110007.conf --namespace axsys get pv pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23 -o yaml \r\napiVersion: v1\r\nkind: PersistentVolume\r\nmetadata:\r\n  annotations:\r\n    kubernetes.io/createdby: aws-ebs-dynamic-provisioner\r\n    pv.kubernetes.io/bound-by-controller: \"yes\"\r\n    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs\r\n  creationTimestamp: 2017-04-11T23:03:50Z\r\n  labels:\r\n    failure-domain.beta.kubernetes.io/region: us-west-2\r\n    failure-domain.beta.kubernetes.io/zone: us-west-2c\r\n  name: pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r\n  resourceVersion: \"425\"\r\n  selfLink: /api/v1/persistentvolumes/pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r\n  uid: 2023c519-1f0b-11e7-bcca-0a92a07a4e23\r\nspec:\r\n  accessModes:\r\n  - ReadWriteOnce\r\n  awsElasticBlockStore:\r\n    fsType: ext4\r\n    volumeID: aws://us-west-2c/vol-0b3bc6fd027da7cf4\r\n  capacity:\r\n    storage: 30Gi\r\n  claimRef:\r\n    apiVersion: v1\r\n    kind: PersistentVolumeClaim\r\n    name: gateway-persistent-storage\r\n    namespace: axsys\r\n    resourceVersion: \"241\"\r\n    uid: e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r\n  persistentVolumeReclaimPolicy: Delete\r\nstatus:\r\n  phase: Bound\r\n```\r\n\r\n\r\n\r\nPVC status (after volume is deleted):\r\n```\r\nroot@2b1dfb511ac9:~# kubectl --kubeconfig /tmp/ax_kube/cluster_hz-mac-ttt93-b1c768d0-1f09-11e7-b6c5-0242ac110007.conf --namespace axsys get pvc gateway-persistent-storage -o yaml             \r\napiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n  annotations:\r\n    pv.kubernetes.io/bind-completed: \"yes\"\r\n    pv.kubernetes.io/bound-by-controller: \"yes\"\r\n    volume.alpha.kubernetes.io/storage-class: anything\r\n  creationTimestamp: 2017-04-11T23:02:08Z\r\n  labels:\r\n    app: gateway\r\n  name: gateway-persistent-storage\r\n  namespace: axsys\r\n  resourceVersion: \"427\"\r\n  selfLink: /api/v1/namespaces/axsys/persistentvolumeclaims/gateway-persistent-storage\r\n  uid: e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r\nspec:\r\n  accessModes:\r\n  - ReadWriteOnce\r\n  resources:\r\n    requests:\r\n      storage: 30Gi\r\n  volumeName: pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r\nstatus:\r\n  accessModes:\r\n  - ReadWriteOnce\r\n  capacity:\r\n    storage: 30Gi\r\n  phase: Bound\r\n```\r\n\r\nkube-controller-manager log about this volume (as well as pv, pvc)\r\n```\r\nI0411 23:02:08.346500       5 util.go:326] Creating volume for PVC \"gateway-persistent-storage\"; chose zone=\"us-west-2c\" from zones=[\"us-west-2c\"]\r\nI0411 23:02:08.382722       5 aws.go:1077] Found instances in zones map[us-west-2c:{}]\r\n......\r\nI0411 23:02:08.631235       5 aws_util.go:123] Successfully created EBS Disk volume aws://us-west-2c/vol-0b3bc6fd027da7cf4\r\n......\r\nI0411 23:03:50.249778       5 controller.go:669] volume \"pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\" entered phase \"Bound\"\r\nI0411 23:03:50.249827       5 controller.go:805] volume \"pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\" bound to claim \"axsys/gateway-persistent-storage\"\r\n......\r\nI0411 23:04:54.685169       5 aws_util.go:56] Successfully deleted EBS Disk volume aws://us-west-2c/vol-0b3bc6fd027da7cf4\r\nI0411 23:04:54.685197       5 controller.go:1179] volume \"pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\" deleted\r\n```\r\n\r\nkube-apiserver panics\r\n```\r\nE0411 23:03:53.202143       5 apiserver.go:516] apiserver was unable to write a JSON response: http: Handler timeout\r\nE0411 23:03:53.202196       5 errors.go:63] apiserver received an error that is not an unversioned.Status: http: Handler timeout\r\nI0411 23:03:53.202292       5 trace.go:61] Trace \"Create /api/v1/persistentvolumes\" (started 2017-04-11 23:02:08.611031602 +0000 UTC):\r\n[27.501µs] [27.501µs] About to convert to expected version\r\n[141.219µs] [113.718µs] Conversion done\r\n[1m44.589453767s] [1m44.589312548s] About to store object in database\r\n[1m44.591236532s] [1.782765ms] END\r\nI0411 23:03:53.202352       5 handlers.go:162] POST /api/v1/persistentvolumes: (1m44.591426253s) 500\r\ngoroutine 5551 [running]:\r\nk8s.io/kubernetes/pkg/httplog.(*respLogger).recordStatus(0xc82398eaf0, 0x1f4)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xa3\r\nk8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc82398eaf0, 0x1f4)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x2b\r\nk8s.io/kubernetes/pkg/apiserver/metrics.(*responseWriterDelegator).WriteHeader(0xc823ad8ff0, 0x1f4)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x4c\r\nk8s.io/kubernetes/pkg/apiserver.errorJSONFatal(0x0, 0x0, 0x7f14ea7b3460, 0xc822d04300, 0x7f14ea5b7d50, 0xc820d1cab0, 0xc820010ae0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:526 +0x47a\r\nk8s.io/kubernetes/pkg/apiserver.writeNegotiated(0x7f14ea7b33e8, 0xc820564f50, 0x0, 0x0, 0x3f1ae88, 0x2, 0x7f14ea5b7d50, 0xc820d1cab0, 0xc822b4eee0, 0x199, ...)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:496 +0x3d4\r\nk8s.io/kubernetes/pkg/apiserver.errorNegotiated(0x7f14ea608528, 0xc822d6b780, 0x7f14ea7b33e8, 0xc820564f50, 0x0, 0x0, 0x3f1ae88, 0x2, 0x7f14ea5b7d50, 0xc820d1cab0, ...)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:509 +0x1a9\r\nk8s.io/kubernetes/pkg/apiserver.(*RequestScope).err(0xc8200afc00, 0x7f14ea608528, 0xc822d6b780, 0x7f14ea5b7d50, 0xc820d1cab0, 0xc822b4eee0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:87 +0x191\r\nk8s.io/kubernetes/pkg/apiserver.createHandler.func1(0xc823ad8f60, 0xc8239abaa0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:410 +0x14cd\r\nk8s.io/kubernetes/pkg/apiserver/metrics.InstrumentRouteFunc.func1(0xc823ad8f60, 0xc8239abaa0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x25f\r\nk8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc820531200, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xf30\r\nk8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x3e\r\nnet/http.HandlerFunc.ServeHTTP(0xc820ca57d0, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /usr/local/go/src/net/http/server.go:1618 +0x3a\r\nnet/http.(*ServeMux).ServeHTTP(0xc8200336e0, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /usr/local/go/src/net/http/server.go:1910 +0x17d\r\nk8s.io/kubernetes/pkg/api.NewRequestContextFilter.func1(0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r\nnet/http.HandlerFunc.ServeHTTP(0xc8208296c0, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /usr/local/go/src/net/http/server.go:1618 +0x3a\r\nk8s.io/kubernetes/pkg/apiserver.RecoverPanics.func1(0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:161 +0x1f3\r\nnet/http.HandlerFunc.ServeHTTP(0xc821c08460, 0x7f14ea5b7988, 0xc820d1ca98, 0xc822b4eee0)\r\n        /usr/local/go/src/net/http/server.go:1618 +0x3a\r\nk8s.io/kubernetes/pkg/apiserver.(*timeoutHandler).ServeHTTP.func1(0xc821c084a0, 0x7f14ea5b7948, 0xc820d1ca98, 0xc822b4eee0, 0xc8239ab9e0)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:194 +0x8d\r\ncreated by k8s.io/kubernetes/pkg/apiserver.(*timeoutHandler).ServeHTTP\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:196 +0x15b\r\n\r\nlogging error output: \"k8s\\x00\\n\\f\\n\\x02v1\\x12\\x06Status\\x12+\\n\\x04\\n\\x00\\x12\\x00\\x12\\aFailure\\x1a\\x15http: Handler timeout\\\"\\x000\\xf4\\x03\\x1a\\x00\\\"\\x00\"\r\n [[kube-controller-manager/v1.4.3 (linux/amd64) kubernetes/397fdc3/persistent-volume-binder] 127.0.0.1:47348]\r\n\r\n(The above happened a couple of times before)\r\n\r\n......\r\n\r\nI0411 23:04:27.534961       5 trace.go:61] Trace \"Create /api/v1/persistentvolumes\" (started 2017-04-11 23:04:19.443869726 +0000 UTC):\r\n[25.663µs] [25.663µs] About to convert to expected version\r\n[132.76µs] [107.097µs] Conversion done\r\n[8.08948191s] [8.08934915s] About to store object in database\r\n[8.091061632s] [1.579722ms] END\r\nI0411 23:04:27.535009       5 handlers.go:162] POST /api/v1/persistentvolumes: (8.091446476s) 409 [[kube-controller-manager/v1.4.3 (linux/amd64) kubernetes/397fdc3/persistent-volume-binder] 127.0.0.1:48064]\r\n\r\n......\r\n\r\nI0411 23:04:44.578049       5 trace.go:61] Trace \"Create /api/v1/persistentvolumes\" (started 2017-04-11 23:04:37.536507929 +0000 UTC):\r\n[28.013µs] [28.013µs] About to convert to expected version\r\n[154.613µs] [126.6µs] Conversion done\r\n[7.03989939s] [7.039744777s] About to store object in database\r\n[7.041511925s] [1.612535ms] END\r\nI0411 23:04:44.578299       5 handlers.go:162] POST /api/v1/persistentvolumes: (7.041757553s) 409 [[kube-controller-manager/v1.4.3 (linux/amd64) kubernetes/397fdc3/persistent-volume-binder] 127.0.0.1:48076]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/294910901","html_url":"https://github.com/kubernetes/kubernetes/issues/44372#issuecomment-294910901","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44372","id":294910901,"user":{"login":"zhan849","id":3003026,"avatar_url":"https://avatars1.githubusercontent.com/u/3003026?v=3","gravatar_id":"","url":"https://api.github.com/users/zhan849","html_url":"https://github.com/zhan849","followers_url":"https://api.github.com/users/zhan849/followers","following_url":"https://api.github.com/users/zhan849/following{/other_user}","gists_url":"https://api.github.com/users/zhan849/gists{/gist_id}","starred_url":"https://api.github.com/users/zhan849/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhan849/subscriptions","organizations_url":"https://api.github.com/users/zhan849/orgs","repos_url":"https://api.github.com/users/zhan849/repos","events_url":"https://api.github.com/users/zhan849/events{/privacy}","received_events_url":"https://api.github.com/users/zhan849/received_events","type":"User","site_admin":false},"created_at":"2017-04-18T17:00:36Z","updated_at":"2017-04-18T17:00:36Z","body":"@jsafrane thanks for the detailed explanation. It might because we are turning `--v=2` so there is little to see. Will use `kubectl describe pvc` to get more information if it happened again. But its weird why API server go routine can crash, as there is no other intervention such as host level OOMKill.\r\n\r\nBtw is this slim possibility fixed/improved in later version? Any pull request / bug I can refer to?"}}
{"actor":{"display_login":"zhan849"}}