{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44798","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44798/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44798/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44798/events","html_url":"https://github.com/kubernetes/kubernetes/pull/44798","id":223562076,"number":44798,"title":"Statefulsets for cinder: allow multi-AZ deployments, spread pods across zones","user":{"login":"zetaab","id":22482503,"avatar_url":"https://avatars1.githubusercontent.com/u/22482503?v=3","gravatar_id":"","url":"https://api.github.com/users/zetaab","html_url":"https://github.com/zetaab","followers_url":"https://api.github.com/users/zetaab/followers","following_url":"https://api.github.com/users/zetaab/following{/other_user}","gists_url":"https://api.github.com/users/zetaab/gists{/gist_id}","starred_url":"https://api.github.com/users/zetaab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zetaab/subscriptions","organizations_url":"https://api.github.com/users/zetaab/orgs","repos_url":"https://api.github.com/users/zetaab/repos","events_url":"https://api.github.com/users/zetaab/events{/privacy}","received_events_url":"https://api.github.com/users/zetaab/received_events","type":"User","site_admin":false},"labels":[{"id":477397086,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":240247879,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/needs-rebase","name":"needs-rebase","color":"BDBDBD","default":false},{"id":200149833,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-note","name":"release-note","color":"c2e0c6","default":false},{"id":239914558,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/openstack","name":"sig/openstack","color":"d2b48c","default":false},{"id":169428334,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false},{"id":253451093,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/XXL","name":"size/XXL","color":"ee0000","default":false}],"state":"open","locked":false,"assignee":{"login":"rootfs","id":7062400,"avatar_url":"https://avatars3.githubusercontent.com/u/7062400?v=3","gravatar_id":"","url":"https://api.github.com/users/rootfs","html_url":"https://github.com/rootfs","followers_url":"https://api.github.com/users/rootfs/followers","following_url":"https://api.github.com/users/rootfs/following{/other_user}","gists_url":"https://api.github.com/users/rootfs/gists{/gist_id}","starred_url":"https://api.github.com/users/rootfs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rootfs/subscriptions","organizations_url":"https://api.github.com/users/rootfs/orgs","repos_url":"https://api.github.com/users/rootfs/repos","events_url":"https://api.github.com/users/rootfs/events{/privacy}","received_events_url":"https://api.github.com/users/rootfs/received_events","type":"User","site_admin":false},"assignees":[{"login":"anguslees","id":224224,"avatar_url":"https://avatars3.githubusercontent.com/u/224224?v=3","gravatar_id":"","url":"https://api.github.com/users/anguslees","html_url":"https://github.com/anguslees","followers_url":"https://api.github.com/users/anguslees/followers","following_url":"https://api.github.com/users/anguslees/following{/other_user}","gists_url":"https://api.github.com/users/anguslees/gists{/gist_id}","starred_url":"https://api.github.com/users/anguslees/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anguslees/subscriptions","organizations_url":"https://api.github.com/users/anguslees/orgs","repos_url":"https://api.github.com/users/anguslees/repos","events_url":"https://api.github.com/users/anguslees/events{/privacy}","received_events_url":"https://api.github.com/users/anguslees/received_events","type":"User","site_admin":false},{"login":"rootfs","id":7062400,"avatar_url":"https://avatars3.githubusercontent.com/u/7062400?v=3","gravatar_id":"","url":"https://api.github.com/users/rootfs","html_url":"https://github.com/rootfs","followers_url":"https://api.github.com/users/rootfs/followers","following_url":"https://api.github.com/users/rootfs/following{/other_user}","gists_url":"https://api.github.com/users/rootfs/gists{/gist_id}","starred_url":"https://api.github.com/users/rootfs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rootfs/subscriptions","organizations_url":"https://api.github.com/users/rootfs/orgs","repos_url":"https://api.github.com/users/rootfs/repos","events_url":"https://api.github.com/users/rootfs/events{/privacy}","received_events_url":"https://api.github.com/users/rootfs/received_events","type":"User","site_admin":false}],"milestone":null,"comments":23,"created_at":"2017-04-22T12:37:17Z","updated_at":"2017-05-01T18:08:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/44798","html_url":"https://github.com/kubernetes/kubernetes/pull/44798","diff_url":"https://github.com/kubernetes/kubernetes/pull/44798.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/44798.patch"},"body":"**What this PR does / why we need it**: Currently if we do not specify availability zone in cinder storageclass, the cinder is provisioned to zone called nova. However, like mentioned in issue, we have situation that we want spread statefulset across 3 different zones. Currently this is not possible with statefulsets and cinder storageclass. In this new solution, if we leave it empty the algorithm will choose the zone for the cinder drive similar style like in aws and gce storageclass solutions. \r\n\r\n**Which issue this PR fixes** fixes #44735\r\n\r\n**Special notes for your reviewer**:\r\n\r\nexample:\r\n\r\n```\r\nkind: StorageClass\r\napiVersion: storage.k8s.io/v1beta1\r\nmetadata:\r\n  name: all\r\nprovisioner: kubernetes.io/cinder\r\n---\r\napiVersion: v1\r\nkind: Service\r\nmetadata:\r\n  annotations:\r\n    service.alpha.kubernetes.io/tolerate-unready-endpoints: \"true\"\r\n  name: galera\r\n  labels:\r\n    app: mysql\r\nspec:\r\n  ports:\r\n  - port: 3306\r\n    name: mysql\r\n  clusterIP: None\r\n  selector:\r\n    app: mysql\r\n---\r\napiVersion: apps/v1beta1\r\nkind: StatefulSet\r\nmetadata:\r\n  name: mysql\r\nspec:\r\n  serviceName: \"galera\"\r\n  replicas: 3\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: mysql\r\n      annotations:\r\n        pod.alpha.kubernetes.io/initialized: \"true\"\r\n    spec:\r\n      containers:\r\n      - name: mysql\r\n        image: adfinissygroup/k8s-mariadb-galera-centos:v002\r\n        imagePullPolicy: Always\r\n        ports:\r\n        - containerPort: 3306\r\n          name: mysql\r\n        - containerPort: 4444\r\n          name: sst\r\n        - containerPort: 4567\r\n          name: replication\r\n        - containerPort: 4568\r\n          name: ist\r\n        volumeMounts:\r\n        - name: storage\r\n          mountPath: /data\r\n        readinessProbe:\r\n          exec:\r\n            command:\r\n            - /usr/share/container-scripts/mysql/readiness-probe.sh\r\n          initialDelaySeconds: 15\r\n          timeoutSeconds: 5\r\n        env:\r\n          - name: POD_NAMESPACE\r\n            valueFrom:\r\n              fieldRef:\r\n                apiVersion: v1\r\n                fieldPath: metadata.namespace\r\n  volumeClaimTemplates:\r\n  - metadata:\r\n      name: storage\r\n      annotations:\r\n        volume.beta.kubernetes.io/storage-class: all\r\n    spec:\r\n      accessModes: [ \"ReadWriteOnce\" ]\r\n      resources:\r\n        requests:\r\n          storage: 12Gi\r\n```\r\n\r\nIf this example is deployed it will automatically create one replica per AZ. This helps us a lot making HA databases.\r\n\r\nCurrent storageclass for cinder is not perfect in case of statefulsets. Lets assume that cinder storageclass is defined to be in zone called nova, but because labels are not added to pv - pods can be started in any zone. The problem is that at least in our openstack it is not possible to use cinder drive located in zone x from zone y. However, should we have possibility to choose between cross-zone cinder mounts or not? Imo it is not good way of doing things that they mount volume from another zone where the pod is located(means more network traffic between zones)? What you think? Current new solution does not allow that anymore (should we have possibility to allow it? it means removing the labels from pv).\r\n\r\nThere might be some things that needs to be fixed still in this release and I need help for that. Some parts of the code is not perfect.\r\n\r\nIssues what i am thinking about (I need some help for these):\r\n1) Can everybody see in openstack what AZ their servers are? Can there be like access policy that do not show that? If AZ is not found from server specs, I have no idea how the code behaves. \r\n2) In GetAllZones() function, is it really needed to make new serviceclient using openstack.NewComputeV2 or could I somehow use existing one\r\n3) This fetches all servers from some openstack tenant(project). However, in some cases kubernetes is maybe deployed only to specific zone. If kube servers are located for instance in zone 1, and then there are another servers in same tenant in zone 2. There might be usecase that cinder drive is provisioned to zone-2 but it cannot start pod, because kubernetes does not have any nodes in zone-2. Could we have better way to fetch kubernetes nodes zones? Currently that information is not added to kubernetes node labels automatically in openstack (which should I think). I have added those labels manually to nodes. If that zone information is not added to nodes, the new solution does not start stateful pods at all, because it cannot target pods.\r\n\r\n\r\ncc @rootfs @anguslees @jsafrane \r\n\r\n```release-note\r\nDefault behaviour in cinder storageclass is changed. If availability is not specified, the zone is chosen by algorithm. It makes possible to spread stateful pods across many zones.\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/298390501","html_url":"https://github.com/kubernetes/kubernetes/pull/44798#issuecomment-298390501","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44798","id":298390501,"user":{"login":"k8s-ci-robot","id":20407524,"avatar_url":"https://avatars3.githubusercontent.com/u/20407524?v=3","gravatar_id":"","url":"https://api.github.com/users/k8s-ci-robot","html_url":"https://github.com/k8s-ci-robot","followers_url":"https://api.github.com/users/k8s-ci-robot/followers","following_url":"https://api.github.com/users/k8s-ci-robot/following{/other_user}","gists_url":"https://api.github.com/users/k8s-ci-robot/gists{/gist_id}","starred_url":"https://api.github.com/users/k8s-ci-robot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k8s-ci-robot/subscriptions","organizations_url":"https://api.github.com/users/k8s-ci-robot/orgs","repos_url":"https://api.github.com/users/k8s-ci-robot/repos","events_url":"https://api.github.com/users/k8s-ci-robot/events{/privacy}","received_events_url":"https://api.github.com/users/k8s-ci-robot/received_events","type":"User","site_admin":false},"created_at":"2017-05-01T18:08:59Z","updated_at":"2017-05-01T18:08:59Z","body":"@zetaab: The following test(s) **failed**:\n\nTest name | Commit | Details | Rerun command\n--- | --- | --- | ---\nJenkins Bazel Build | 6cf4c16b258772bd7592cb2aa7a19d803219c5e9 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-bazel/15458/) | `@k8s-bot bazel test this`\nJenkins Kubemark GCE e2e | 6cf4c16b258772bd7592cb2aa7a19d803219c5e9 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-kubemark-e2e-gce/27710/) | `@k8s-bot kubemark e2e test this`\nJenkins GCE etcd3 e2e | 6cf4c16b258772bd7592cb2aa7a19d803219c5e9 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-e2e-gce-etcd3/28000/) | `@k8s-bot gce etcd3 e2e test this`\nJenkins kops AWS e2e | 6cf4c16b258772bd7592cb2aa7a19d803219c5e9 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-e2e-kops-aws/22806/) | `@k8s-bot kops aws e2e test this`\nJenkins GCE Node e2e | 6cf4c16b258772bd7592cb2aa7a19d803219c5e9 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-node-e2e/27822/) | `@k8s-bot node e2e test this`\nJenkins verification | 6cf4c16b258772bd7592cb2aa7a19d803219c5e9 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-verify/27711/) | `@k8s-bot verify test this`\npull-kubernetes-cross | 5315fd48d3d309537a9da6cdf63d0a2d515e6956 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-cross/2375/) | `@k8s-bot pull-kubernetes-cross test this`\nJenkins unit/integration | 5315fd48d3d309537a9da6cdf63d0a2d515e6956 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44798/pull-kubernetes-unit/28508/) | `@k8s-bot unit test this`\n\n[Full PR test history](https://k8s-gubernator.appspot.com/pr/44798). [Your PR dashboard](https://k8s-gubernator.appspot.com/pr/zetaab). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/kubernetes/issues?q=is:issue+is:open) when you hit one in your PR.\n\n<details>\n\nInstructions for interacting with me using PR comments are available [here](https://github.com/kubernetes/community/blob/master/contributors/devel/pull-request-commands.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://github.com/kubernetes/test-infra/blob/master/commands.md).\n</details>\n<!-- test report -->"}}
{"actor":{"display_login":"k8s-ci-robot"}}