{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/community/pulls/comments/111833286","pull_request_review_id":33101904,"id":111833286,"diff_hunk":"@@ -0,0 +1,571 @@\n+# Local Storage Management\n+Authors: vishh@, msau42@\n+\n+This document presents a strawman for managing local storage in Kubernetes. We expect to provide a UX and high level design overview for managing most user workflows. More detailed design and implementation will be added once the community agrees with the high level design presented here.\n+\n+# Goals\n+* Enable ephemeral & durable access to local storage\n+* Support storage requirements for all workloads supported by Kubernetes\n+* Provide flexibility for users/vendors to utilize various types of storage devices\n+* Define a standard partitioning scheme for storage drives for all Kubernetes nodes\n+* Provide storage usage isolation for shared partitions\n+* Support random access storage devices only (e.g., hard disks and SSDs)\n+\n+# Non Goals\n+* Provide storage usage isolation for non-shared partitions.\n+* Support all storage devices natively in upstream Kubernetes. Non standard storage devices are expected to be managed using extension mechanisms.\n+* Support for I/O isolation using CFS & blkio cgroups.\n+  * IOPS isn't safe to be a schedulable resource. IOPS on rotational media is very limited compared to other resources like CPU and Memory. This leads to severe resource stranding.\n+  * Blkio cgroup + CFS based I/O isolation doesn't provide deterministic behavior compared to memory and cpu cgroups. Years of experience at Google with Borg has taught that relying on blkio or I/O scheduler isn't suitable for multi-tenancy.\n+  * Blkio cgroup based I/O isolation isn't suitable for SSDs. Turning on CFQ on SSDs will hamper performance. Its better to statically partition SSDs and share them instead of using CFS.\n+  * I/O isolation can be achieved by using a combination of static partitioning and remote storage. This proposal recommends this approach with illustrations below.\n+  * Pod level resource isolation extensions will be made available in the Kubelet which will let vendors add support for CFQ if necessary for their deployments.\n+  \n+# Use Cases\n+\n+## Ephemeral Local Storage\n+Today, ephemeral local storage is exposed to pods via the container’s writable layer, logs directory, and EmptyDir volumes.  Pods use ephemeral local storage for scratch space, caching and logs.  There are many issues related to the lack of local storage accounting and isolation, including:\n+\n+* Pods do not know how much local storage is available to them.\n+* Pods cannot request “guaranteed” local storage.\n+* Local storage is a “best-effort” resource.\n+* Pods can get evicted due to other pods filling up the local storage, after which no new pods will be admitted until sufficient storage has been reclaimed.\n+\n+## Persistent Local Storage\n+Distributed filesystems and databases are the primary use cases for persistent local storage due to the following factors:\n+\n+* Performance: On cloud providers, local SSDs give better performance than remote disks.\n+* Cost: On baremetal, in addition to performance, local storage is typically cheaper and using it is a necessity to provision distributed filesystems.\n+\n+Distributed systems often use replication to provide fault tolerance, and can therefore tolerate node failures. However, data gravity is preferred for reducing replication traffic and cold startup latencies.\n+\n+# Design Overview\n+\n+A node’s local storage can be broken into primary and secondary partitions.\n+\n+## Primary Partitions\n+Primary partitions are shared partitions that can provide ephemeral local storage.  The two supported primary partitions are:\n+\n+### Root\n+ This partition holds the kubelet’s root directory (`/var/lib/kubelet` by default) and `/var/log` directory. This partition may be shared between user pods, OS and Kubernetes system daemons. This partition can be consumed by pods via EmptyDir volumes, container logs, image layers and container writable layers. Kubelet will manage shared access and isolation of this partition. This partition is “ephemeral” and applications cannot expect any performance SLAs (Disk IOPS for example) from this partition.\n+\n+### Runtime\n+This is an optional partition which runtimes can use for overlay filesystems. Kubelet will attempt to identify and provide shared access along with isolation to this partition.\n+\n+## Secondary Partitions\n+All other partitions are exposed as local persistent volumes.  Each local volume uses an entire partition.  The PV interface allows for varying storage configurations to be supported, while hiding specific configuration details to the pod.  All the local PVs can be queried and viewed from a cluster level using the existing PV object.  Applications can continue to use their existing PVC specifications with minimal changes to request local storage.\n+\n+The local PVs can be precreated by an addon DaemonSet that discovers all the secondary partitions at well-known directories, and can create new PVs as partitions are added to the node.  A default addon can be provided to handle common configurations.\n+\n+Local PVs can only provide semi-persistence, and are only suitable for specific use cases that need performance, data gravity and can tolerate data loss.  If the node or PV fails, then either the pod cannot run, or the pod has to give up on the local PV and find a new one.  Failure scenarios can be handled by unbinding the PVC from the local PV, and forcing the pod to reschedule and find a new PV.\n+\n+Since local PVs are only accessible from specific nodes, a new PV-node association will be used by the scheduler to place pods.  The association can be generalized to support any type of PV, not just local PVs.  This allows for any volume plugin to take advantage of this behavior.\n+\n+# User Workflows\n+\n+### Alice manages a deployment and requires “Guaranteed” ephemeral storage\n+\n+1. Kubelet running across all nodes will identify primary partition and expose capacity and allocatable for the primary partitions.  This allows primary partitions' storage capacity to be considered as a first class resource when scheduling.\n+\n+    ```yaml\n+    apiVersion: v1\n+    kind: Node\n+    metadata:\n+      name: foo\n+    status:\n+      capacity:\n+        storage.kubernetes.io/overlay: 100Gi\n+        storage.kubernetes.io/scratch: 100Gi\n+      allocatable:\n+        storage.kubernetes.io/overlay: 100Gi\n+        storage.kubernetes.io/scratch: 90Gi\n+```\n+\n+2. Alice adds new storage resource requirements to her pod, specifying limits for the container's writeable and overlay layers, and emptyDir volumes.\n+\n+    ```yaml\n+    apiVersion: v1\n+    kind: pod\n+    metadata:\n+     name: foo\n+    spec:\n+     containers:\n+     - name: fooc\n+       resources:\n+         limits:\n+           storage.kubernetes.io/logs: 500Mi\n+           storage.kubernetes.io/overlay: 1Gi\n+       volumeMounts:\n+       - name: myEmptyDir\n+         mountPath: /mnt/data\n+     volumes:\n+     - name: myEmptyDir\n+       emptyDir:\n+\t     size: 20Gi\n+    ```\n+\n+3. Alice’s pod “foo” is Guaranteed a total of “21.5Gi” of local storage. The container “fooc” in her pod cannot consume more than 1Gi for writable layer and 500Mi for logs, and “myEmptyDir” volume cannot consume more than 20Gi.\n+4. For the pod resources, `storage.kubernetes.io/logs` resource is meant for logs. `storage.kubernetes.io/overlay` is meant for writable layer.\n+5. `storage.kubernetes.io/logs` is satisfied by `storage.kubernetes.io/scratch`.\n+6. `storage.kubernetes.io/overlay` resource can be satisfied by `storage.kubernetes.io/overlay` if exposed by nodes or by `storage.kubernetes.io/scratch` otherwise. The scheduler follows this policy to find an appropriate node which can satisfy the storage resource requirements of the pod.\n+7. EmptyDir.size is both a request and limit that is satisfied by `storage.kubernetes.io/scratch`. \n+8. Kubelet will rotate logs to keep scratch space usage of “fooc” under 500Mi\n+9. Kubelet will track the usage of pods across logs and overlay filesystem and restart the container if it's total usage exceeds it's storage limits. If usage on `EmptyDir` volume exceeds its `limit`, then the pod will be evicted by the kubelet. By performing soft limiting, users will be able to easily identify pods that run out of storage.\n+10. Health is monitored by an external entity like the “Node Problem Detector” which is expected to place appropriate taints.\n+11. If a primary partition becomes unhealthy, the node is tainted and all pods running in it will be evicted by default, unless they tolerate that taint. Kubelet’s behavior on a node with unhealthy primary partition is undefined. Cluster administrators are expected to fix unhealthy primary partitions on nodes.\n+\n+### Bob runs batch workloads and is unsure of “storage” requirements\n+\n+1. Bob can create pods without any “storage” resource requirements. \n+\n+    ```yaml\n+    apiVersion: v1\n+    kind: pod\n+    metadata:\n+     name: foo\n+     namespace: myns\n+    spec:\n+     containers:\n+     - name: fooc\n+       volumeMounts:\n+       - name: myEmptyDir\n+         mountPath: /mnt/data\n+     volumes:\n+     - name: myEmptyDir\n+       emptyDir:\n+    ```\n+\n+2. His cluster administrator being aware of the issues with disk reclamation latencies has intelligently decided not to allow overcommitting primary partitions. The cluster administrator has installed a [LimitRange](https://kubernetes.io/docs/user-guide/compute-resources/) to “myns” namespace that will set a default storage size. Note: A cluster administrator can also specify burst ranges and a host of other features supported by LimitRange for local storage.\n+\n+    ```yaml\n+    apiVersion: v1\n+    kind: LimitRange\n+    metadata:\n+      name: mylimits\n+    spec:\n+       - default:\n+         storage.kubernetes.io/logs: 200Mi\n+         storage.kubernetes.io/overlay: 200Mi\n+         type: Container\n+       - default:\n+         size: 1Gi\n+         type: EmptyDir\n+    ```\n+\n+3. The limit range will update the pod specification as follows:\n+\n+    ```yaml\n+    apiVersion: v1\n+    kind: pod\n+    metadata:\n+     name: foo\n+    spec:\n+     containers:\n+     - name: fooc\n+       resources:\n+         limits:\n+           storage.kubernetes.io/logs: 200Mi\n+           storage.kubernetes.io/overlay: 200Mi\n+       volumeMounts:\n+       - name: myEmptyDir\n+         mountPath: /mnt/data\n+     volumes:\n+     - name: myEmptyDir\n+       emptyDir:\n+\t     size: 1Gi\n+    ```\n+\n+4. Bob’s “foo” pod can use upto “200Mi” for its containers logs and writable layer each, and “1Gi” for its “myEmptyDir” volume. \n+5. If Bob’s pod “foo” exceeds the “default” storage limits and gets evicted, then Bob can set a minimum storage requirement for his containers and a higher “capacity” for his EmptyDir volumes.\n+\n+  ```yaml\n+  apiVersion: v1\n+  kind: pod\n+  metadata:\n+   name: foo\n+  spec:\n+   containers:\n+   - name: fooc\n+     resources:\n+       requests:\n+         storage.kubernetes.io/logs: 500Mi\n+         storage.kubernetes.io/overlay: 500Mi\n+     volumeMounts:\n+     - name: myEmptyDir\n+       mountPath: /mnt/data\n+   volumes:\n+   - name: myEmptyDir\n+     emptyDir:\n+\t\tsize: 2Gi\n+  ```\n+\n+6. It is recommended to require `limits` to be specified for `storage` in all pods. `storage` will not affect the `QoS` Class of a pod since no SLA is intended to be provided for storage capacity isolation. it is recommended to use Persistent Durable Volumes as much as possible and avoid primary partitions.\n+\n+### Alice manages a Database which needs access to “durable” and fast scratch space\n+\n+1. Cluster administrator provisions machines with local SSDs and brings up the cluster\n+2. When a new node instance starts up, an addon DaemonSet discovers local “secondary” partitions which are mounted at a well known location and creates Local PVs for them if one doesn’t exist already. The PVs will include a path to the secondary device mount points, and a new node annotation that ties the volume to a specific node.  A StorageClass is required and will have a new optional field `locality` for the system to be able to differentiate between local and remote storage.  Labels may also be specified.  The volume consumes the entire partition.\n+\n+    ```yaml\n+    kind: StorageClass\n+    apiVersion: storage.k8s.io/v1\n+    metadata:\n+      name: local-fast\n+    provisioner: \"\"","path":"contributors/design-proposals/local-storage-overview.md","position":214,"original_position":214,"commit_id":"e4c61005b033621aba74758a4d816e6510cb848b","original_commit_id":"f50fca41c9aa3722e3b4d8d986fd994b927d96e4","user":{"login":"msau42","id":24448061,"avatar_url":"https://avatars0.githubusercontent.com/u/24448061?v=3","gravatar_id":"","url":"https://api.github.com/users/msau42","html_url":"https://github.com/msau42","followers_url":"https://api.github.com/users/msau42/followers","following_url":"https://api.github.com/users/msau42/following{/other_user}","gists_url":"https://api.github.com/users/msau42/gists{/gist_id}","starred_url":"https://api.github.com/users/msau42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msau42/subscriptions","organizations_url":"https://api.github.com/users/msau42/orgs","repos_url":"https://api.github.com/users/msau42/repos","events_url":"https://api.github.com/users/msau42/events{/privacy}","received_events_url":"https://api.github.com/users/msau42/received_events","type":"User","site_admin":false},"body":"Yes, it is in the StorageClass for the dynamic provisioning case.  In the near term, I think this could help with the zone scheduling issue.  And in the future, this could help with supporting dynamic provisioning of local PVs.\r\n\r\nI'll update the reasoning here.","created_at":"2017-04-17T22:11:31Z","updated_at":"2017-04-17T22:19:57Z","html_url":"https://github.com/kubernetes/community/pull/306#discussion_r111833286","pull_request_url":"https://api.github.com/repos/kubernetes/community/pulls/306","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/community/pulls/comments/111833286"},"html":{"href":"https://github.com/kubernetes/community/pull/306#discussion_r111833286"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/community/pulls/306"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/community/pulls/306","id":103805548,"html_url":"https://github.com/kubernetes/community/pull/306","diff_url":"https://github.com/kubernetes/community/pull/306.diff","patch_url":"https://github.com/kubernetes/community/pull/306.patch","issue_url":"https://api.github.com/repos/kubernetes/community/issues/306","number":306,"state":"open","locked":false,"title":"[Proposal] Improve Local Storage Management","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars1.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"body":"A note to reviewers: A detailed design proposal will be posted once the overall design in this proposal has been accepted by the community. So kindly hold on to specific design questions or suggestions that are not relevant to the overall high level design for local storage. \r\n\r\ncc @kubernetes/sig-storage-proposals  @kubernetes/sig-node-proposals @kubernetes/sig-apps-proposals @kubernetes/sig-scheduling-proposals \r\n\r\ncc @msau42  \r\n\r\nFor https://github.com/kubernetes/features/issues/121\r\n\r\nTODO:\r\n- [x] Handle scheduling of `runtime` primary partition capacity\r\n- [x] Propose design choices for mitigating or avoiding Disk IO issues for system daemons, container logs and writable layer. EmptyDir IO isolation is out of scope because local PV can be used instead.","created_at":"2017-01-30T19:43:22Z","updated_at":"2017-04-17T22:19:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7843166a3fa56c8b5c28fbd10dde97fe7d46f899","assignee":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars3.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"assignees":[{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars3.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},{"login":"wattsteve","id":3201484,"avatar_url":"https://avatars0.githubusercontent.com/u/3201484?v=3","gravatar_id":"","url":"https://api.github.com/users/wattsteve","html_url":"https://github.com/wattsteve","followers_url":"https://api.github.com/users/wattsteve/followers","following_url":"https://api.github.com/users/wattsteve/following{/other_user}","gists_url":"https://api.github.com/users/wattsteve/gists{/gist_id}","starred_url":"https://api.github.com/users/wattsteve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wattsteve/subscriptions","organizations_url":"https://api.github.com/users/wattsteve/orgs","repos_url":"https://api.github.com/users/wattsteve/repos","events_url":"https://api.github.com/users/wattsteve/events{/privacy}","received_events_url":"https://api.github.com/users/wattsteve/received_events","type":"User","site_admin":false},{"login":"thockin","id":5595220,"avatar_url":"https://avatars3.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars0.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars2.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/community/pulls/306/commits","review_comments_url":"https://api.github.com/repos/kubernetes/community/pulls/306/comments","review_comment_url":"https://api.github.com/repos/kubernetes/community/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/community/issues/306/comments","statuses_url":"https://api.github.com/repos/kubernetes/community/statuses/e4c61005b033621aba74758a4d816e6510cb848b","head":{"label":"vishh:storage-proposal","ref":"storage-proposal","sha":"e4c61005b033621aba74758a4d816e6510cb848b","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars1.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"repo":{"id":80394404,"name":"community","full_name":"vishh/community","owner":{"login":"vishh","id":7063592,"avatar_url":"https://avatars1.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/vishh/community","description":"Kubernetes community content","fork":true,"url":"https://api.github.com/repos/vishh/community","forks_url":"https://api.github.com/repos/vishh/community/forks","keys_url":"https://api.github.com/repos/vishh/community/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vishh/community/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vishh/community/teams","hooks_url":"https://api.github.com/repos/vishh/community/hooks","issue_events_url":"https://api.github.com/repos/vishh/community/issues/events{/number}","events_url":"https://api.github.com/repos/vishh/community/events","assignees_url":"https://api.github.com/repos/vishh/community/assignees{/user}","branches_url":"https://api.github.com/repos/vishh/community/branches{/branch}","tags_url":"https://api.github.com/repos/vishh/community/tags","blobs_url":"https://api.github.com/repos/vishh/community/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vishh/community/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vishh/community/git/refs{/sha}","trees_url":"https://api.github.com/repos/vishh/community/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vishh/community/statuses/{sha}","languages_url":"https://api.github.com/repos/vishh/community/languages","stargazers_url":"https://api.github.com/repos/vishh/community/stargazers","contributors_url":"https://api.github.com/repos/vishh/community/contributors","subscribers_url":"https://api.github.com/repos/vishh/community/subscribers","subscription_url":"https://api.github.com/repos/vishh/community/subscription","commits_url":"https://api.github.com/repos/vishh/community/commits{/sha}","git_commits_url":"https://api.github.com/repos/vishh/community/git/commits{/sha}","comments_url":"https://api.github.com/repos/vishh/community/comments{/number}","issue_comment_url":"https://api.github.com/repos/vishh/community/issues/comments{/number}","contents_url":"https://api.github.com/repos/vishh/community/contents/{+path}","compare_url":"https://api.github.com/repos/vishh/community/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vishh/community/merges","archive_url":"https://api.github.com/repos/vishh/community/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vishh/community/downloads","issues_url":"https://api.github.com/repos/vishh/community/issues{/number}","pulls_url":"https://api.github.com/repos/vishh/community/pulls{/number}","milestones_url":"https://api.github.com/repos/vishh/community/milestones{/number}","notifications_url":"https://api.github.com/repos/vishh/community/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vishh/community/labels{/name}","releases_url":"https://api.github.com/repos/vishh/community/releases{/id}","deployments_url":"https://api.github.com/repos/vishh/community/deployments","created_at":"2017-01-30T04:48:28Z","updated_at":"2017-01-30T04:48:31Z","pushed_at":"2017-04-14T19:35:27Z","git_url":"git://github.com/vishh/community.git","ssh_url":"git@github.com:vishh/community.git","clone_url":"https://github.com/vishh/community.git","svn_url":"https://github.com/vishh/community","homepage":"","size":7228,"stargazers_count":0,"watchers_count":0,"language":"Makefile","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"a78e7d11ef10b95d6391ef5096c9355e028d882e","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars1.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":57939112,"name":"community","full_name":"kubernetes/community","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars1.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/community","description":"Kubernetes community content","fork":false,"url":"https://api.github.com/repos/kubernetes/community","forks_url":"https://api.github.com/repos/kubernetes/community/forks","keys_url":"https://api.github.com/repos/kubernetes/community/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/community/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/community/teams","hooks_url":"https://api.github.com/repos/kubernetes/community/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/community/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/community/events","assignees_url":"https://api.github.com/repos/kubernetes/community/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/community/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/community/tags","blobs_url":"https://api.github.com/repos/kubernetes/community/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/community/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/community/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/community/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/community/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/community/languages","stargazers_url":"https://api.github.com/repos/kubernetes/community/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/community/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/community/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/community/subscription","commits_url":"https://api.github.com/repos/kubernetes/community/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/community/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/community/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/community/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/community/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/community/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/community/merges","archive_url":"https://api.github.com/repos/kubernetes/community/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/community/downloads","issues_url":"https://api.github.com/repos/kubernetes/community/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/community/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/community/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/community/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/community/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/community/releases{/id}","deployments_url":"https://api.github.com/repos/kubernetes/community/deployments","created_at":"2016-05-03T03:36:05Z","updated_at":"2017-04-17T18:16:43Z","pushed_at":"2017-04-17T14:12:08Z","git_url":"git://github.com/kubernetes/community.git","ssh_url":"git@github.com:kubernetes/community.git","clone_url":"https://github.com/kubernetes/community.git","svn_url":"https://github.com/kubernetes/community","homepage":"","size":8089,"stargazers_count":335,"watchers_count":335,"language":"Makefile","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":324,"mirror_url":null,"open_issues_count":114,"forks":324,"open_issues":114,"watchers":335,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/community/pulls/306"},"html":{"href":"https://github.com/kubernetes/community/pull/306"},"issue":{"href":"https://api.github.com/repos/kubernetes/community/issues/306"},"comments":{"href":"https://api.github.com/repos/kubernetes/community/issues/306/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/community/pulls/306/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/community/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/community/pulls/306/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/community/statuses/e4c61005b033621aba74758a4d816e6510cb848b"}}}}
{"actor":{"display_login":"msau42"}}