{"action":"created","issue":{"url":"https://api.github.com/repos/google/kubeflow/issues/33","repository_url":"https://api.github.com/repos/google/kubeflow","labels_url":"https://api.github.com/repos/google/kubeflow/issues/33/labels{/name}","comments_url":"https://api.github.com/repos/google/kubeflow/issues/33/comments","events_url":"https://api.github.com/repos/google/kubeflow/issues/33/events","html_url":"https://github.com/google/kubeflow/issues/33","id":281070107,"number":33,"title":"Proposal: our expectation on KubeFlow","user":{"login":"DjangoPeng","id":16943353,"avatar_url":"https://avatars3.githubusercontent.com/u/16943353?v=4","gravatar_id":"","url":"https://api.github.com/users/DjangoPeng","html_url":"https://github.com/DjangoPeng","followers_url":"https://api.github.com/users/DjangoPeng/followers","following_url":"https://api.github.com/users/DjangoPeng/following{/other_user}","gists_url":"https://api.github.com/users/DjangoPeng/gists{/gist_id}","starred_url":"https://api.github.com/users/DjangoPeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DjangoPeng/subscriptions","organizations_url":"https://api.github.com/users/DjangoPeng/orgs","repos_url":"https://api.github.com/users/DjangoPeng/repos","events_url":"https://api.github.com/users/DjangoPeng/events{/privacy}","received_events_url":"https://api.github.com/users/DjangoPeng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2017-12-11T16:00:56Z","updated_at":"2017-12-12T21:02:45Z","closed_at":null,"author_association":"NONE","body":"**Authors:**\r\n\r\n- **@DjangoPeng** - Jingtian Peng \\<pengjingtian@caicloud.io\\>\r\n- **@zjx-caicloud** - Jianxin Zhang \\<zhangjianxin@caicloud.io\\>\r\n- **@gaocegege** - Ce Gao \\<gaoce@caicloud.io\\>\r\n- **@ScorpioCPH** - Penghao Cen \\<cenph@caicloud.io\\>\r\n\r\n### Motivation\r\n\r\nTensorFlow users can run training jobs on Kubernetes with the support of this project: [tensorflow/k8s](https://github.com/tensorflow/k8s), which is a CRD controller (or TensorFlow operator) for deploying distributed TensorFlow on Kubernetes.\r\n\r\nMaybe it is not sustainable for TensorFlow users (e.g. data scientist or deep learning engineer\r\n) to write a complex YAML file to deploy their TensorFlow Training and other related service (e.g. launch TensorFlow Serving and TensorBoard). And the users could not get the training status in the current design.\r\n\r\nThe KubeFlow project is dedicated to making a high-efficient, easy-to-use, cloud-native and scalable Machine Learning platform based on Kubernetes. Our goal is building an **end-to-end** system to **manage the whole lifecycle of Machine Learning Jobs**, which makes the TensorFlow to be kubernetes-friendly and user-friendly.\r\n\r\nThis doc describes a user-oriented distributed ML platform with:\r\n\r\n- Defining a user spec to deploy TensorFlow Training/Serving/TensorBoard job\r\n- Workflow from user side\r\n- Components of the architecture\r\n\r\n### Use Cases\r\n\r\n- I have no idea about Kubernetes and just want to focus on developing TensorFlow  model.\r\n- I don’t need to write YAML files for any kind of TensorFlow jobs, including training, serving and TensorBoard jobs.\r\n- I can specify computing resources (e.g. GPUs) which will be used in TensorFlow training or serving.\r\n- I can get the processing status and logs from KubeFlow API after triggering TensorFlow training jobs.\r\n- I can launch TensorFlow model as a service to expose trained TensorFlow model.\r\n- I can launch TensorBoard as a service which would track multiple experiments.\r\n\r\n### Goals\r\n\r\n- Implement a server\r\n  - Accept training code files, e.g. python files as input, and run training jobs on Kubernetes, with no need for additional configs.\r\n  - watch job status running on Kubernetes and do some cleanup works on the cluster\r\n- Implement a monitor, to get the job status from Kubernetes and report specific status to the KubeFlow server\r\n- Update CRD specification and controller to fit the new design\r\n- For scheduling (future work):\r\n  - Implement customized scheduler or reuse the community project [kubernetes-incubator/kube-arbitrator](https://github.com/kubernetes-incubator/kube-arbitrator) to support fine-grained scheduling for TensorFlow jobs\r\n- For resource management (future work):\r\n  - Use device plugin (in Kubelet) and NVIDIA [k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin) with nvidia-docker v2 for NVIDIA GPU scheduling\r\n  - Implement Resource Class with CRD and Custom Scheduler for different type GPUs scheduling (on homogeneous nodes)\r\n\r\n### Workflow\r\n\r\n![kubeflow_training](https://user-images.githubusercontent.com/16943353/33839179-88469962-decc-11e7-8483-42e8054e7b09.png)\r\n\r\n\r\n**Fig 1** KubeFlow distributed training job\r\n\r\nAs shown in Fig 1, users develop some ML models in the JupyterHub. Then they send the TensorFlow Python script (e.g. inception_v3.py) into KubeFlow.  When receiving the script, KubeFlow creates TensorFlow distributed training job based on Kubernetes. Finally, the TensorFlow training job would delivery sort of files, such as tf.events for TensorBoard visualization and saved_model.pb for TensorFlow Serving.\r\n\r\n![kubeflow_serving](https://user-images.githubusercontent.com/16943353/33839189-8d36af48-decc-11e7-80d6-54a9053164ad.png)\r\n\r\n\r\n**Fig 2** KubeFlow serving job\r\n\r\nAs shown in Fig 2, users send the saved_model.pb file into KubeFlow. Then KubeFlow creates a TensorFlow Serving job to provide a public service. With the number of requests increasing, KubeFlow would automatically scale out the job.\r\n\r\n![kubeflow_tensorboard](https://user-images.githubusercontent.com/16943353/33839202-93c33c14-decc-11e7-9e9f-8dc7c09dea66.png)\r\n\r\n\r\n**Fig 3** KubeFlow TensorBoard job\r\n\r\nAs shown in Fig 3, users send the tf.events file into KubeFlow. Then KubeFlow creates a TensorBoard job to track multiple experiments. \r\n\r\n### Components\r\n\r\n![kubeflow_components](https://user-images.githubusercontent.com/16943353/33839209-97e32fac-decc-11e7-9767-471d56859b94.png)\r\n\r\n\r\n**Fig 4** KubeFlow components\r\n\r\nThe design has several components:\r\n\r\n- Kubernetes Extended (to better serve Machine Learning workload)\r\n  - TensorFlow CRD\r\n  - TensorFlow Controller\r\n  - TensorFlow Scheduler\r\n- Server (to manage TensorFlow jobs and interact with the client)\r\n- Monitor (to watch and report statuses of all TensorFlow jobs)\r\n\r\n#### CRD\r\n\r\nAccording to the use case and design goal, TensorFlow jobs are always divided into three types as below:\r\n\r\n- Training Job\r\n  - Local Training Job\r\n  - Distributed Training Job\r\n- Serving Job\r\n- TensorBoard Job\r\n\r\nTensorFlow distributed training job is most complicated and elusive among them. It adopts the classical PS-Worker distributed framework. PS (Parameter Server) is responsible for storing and updating the model's parameters. Worker is responsible for computing and applying gradients.\r\n\r\nBut, the native implementation of TensorFlow distributed framework is not perfect. For example, PS processes would join forever even though all Workers processes completed their computation works, which is a waste of resource. At the same time, Kubernetes has no idea of the TenforFlow job for lack of TensorFlow conceptions and semantics. So, it's difficult for native Kubernetes to make a realtime monitor watching the TenforFlow job status. In this case, KubeFlow aims to handle all problems above and fill in the gap between TensorFlow and Kubernetes.\r\n\r\nThe state-of-art implementation tensorflow/k8s generally works well and lots of works could be reused in the new CRD design. We have some changes based on [tensorflow/k8s](https://github.com/tensorflow/k8s). We divides TensorFlow jobs into three types: training jobs, serving jobs and TensorBoard jobs.\r\n\r\nIn general, we always launch more than one TensorFlow training jobs with multiple kinds of hyper parameters combinations to get the best training results. At the same time, we can use one TensorBoard instance to visualize all of TensorFlow events outputted from training jobs. It is really a common case for data scientists and algorithm engineers. But, in the current design, the TensorBoard job is binding to one TensorFlow job.\r\n\r\nIn order to make above come true, we need to decouple the TensorFlow job and TensorBoard job. We can make some changes on the TenforFlow job CRD. At first, both jobs can share TFReplicaSpec. Moreover, we can use TFReplicaSpec to specify the TensorFlow Serving job. At the same time, we can add a type field to define the TensorFlow job type (one of Training, TensorBoard or Serving).\r\n\r\n```\r\napiVersion: \"tensorflow.org/v1alpha1\"\r\nkind: \"TFJob\"\r\nmetadata:\r\n  name: \"training-job\"\r\nspec:\r\n  type: Training\r\n  tfReplicaSpec:\r\n  - replicas: 2\r\n    tfReplicaType: PS\r\n    template:\r\n      spec:\r\n        containers:\r\n        - name: tensorflow\r\n          image: tensorflow/tensorflow:1.4.0\r\n          command:\r\n          - \"/workdir/tensorflow/launch_training.sh\"\r\n          volumeMounts:\r\n          - name: workdir\r\n            mountPath: /workdir\r\n        volumes:\r\n        - name: workdir\r\n          glusterfs:\r\n            endpoints: <gluster-cluster>\r\n            path: <gluster_vol_subpath>\r\n        restartPolicy: OnFailure\r\n      tfReplicaSpec:\r\n  - replicas: 4\r\n    tfReplicaType: Worker\r\n    template:\r\n      spec:\r\n        containers:\r\n        - name: tensorflow\r\n          image: tensorflow/tensorflow:1.4.0\r\n          command:\r\n          - \"/workdir/tensorflow/launch_training.sh\"\r\n          args:\r\n          - \"--data_dir=/workdir/data\"\r\n          - \"--train_dir=/workdir/train\"\r\n          - \"--model_dir=/workdir/model\"\r\n          volumeMounts:\r\n          - name: workdir\r\n            mountPath: /workdir\r\n        volumes:\r\n        - name: workdir\r\n          glusterfs:\r\n            endpoints: <gluster-cluster>\r\n            path: <gluster_vol_subpath>\r\n        restartPolicy: OnFailure\r\n```\r\n**Algo 1** YAML config for distributed training job\r\n\r\nRelated issues: [tensorflow/k8s#209](https://github.com/tensorflow/k8s/issues/209)\r\n\r\n\r\n#### Controller\r\n\r\n[tensorflow/k8s](https://github.com/tensorflow/k8s) follows the CoreOS operator pattern to handle the events from Kubernetes, we aim to refactor the project to controller pattern which is similar to [kubernetes/sample-controller](https://github.com/kubernetes/sample-controller) to be extensible and robust.\r\n\r\nBut it has the same function as [tensorflow/k8s](https://github.com/tensorflow/k8s): the controller watches the shared state of the cluster through the Kubernetes apiserver and makes changes attempting to move the current state towards the desired state for TensorFlow jobs on Kubernetes.\r\n\r\n#### Server\r\n\r\nServer is the most important component in KubeFlow, which are responsible for:\r\n\r\nSetting up RESTful apiserver to listen for requests for creating TensorFlow jobs or requests from KubeFlow monitor for status updating.\r\n\r\nGenerating objects that can be accepted by Kubernetes apiserver to create TensorFlow jobs according to user requests.\r\n\r\nRecording status of TensorFlow jobs in order to reschedule TensorFlow jobs when something goes wrong during training.\r\n\r\n#### Monitor\r\n\r\nKubeFlow controller allows users to create TensorFlow training jobs on Kubernetes, but the parameter servers are always running even after the training job finished. Besides this, users could not get the Tensorflow job's' status unless they access to Kubernetes.\r\n\r\nThen we aim to implement a component which is called KubeFlow monitor, to watch the status of TensorFlow jobs on Kubernetes. And the monitor should report the status to KubeFlow server, then the server could move the current state towards the desired state."},"comment":{"url":"https://api.github.com/repos/google/kubeflow/issues/comments/351193340","html_url":"https://github.com/google/kubeflow/issues/33#issuecomment-351193340","issue_url":"https://api.github.com/repos/google/kubeflow/issues/33","id":351193340,"user":{"login":"erikerlandson","id":259898,"avatar_url":"https://avatars0.githubusercontent.com/u/259898?v=4","gravatar_id":"","url":"https://api.github.com/users/erikerlandson","html_url":"https://github.com/erikerlandson","followers_url":"https://api.github.com/users/erikerlandson/followers","following_url":"https://api.github.com/users/erikerlandson/following{/other_user}","gists_url":"https://api.github.com/users/erikerlandson/gists{/gist_id}","starred_url":"https://api.github.com/users/erikerlandson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikerlandson/subscriptions","organizations_url":"https://api.github.com/users/erikerlandson/orgs","repos_url":"https://api.github.com/users/erikerlandson/repos","events_url":"https://api.github.com/users/erikerlandson/events{/privacy}","received_events_url":"https://api.github.com/users/erikerlandson/received_events","type":"User","site_admin":false},"created_at":"2017-12-12T21:02:45Z","updated_at":"2017-12-12T21:02:45Z","author_association":"NONE","body":"Regarding the cost of moving the project, any resulting friction from moving it seems likely to only increase over time.\r\n\r\nIt could be ported into a new GH org like so:\r\n1. create GH org `https://github.com/kubeflow`\r\n1. fork repo `https://github.com/google/kubeflow` into org `https://github.com/kubeflow`\r\n1. decomission repo `https://github.com/google/kubeflow`\r\n1. move forward with repo (and org) `https://github.com/kubeflow/kubeflow`\r\n\r\nWe did something very similar with spark-on-kube about a year ago, and it has been a great platform for growing a dev and user community around.  Pragmatically it's very convenient to on-board new org owners or committers, in a way that is decoupled from any particular company. It provides a foundation for community governance.\r\n\r\nJust as a POC, I just created an org `kubeflowio` (kubeflow is taken) and forked the repo:\r\nhttps://github.com/kubeflowio/kubeflow\r\n\r\n\r\n"}}
{"actor":{"display_login":"erikerlandson"}}