{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/orleans/issues/2571","repository_url":"https://api.github.com/repos/dotnet/orleans","labels_url":"https://api.github.com/repos/dotnet/orleans/issues/2571/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/orleans/issues/2571/comments","events_url":"https://api.github.com/repos/dotnet/orleans/issues/2571/events","html_url":"https://github.com/dotnet/orleans/issues/2571","id":199049217,"number":2571,"title":"OrleansDockerUtils -> Docker and Docker Swarm Clusters for Orleans","user":{"login":"galvesribeiro","id":4714040,"avatar_url":"https://avatars0.githubusercontent.com/u/4714040?v=3","gravatar_id":"","url":"https://api.github.com/users/galvesribeiro","html_url":"https://github.com/galvesribeiro","followers_url":"https://api.github.com/users/galvesribeiro/followers","following_url":"https://api.github.com/users/galvesribeiro/following{/other_user}","gists_url":"https://api.github.com/users/galvesribeiro/gists{/gist_id}","starred_url":"https://api.github.com/users/galvesribeiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galvesribeiro/subscriptions","organizations_url":"https://api.github.com/users/galvesribeiro/orgs","repos_url":"https://api.github.com/users/galvesribeiro/repos","events_url":"https://api.github.com/users/galvesribeiro/events{/privacy}","received_events_url":"https://api.github.com/users/galvesribeiro/received_events","type":"User","site_admin":false},"labels":[{"id":159047420,"url":"https://api.github.com/repos/dotnet/orleans/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/dotnet/orleans/milestones/18","html_url":"https://github.com/dotnet/orleans/milestone/18","labels_url":"https://api.github.com/repos/dotnet/orleans/milestones/18/labels","id":2247963,"number":18,"title":"1.5.0","description":"","creator":{"login":"sergeybykov","id":8248806,"avatar_url":"https://avatars2.githubusercontent.com/u/8248806?v=3","gravatar_id":"","url":"https://api.github.com/users/sergeybykov","html_url":"https://github.com/sergeybykov","followers_url":"https://api.github.com/users/sergeybykov/followers","following_url":"https://api.github.com/users/sergeybykov/following{/other_user}","gists_url":"https://api.github.com/users/sergeybykov/gists{/gist_id}","starred_url":"https://api.github.com/users/sergeybykov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sergeybykov/subscriptions","organizations_url":"https://api.github.com/users/sergeybykov/orgs","repos_url":"https://api.github.com/users/sergeybykov/repos","events_url":"https://api.github.com/users/sergeybykov/events{/privacy}","received_events_url":"https://api.github.com/users/sergeybykov/received_events","type":"User","site_admin":false},"open_issues":28,"closed_issues":40,"state":"open","created_at":"2017-01-12T16:44:03Z","updated_at":"2017-04-20T17:36:17Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2017-01-05T20:18:23Z","updated_at":"2017-04-26T16:37:36Z","closed_at":"2017-01-30T23:50:39Z","body":"This issue is for discussion about the design of `OrleansDockerUtils` so we can add native support to docker containers as Orleans Silos. The implementation is on PR #2569 \r\n\r\n## Goal\r\n\r\nThe idea behind this effort is to enable an Orleans Silo to run in containers with minimal efforts by leveraging both Orleans' provider model and Docker APIs. This PR introduces a `DockerMembershipOracle` and a `DockerGatewayProvider` following the design of #2542 but instead of use SF services, use Docker APIs.\r\n\r\n## Some context\r\n\r\nWithout going too deep, a brief explanation about common topics on docker ecosystem so the reviewers feels more comfortable with it.\r\n\r\n### Docker Daemon\r\nThis is the Docker process running in the underlying OS (`dockerd` on Linux and OSX and `dockerd.exe` on Windows). This process is responsible to manage the containers and provide a way (using `libcontainer`) to talk to underlying OS' kernel in order to provide the container runtime environment. Each docker host (in production or even your development machine and regardless if single-node or clustered) has one instance of this process. This process also expose a API so 3rd party developers can integrate/automate container management. \r\n\r\n### Docker Swarm\r\nSwarm is the most used clustering (and orchestration) technology for Docker and is created by Docker Inc. Swarm implement almost 100% of the Docker APIs which means, that Docker CLI (and API) commands issued to swarm has (almost) no difference to the ones issued to a single node Docker Daemon. While in Swarm mode, commands are issue to the Swarm Manager nodes and it apply the request actions across the cluster of Docker Daemons properly.\r\n\r\n### Docker Compose\r\n*YAML* file describing grouped/correlated services/containers and the dependencies between each other as long as networking and volume mapping settings. This file can be pushed to a single Docker Daemon or to a Swarm cluster and will deploy the whole set of containers, networks and whatever artifact described in it. It is the most used composition tech for Docker containers nowadays.\r\n\r\n### Container\r\nA container is basically an isolation share of the host kernel with its own processes. The PID 1 of each container is the command passed when container start or described in the `Dockerfile` (the container image description file). In our case it would be our Silo host application. If for whatever reason that process crash or return/exit, the container will also exit.\r\n\r\n### Labels\r\nDocker allow one to add metadata to container and/or images thru labels. They are essentially key/value pairs. It can be anything as long as it can be string represented (key or value) and `JSON.stringfy()`(only value). You can even apply a label without a value. \r\n\r\n## Design\r\n\r\nI'm trying to make simple the clustering of Orleans silos by leveraging the existent features from Docker. I'm not going to enter in details about how Swarm (or any other tech) cluster work since it is beyond the scope here, but some aspects will be mentioned when necessary. Regardless of running with a single Docker Daemon, a Swarm Cluster, or whatever other clustering stack, as long as it implement the Docker API, we should be fine.\r\n\r\nThe main idea is to follow almost the same implementation of SF Membership Oracle and Gateway provider, except that instead of having partitions and SF Service discover services to keep the cluster data, we are not keeping the cluster data anywhere! (I know it sounds crazy, but keep reading)\r\n\r\n4 labels where defined. Those labels can be applied at each container at startup, in `docker-compose.yml`, or in their images so it is always applied on every container that derive from it (they can be overridden while starting a container). \r\n\r\n`IS_DOCKER_SILO = \"com.microsoft.orleans.silo\"` -> Should be set on docker images/containers which run an Orleans Silo host\r\n`SILO_PORT = \"com.microsoft.orleans.siloport\"` -> The inter-silo port used in the cluster for all Silos\r\n`GATEWAY_PORT = \"com.microsoft.orleans.gatewayport\"` -> The client gateway port\r\n`GENERATION = \"com.microsoft.orleans.generation\"` -> The silo generation \r\n`DEPLOYMENT_ID = \"com.microsoft.orleans.deploymentid\"` -> This is the key for a deployment using this component. Will make sure that everyone involved in the Orleans Deployment (either client or silo) will talk to everyone on the same deployment.\r\n\r\nBesides all the common Orleans Membership protocol, ping, etc which is already known, and all the implementation of SF Membership Oracle, here is how we describe the two key components of this Membership Oracle for Docker:\r\n\r\n`DockerSiloResolver` -> This class is responsible for read the Docker API (now with a `Timer`) and refresh its silo list, and publish it to `IDockerStatusListener` listeners. It query Docker APIs for all containers that have a given `DEPLOYMENT_ID` label set to the `deploymentId` and `IS_DOCKER_SILO` with no value. This ensure that even if there are other containers using that deploymentId (lets say if the developer uses it as a general aggregate of containers) for something else, only the containers which is suppose to have silos are returned.\r\n\r\n`DockerMembershipOracle` -> When initialized, register itself to the `IDockerSiloResolver` registered on DI, and wait for updates. All its implementation is exactly a copy from the SF one, so threre not much to explain about it. (I would suggest later in other PR to make this class abstract or something else so we can remove this boilerplate code)\r\n\r\n`DockerGatewayProvider` -> When initialized, it register itself with a `DockerSiloResolver` (no DI on client!!!) and wait for updates. This inherit most of the implementation of SF one and uses the same filter logic but only returns to the client the silo which have gateways installed. ~~Yes, now I'm assuming all silos in the cluster are gateways. We can probably change that and ask for ones with `GATEWAY_PORT` label in it as well. For the sake of V1, we would live with it.~~\r\n\r\nThis implementation works. Silos come up and down (and crash) and they are refreshed on the silo/gateway lists. You are either UP or Down. Since a container is immutable, you don't restart a container because the silo process is dead. If it die, it is automatically killed and discarded. You need to start another fresh container, which will have another container name, IP address, ID, etc.\r\n\r\n### Silo health\r\n\r\nThe common MBR we have (Azure, SQL, AWS) rely on a consistent persistence table to store the cluster's silo health. Inside the membership protocol, silos \"ping\" each other and if it don't get a reply, it *suspect* that this silo is dead and write its own identity+timestamp of that attempt to the suspicious silo record in the MBR table. If the number of suspicious are > than configured threshold, that silo is marked as dead and no messages/activations are sent to it. \r\n\r\nOk. Given that context, we have a problem with this implementation that rely on Docker... \r\n\r\n> persistence table to store the cluster's silo health\r\n\r\nWe don't have a table! This implementation don't have a `IMembershipTable`. We are instead implementing a `IMembershipOracle`. The idea behind this provider is to rely on `Docker Labels` to aggregate silos and gateways by attaching labels to the containers part of the cluster.\r\n\r\nThere are 2 ways to detect if a silo is alive in this implementation:\r\n\r\n1. The container is in `running` state.\r\n2. The `HEALTHCHECK` tag in `Dockerfile` -> This will run an arbitrary command inside the container. We can use it to try connect to the silo ports (common approach on linux based containers). If it doesn't connect or timeout, Docker will kill the container and the next refresh on the MBR Oracle will catch that change.\r\n\r\nThe problem is, what if the other containers/silos part of the cluster can't ping another silo? For example in a network partition... That is why we have the votes. The problem is that we can't store anything in Docker. The labels aren't updatable and the Raft implementation on Swarm, is just for Swarm and we can't touch. \r\n\r\nMy first idea was to use (another common approach on unix systems) to create a lock file.\r\n\r\n#### Suggested approach to votes\r\n\r\nThe votes would happen this way:\r\n\r\n1. *SiloA* try to ping *SiloB* and fail (no response from *SiloB* for whatever reason)\r\n2. *SiloA* ask Docker to check inside that container the number of files in a specific directory (an arbitrary directory that holds those files)\r\n3. If the number of files are > than allowed threshold, ask Docker to kill that container\r\n4. If it is not, ask Docker to create an empty lock file with the *SiloA* identity + timestamp on *SiloB* at that give directory. -> Note that there isn't races on writing this files since each silo ask to write its own identity so 0 chance of conflict.\r\n5. Repeat\r\n\r\nIn case of a network partition (or whatever other reason), *SiloB* Oracle also keep checking its own directory, so it can suicide in case of the files are > than allowed threshold. \r\n\r\nI know it is not the best thing to do (deal with files) but I don't see a way to make votes to work. Suggestions are appreciated.\r\n\r\nIMHO this is enough for a V1 but I appreciate any suggestions.\r\n\r\n## Challengers for a V2\r\n\r\n### Label updates\r\nSince containers are immutable, it can't be changed/updated. However, labels are metadata essentially. Docker allow you to update a label in runtime but there are two downsides:\r\n\r\n1. The container is paused while updating. This means the whole container memory is frozen, the label changes are applied, a commit on that container happen, and you have the container back running. This may not noticeable while starting a silo (the update is fast) but I wonder what side effects to the cluster it will have if we do it after the silo started and is joined to the it. Examples of updates would be to mark a silo as dead, so in the next refresh even if the silo process (and the container) is running, we will not have that frozen container in the list or being able to generate the `GENERATION` value at silo start and update it.\r\n2. Windows Containers does not support yet update running containers. Linux works just fine (I tested it).\r\n\r\n### Docker event stream\r\n\r\nWould be nice if instead of pooling the Docker API only, we could listen to Docker API stream of events like `container_created`, `container_running`, `container_dead` and react accordingly to update the silo list. However, the stream API is very unstable/experimental at the moment and we shouldn't rely on that yet because only Docker Daemons on experimental mode turned on will emit them. Production environments like AWS or Azure ACS doesn't have it.\r\n\r\n### Kill an unresponsive silo\r\n\r\nSomehow we could kill a container even if it is running but become unresponsive. This is just a matter of invoke an API. I just don't know the moment for it or how the best way to declare a silo dead so this API can be invoked. \r\n\r\nI think that is it. \r\n\r\ncc: @gabikliot @sergeybykov @ReubenBond "},"comment":{"url":"https://api.github.com/repos/dotnet/orleans/issues/comments/297469413","html_url":"https://github.com/dotnet/orleans/issues/2571#issuecomment-297469413","issue_url":"https://api.github.com/repos/dotnet/orleans/issues/2571","id":297469413,"user":{"login":"galvesribeiro","id":4714040,"avatar_url":"https://avatars0.githubusercontent.com/u/4714040?v=3","gravatar_id":"","url":"https://api.github.com/users/galvesribeiro","html_url":"https://github.com/galvesribeiro","followers_url":"https://api.github.com/users/galvesribeiro/followers","following_url":"https://api.github.com/users/galvesribeiro/following{/other_user}","gists_url":"https://api.github.com/users/galvesribeiro/gists{/gist_id}","starred_url":"https://api.github.com/users/galvesribeiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galvesribeiro/subscriptions","organizations_url":"https://api.github.com/users/galvesribeiro/orgs","repos_url":"https://api.github.com/users/galvesribeiro/repos","events_url":"https://api.github.com/users/galvesribeiro/events{/privacy}","received_events_url":"https://api.github.com/users/galvesribeiro/received_events","type":"User","site_admin":false},"created_at":"2017-04-26T16:37:36Z","updated_at":"2017-04-26T16:37:36Z","body":"@mtdaniels - Will try get to it today. If you have any immediate questions please ping me on gitter and I'll try to help."}}
{"actor":{"display_login":"galvesribeiro"}}