{"action":"created","issue":{"url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy/issues/43","repository_url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy","labels_url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy/issues/43/labels{/name}","comments_url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy/issues/43/comments","events_url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy/issues/43/events","html_url":"https://github.com/bluemixgaragelondon/cf-blue-green-deploy/issues/43","id":241234631,"number":43,"title":"Routes temporarily disappear when using CF v265","user":{"login":"henrytk","id":8709339,"avatar_url":"https://avatars2.githubusercontent.com/u/8709339?v=4","gravatar_id":"","url":"https://api.github.com/users/henrytk","html_url":"https://github.com/henrytk","followers_url":"https://api.github.com/users/henrytk/followers","following_url":"https://api.github.com/users/henrytk/following{/other_user}","gists_url":"https://api.github.com/users/henrytk/gists{/gist_id}","starred_url":"https://api.github.com/users/henrytk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/henrytk/subscriptions","organizations_url":"https://api.github.com/users/henrytk/orgs","repos_url":"https://api.github.com/users/henrytk/repos","events_url":"https://api.github.com/users/henrytk/events{/privacy}","received_events_url":"https://api.github.com/users/henrytk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-07-07T11:19:55Z","updated_at":"2017-09-11T15:36:20Z","closed_at":null,"author_association":"NONE","body":"## The problem\r\n\r\nWhen using [Cloud Foundry v265](https://github.com/cloudfoundry/cf-release/releases/tag/v265) applications pushed with the blue-green-deploy plugin suffer downtime of up to 60 seconds. The steps to reproduce listed below demonstrate this is not actually caused by your plugin, but I am raising this issue because you are affected by it. When I find the Cloud Foundry component responsible I will raise it with them as well.\r\n\r\n**Update:** We have tracked this down to the route-emitter and have raised an issue with its maintainers: https://github.com/cloudfoundry/route-emitter/issues/8. A fix is in diego-release v1.22.0. The earliest cf-release that includes this fix is v268.\r\n\r\n## Context\r\n\r\nWe have deployed open source Cloud Foundry v265, upgrading from v257. We fully automate the deployment of applications using the blue-green-deployment plugin. There was zero downtime with CF v257, but now with CF v265 there is up to 60 seconds where requests to the app are responded to with a status code of 404.\r\n\r\n## Steps to reproduce\r\n\r\nThis can be reproduced with all versions of the plugin, but we use scripting because the root cause is not this plugin. It seems it is a race condition in Cloud Foundry:\r\n\r\n* Deploy Cloud Foundry v265\r\n* Deploy an application\r\n* Set up a `curl` to monitor the app:\r\n\r\n  ```\r\n  while true; do curl -s -o /dev/null -w \"%{http_code} \" https://${APP_NAME}.${APP_DOMAIN} -k; date; sleep 1; done | tee /tmp/bgd.out\r\n  ```\r\n\r\n* Do a blue-green deploy using this [example script](https://gist.github.com/henrytk/3a45e082b5713c268b5eea03dd922cfa).\r\n\r\nYou should see 404s returned after the unmapping of the route from the old application for up to 60 seconds.\r\n\r\nAdd a sleep of 60 seconds before the unmapping of the route from the old app and the problem goes away.\r\n\r\n## A theory\r\n\r\nIn our setup the route-emitter emits registration and unregistration messages via NATS every 20 seconds and does a bulk update every 60 seconds. I believe this is representative of most deployments. Because the downtime is always less than 60 seconds I believe the state in BBS is correct and on the bulk update the problem gets fixed. Therefore, the route-emitter is somehow unregistering the route from the new app instead of the old app.\r\n\r\n**Update:** I've confirmed by tailing the logs of the GoRouter that the route is definitely unregistered from the new app, when it should be unregistered from the old app.\r\n\r\n## Versions\r\n\r\ncf-release v265\r\ndiego-release v1.19.0\r\nAll versions of this plugin\r\nCF CLI 6.26"},"comment":{"url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy/issues/comments/328568328","html_url":"https://github.com/bluemixgaragelondon/cf-blue-green-deploy/issues/43#issuecomment-328568328","issue_url":"https://api.github.com/repos/bluemixgaragelondon/cf-blue-green-deploy/issues/43","id":328568328,"user":{"login":"henrytk","id":8709339,"avatar_url":"https://avatars2.githubusercontent.com/u/8709339?v=4","gravatar_id":"","url":"https://api.github.com/users/henrytk","html_url":"https://github.com/henrytk","followers_url":"https://api.github.com/users/henrytk/followers","following_url":"https://api.github.com/users/henrytk/following{/other_user}","gists_url":"https://api.github.com/users/henrytk/gists{/gist_id}","starred_url":"https://api.github.com/users/henrytk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/henrytk/subscriptions","organizations_url":"https://api.github.com/users/henrytk/orgs","repos_url":"https://api.github.com/users/henrytk/repos","events_url":"https://api.github.com/users/henrytk/events{/privacy}","received_events_url":"https://api.github.com/users/henrytk/received_events","type":"User","site_admin":false},"created_at":"2017-09-11T15:36:20Z","updated_at":"2017-09-11T15:36:20Z","author_association":"NONE","body":"@holly-cummins I am not sure how a user of the plugin can be constrained to use it against certain versions of CloudFoundry. If there are actual mechanisms for such a thing then fine, otherwise it seems more like a documentation issue."}}
{"actor":{"display_login":"henrytk"}}