{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/806","repository_url":"https://api.github.com/repos/hashicorp/nomad","labels_url":"https://api.github.com/repos/hashicorp/nomad/issues/806/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/nomad/issues/806/comments","events_url":"https://api.github.com/repos/hashicorp/nomad/issues/806/events","html_url":"https://github.com/hashicorp/nomad/issues/806","id":134113761,"number":806,"title":"Stuck allocation on dead job","user":{"login":"far-blue","id":344317,"avatar_url":"https://avatars1.githubusercontent.com/u/344317?v=3","gravatar_id":"","url":"https://api.github.com/users/far-blue","html_url":"https://github.com/far-blue","followers_url":"https://api.github.com/users/far-blue/followers","following_url":"https://api.github.com/users/far-blue/following{/other_user}","gists_url":"https://api.github.com/users/far-blue/gists{/gist_id}","starred_url":"https://api.github.com/users/far-blue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/far-blue/subscriptions","organizations_url":"https://api.github.com/users/far-blue/orgs","repos_url":"https://api.github.com/users/far-blue/repos","events_url":"https://api.github.com/users/far-blue/events{/privacy}","received_events_url":"https://api.github.com/users/far-blue/received_events","type":"User","site_admin":false},"labels":[{"id":218895783,"url":"https://api.github.com/repos/hashicorp/nomad/labels/bug","name":"bug","color":"f7c6c7","default":true},{"id":273201424,"url":"https://api.github.com/repos/hashicorp/nomad/labels/scheduling","name":"scheduling","color":"B722E4","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-02-16T22:25:20Z","updated_at":"2017-03-01T20:59:58Z","closed_at":"2016-03-26T02:27:21Z","body":"I'm new to all this so maybe I've just missed something but I appear to have an orphan allocation from a dead job that failed to completely start.\n\nContext: Running v0.3.0rc1 in the dev environment created by the included vagrantfile. Running in --dev' mode (dual agent/client mode).\n\nI started with a modified version of the example.nomad file created with nomad init and I modified the task to run a mysql container and added a second task to run an apache container. I started the job with `nomad run` but it failed to complete because I'd typo'd the apache container image name.\n\nAt this point I had a mysql container running but no apache container. So I edited the job to correct my typo and called `nomad run` again. My understanding was that it would evaluate the difference and just start the apache container (because the mysql container was already running).\n\nHowever, it actually re-evaluated the entire job and started both the apache container _and_ a second mysql container, while leaving the original container running. Note that I have not changed the name of the job or the task group (I left them as example and cache, as per the original job config).\n\nSo I called `nomad stop` thinking it would clean everything up but it only stopped the new containers, leaving the original mysql container. I thought maybe nomad had 'forgotten' about it so killed it with Docker directly - but nomad put it back.\n\nSo now I have a mysql container that nomad is keeping alive but no job to control it with.\n\n```\n> nomad status example\nNo job(s) with prefix or id \"example\" found\n```\n\n```\n> docker ps\nCONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                  NAMES\n266885ac1ee4        mysql:latest        \"/entrypoint.sh mysql\"   33 minutes ago      Up 33 minutes       127.0.0.1:23968->3306/tcp, 127.0.0.1:23968->3306/udp   mysql-bc506dfd-6351-ab4e-ad23-95c3fd971baa\n```\n\n```\n> nomad alloc-status bc506dfd\nID              = bc506dfd\nEval ID         = 3226e9b9\nName            = example.cache[0]\nNode ID         = f8e6eacc\nJob ID          = example\nClient Status   = failed\nEvaluated Nodes = 1\nFiltered Nodes  = 0\nExhausted Nodes = 0\nAllocation Time = 2.21072ms\nFailures        = 0\n\n==> Task \"apache\" is \"dead\"\nRecent Events:\nTime                   Type            Description\n16/02/16 21:33:52 UTC  Driver Failure  failed to start: Failed to pull `apache:latest`: Error: image library/apache not found\n\n==> Task \"mysql\" is \"running\"\nRecent Events:\nTime                   Type        Description\n16/02/16 21:49:12 UTC  Started     <none>\n16/02/16 21:48:39 UTC  Terminated  Exit Code: 0\n16/02/16 21:34:23 UTC  Started     <none>\n\n==> Status\nAllocation \"bc506dfd\" status \"failed\" (0/1 nodes filtered)\n  * Score \"f8e6eacc-46f7-18b0-df52-350346732e60.binpack\" = 7.683003\n```\n\nSo I'm not quite sure what to do next and I'm pretty certain this is not expected behaviour.\n\nAny thoughts anyone?\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/comments/283468279","html_url":"https://github.com/hashicorp/nomad/issues/806#issuecomment-283468279","issue_url":"https://api.github.com/repos/hashicorp/nomad/issues/806","id":283468279,"user":{"login":"rickardrosen","id":11421608,"avatar_url":"https://avatars1.githubusercontent.com/u/11421608?v=3","gravatar_id":"","url":"https://api.github.com/users/rickardrosen","html_url":"https://github.com/rickardrosen","followers_url":"https://api.github.com/users/rickardrosen/followers","following_url":"https://api.github.com/users/rickardrosen/following{/other_user}","gists_url":"https://api.github.com/users/rickardrosen/gists{/gist_id}","starred_url":"https://api.github.com/users/rickardrosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rickardrosen/subscriptions","organizations_url":"https://api.github.com/users/rickardrosen/orgs","repos_url":"https://api.github.com/users/rickardrosen/repos","events_url":"https://api.github.com/users/rickardrosen/events{/privacy}","received_events_url":"https://api.github.com/users/rickardrosen/received_events","type":"User","site_admin":false},"created_at":"2017-03-01T20:59:58Z","updated_at":"2017-03-01T20:59:58Z","body":"I am seeing this on Nomad v0.5.4.\r\n\r\nI had a job that no longer exists with an allocation stuck on a node, trying to pull a container that no longer exists and receiving a 400 from the registry.\r\nIt's been doing this for a couple of weeks without getting cleaned up, so tonight I decided to restart the nomad agent which allowed the task to be killed.\r\n\r\nIs it a regression, or have I triggered something completely new for some reason?"}}
{"actor":{"display_login":"rickardrosen"}}