{"action":"closed","issue":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/2205","repository_url":"https://api.github.com/repos/hashicorp/nomad","labels_url":"https://api.github.com/repos/hashicorp/nomad/issues/2205/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/nomad/issues/2205/comments","events_url":"https://api.github.com/repos/hashicorp/nomad/issues/2205/events","html_url":"https://github.com/hashicorp/nomad/issues/2205","id":201247316,"number":2205,"title":"Template:  Allocation remains pending forever if consul key cannot be read.","user":{"login":"cyrilgdn","id":4532947,"avatar_url":"https://avatars.githubusercontent.com/u/4532947?v=3","gravatar_id":"","url":"https://api.github.com/users/cyrilgdn","html_url":"https://github.com/cyrilgdn","followers_url":"https://api.github.com/users/cyrilgdn/followers","following_url":"https://api.github.com/users/cyrilgdn/following{/other_user}","gists_url":"https://api.github.com/users/cyrilgdn/gists{/gist_id}","starred_url":"https://api.github.com/users/cyrilgdn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyrilgdn/subscriptions","organizations_url":"https://api.github.com/users/cyrilgdn/orgs","repos_url":"https://api.github.com/users/cyrilgdn/repos","events_url":"https://api.github.com/users/cyrilgdn/events{/privacy}","received_events_url":"https://api.github.com/users/cyrilgdn/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-01-17T11:10:11Z","updated_at":"2017-01-17T18:04:56Z","closed_at":"2017-01-17T18:04:56Z","body":"### Nomad version\r\nNomad v0.5.2\r\n\r\n### Issue\r\n\r\nI created a simple test job (see below) which try to download a consul key in a file with:\r\n`data = \"{{ key \\\"configtest\\\" }}\"` \r\n\r\nWhen I run this job, if the read of consul key return a 404 (key does not exists or, in my case, key is protected by ACL), the allocation remains in Pending state forever (an job in Starting state)\r\n\r\nIs this a wanted behavior?\r\nMaybe the job should failed after a few tries?\r\n\r\n### Reproduction steps\r\n```\r\n# Run the job (see Job File below)\r\n~ nomad run template-test.nomad\r\n\r\n# Job status remains in this state:\r\n~ nomad status  template-test                       \r\nID          = template-test\r\nName        = template-test\r\nType        = batch\r\nPriority    = 50\r\nDatacenters = dc1\r\nStatus      = running\r\nPeriodic    = false\r\n\r\nSummary\r\nTask Group     Queued  Starting  Running  Failed  Complete  Lost\r\ntemplate-test  0       1         0        0       0         0\r\n\r\nAllocations\r\nID        Eval ID   Node ID   Task Group     Desired  Status   Created At\r\n47847e8c  efe9ec58  6a744838  template-test  run      pending  01/17/17 10:17:56 CET\r\n\r\n# Allocation status\r\n~ nomad alloc-status 47847e8c\r\nID                 = 47847e8c\r\nEval ID            = efe9ec58\r\nName               = template-test.template-test[0]\r\nNode ID            = 6a744838\r\nJob ID             = template-test\r\nClient Status      = pending\r\nClient Description = <none>\r\nCreated At         = 01/17/17 10:17:56 CET\r\n\r\nTask \"template-test\" is \"pending\"\r\nTask Resources\r\nCPU      Memory  Disk  IOPS  Addresses\r\n100 MHz  10 MiB  0 B   0     \r\n\r\nRecent Events:\r\nTime                   Type      Description\r\n01/17/17 10:17:56 CET  Received  Task received by client\r\n```\r\n\r\n\r\n### Job file\r\n```\r\njob \"template-test\" {\r\n    datacenters = [\"dc1\"]\r\n\r\n    type = \"batch\"\r\n\r\n    group \"template-test\" {\r\n        task \"template-test\" {\r\n            driver = \"exec\"\r\n\r\n            config {\r\n                command = \"sh\"\r\n                args = [\"-c\", \"sleep 5; cat local/test.conf; exit 0\"]\r\n            }\r\n\r\n            template {\r\n                destination = \"local/test.conf\"\r\n                data = \"{{ key \\\"configtest\\\" }}\"\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\n### Nomad Server logs\r\n```\r\n    No configuration files loaded\r\n==> Starting Nomad agent...\r\n==> Nomad agent configuration:\r\n\r\n                 Atlas: <disabled>\r\n                Client: true\r\n             Log Level: DEBUG\r\n                Region: global (DC: dc1)\r\n                Server: true\r\n               Version: 0.5.2\r\n\r\n==> Nomad agent started! Log data will stream in below:\r\n\r\n    2017/01/17 10:17:44 [INFO] serf: EventMemberJoin: cyril.global 127.0.0.1\r\n    2017/01/17 10:17:44.021771 [INFO] nomad: starting 4 scheduling worker(s) for [service batch system _core]\r\n    2017/01/17 10:17:44.021974 [INFO] client: using state directory /tmp/nomad/client\r\n    2017/01/17 10:17:44.021997 [INFO] client: using alloc directory /tmp/nomad/alloc\r\n    2017/01/17 10:17:44.022055 [DEBUG] client: built-in fingerprints: [arch cgroup consul cpu host memory network nomad signal storage vault env_aws env_gce]\r\n    2017/01/17 10:17:44.022090 [INFO] nomad: adding server cyril.global (Addr: 127.0.0.1:4647) (DC: dc1)\r\n    2017/01/17 10:17:44 [INFO] raft: Node at 127.0.0.1:4647 [Follower] entering Follower state (Leader: \"\")\r\n    2017/01/17 10:17:44.022229 [INFO] fingerprint.cgroups: cgroups are available\r\n    2017/01/17 10:17:44.022425 [DEBUG] client: fingerprinting cgroup every 15s\r\n    2017/01/17 10:17:44.023563 [INFO] fingerprint.consul: consul agent is available\r\n    2017/01/17 10:17:44.023687 [DEBUG] client: fingerprinting consul every 15s\r\n    2017/01/17 10:17:44.023913 [DEBUG] fingerprint.cpu: frequency: 2800 MHz\r\n    2017/01/17 10:17:44.023917 [DEBUG] fingerprint.cpu: core count: 4\r\n    2017/01/17 10:17:44.230647 [DEBUG] fingerprint.network: Detected interface lo with IP 127.0.0.1 during fingerprinting\r\n    2017/01/17 10:17:44.230698 [DEBUG] fingerprint.network: Unable to read link speed from /sys/class/net/lo/speed\r\n    2017/01/17 10:17:44.230701 [DEBUG] fingerprint.network: link speed could not be detected and no speed specified by user. Defaulting to 1000\r\n    2017/01/17 10:17:44.232203 [DEBUG] client: fingerprinting vault every 15s\r\n    2017/01/17 10:17:45 [WARN] raft: Heartbeat timeout from \"\" reached, starting election\r\n    2017/01/17 10:17:45 [INFO] raft: Node at 127.0.0.1:4647 [Candidate] entering Candidate state\r\n    2017/01/17 10:17:45 [DEBUG] raft: Votes needed: 1\r\n    2017/01/17 10:17:47.370578 [INFO] client: node registration complete    2017/01/17 10:17:45 [DEBUG] raft: Vote granted from 127.0.0.1:4647. Tally: 1\r\n    2017/01/17 10:17:47.370742 [DEBUG] client: updated allocations at index 1 (pulled 0) (filtered 0)\r\n    2017/01/17 10:17:47.370838 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 0)\r\n    2017/01/17 10:17:45 [INFO] raft: Election won. Tally: 1\r\n    2017/01/17 10:17:45 [INFO] raft: Node at 127.0.0.1:4647 [Leader] entering Leader state    2017/01/17 10:17:47.370964 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"test\"]\r\n    2017/01/17 10:17:45 [INFO] raft: Disabling EnableSingleNode (bootstrap)\r\n    2017/01/17 10:17:45.507179 [INFO] nomad: cluster leadership acquired\r\n    2017/01/17 10:17:45 [DEBUG] raft: Node 127.0.0.1:4647 updated peer set (2): [127.0.0.1:4647]\r\n    2017/01/17 10:17:45.507725 [DEBUG] leader: reconciling job summaries at index: 0\r\n    2017/01/17 10:17:46.232749 [DEBUG] fingerprint.env_aws: Error querying AWS Metadata URL, skipping\r\n    2017/01/17 10:17:47.284615 [DEBUG] fingerprint.env_gce: Could not read value for attribute \"machine-type\"\r\n    2017/01/17 10:17:47.284636 [DEBUG] fingerprint.env_gce: Error querying GCE Metadata URL, skipping\r\n    2017/01/17 10:17:47.284683 [DEBUG] client: applied fingerprints [arch cgroup consul cpu host memory network nomad signal storage]\r\n    2017/01/17 10:17:47.284904 [DEBUG] driver.docker: using client connection initialized from environment\r\n    2017/01/17 10:17:47.285082 [DEBUG] client: fingerprinting rkt every 15s\r\n    2017/01/17 10:17:47.286854 [DEBUG] driver.exec: exec driver is enabled\r\n    2017/01/17 10:17:47.286987 [DEBUG] client: fingerprinting docker every 15s\r\n    2017/01/17 10:17:47.287101 [DEBUG] client: fingerprinting exec every 15s\r\n    2017/01/17 10:17:47.369927 [DEBUG] client: available drivers [docker exec raw_exec java]\r\n    2017/01/17 10:17:47.370057 [INFO] client: Node ID \"6a744838-96b4-8bd1-d7ba-c472ce26aa1a\"\r\n    2017/01/17 10:17:47.371539 [ERR] client.consul: error discovering nomad servers: no Nomad Servers advertising service \"nomad\" in Consul datacenters: [\"test\"]\r\n    2017/01/17 10:17:47.371612 [DEBUG] client: periodically checking for node changes at duration 5s\r\n    2017/01/17 10:17:47.371829 [DEBUG] client: state updated to ready\r\n    2017/01/17 10:17:56.100297 [DEBUG] worker: dequeued evaluation efe9ec58-8b9f-6ea1-d761-0ee180a2bc06\r\n    2017/01/17 10:17:56.100378 [DEBUG] sched: <Eval 'efe9ec58-8b9f-6ea1-d761-0ee180a2bc06' JobID: 'template-test'>: allocs: (place 1) (update 0) (migrate 0) (stop 0) (ignore 0) (lost 0)\r\n    2017/01/17 10:17:56.101104 [DEBUG] http: Request /v1/jobs?region=global (1.500512ms)\r\n    2017/01/17 10:17:56.101780 [DEBUG] worker: submitted plan for evaluation efe9ec58-8b9f-6ea1-d761-0ee180a2bc06\r\n    2017/01/17 10:17:56.101856 [DEBUG] sched: <Eval 'efe9ec58-8b9f-6ea1-d761-0ee180a2bc06' JobID: 'template-test'>: setting status to complete\r\n    2017/01/17 10:17:56.101902 [DEBUG] client: updated allocations at index 8 (pulled 1) (filtered 0)\r\n    2017/01/17 10:17:56.102229 [DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 0)\r\n    2017/01/17 10:17:56.102500 [DEBUG] worker: updated evaluation <Eval 'efe9ec58-8b9f-6ea1-d761-0ee180a2bc06' JobID: 'template-test'>\r\n    2017/01/17 10:17:56.102554 [DEBUG] worker: ack for evaluation efe9ec58-8b9f-6ea1-d761-0ee180a2bc06\r\n    2017/01/17 10:17:56.103181 [DEBUG] http: Request /v1/evaluation/efe9ec58-8b9f-6ea1-d761-0ee180a2bc06?region=global (555.412µs)\r\n    2017/01/17 10:17:56.104512 [DEBUG] http: Request /v1/evaluation/efe9ec58-8b9f-6ea1-d761-0ee180a2bc06/allocations?region=global (214.695µs)\r\n    2017/01/17 10:17:56.110735 [DEBUG] client: starting task runners for alloc '47847e8c-f805-ca4b-e981-3ae38777e036'\r\n    2017/01/17 10:17:56.110867 [DEBUG] client: starting task context for 'template-test' (alloc '47847e8c-f805-ca4b-e981-3ae38777e036')\r\n    2017/01/17 10:17:56 [INFO] (runner) creating new runner (dry: false, once: false)\r\n    2017/01/17 10:17:56 [DEBUG] (runner) final config (tokens suppressed):\r\n\r\n{\r\n  \"Path\": \"\",\r\n  \"Consul\": \"\",\r\n  \"Token\": \"\",\r\n  \"ReloadSignal\": 1,\r\n  \"DumpSignal\": 3,\r\n  \"KillSignal\": 2,\r\n  \"Auth\": {\r\n    \"Enabled\": false,\r\n    \"Username\": \"\",\r\n    \"Password\": \"\"\r\n  },\r\n  \"Vault\": {\r\n    \"Address\": \"\",\r\n    \"UnwrapToken\": false,\r\n    \"RenewToken\": false,\r\n    \"SSL\": {\r\n      \"Enabled\": true,\r\n      \"Verify\": true,\r\n      \"Cert\": \"\",\r\n      \"Key\": \"\",\r\n      \"CaCert\": \"\",\r\n      \"CaPath\": \"\",\r\n      \"ServerName\": \"\"\r\n    }\r\n  },\r\n  \"SSL\": {\r\n    \"Enabled\": false,\r\n    \"Verify\": true,\r\n    \"Cert\": \"\",\r\n    \"Key\": \"\",\r\n    \"CaCert\": \"\",\r\n    \"CaPath\": \"\",\r\n    \"ServerName\": \"\"\r\n  },\r\n  \"Syslog\": {\r\n    \"Enabled\": false,\r\n    \"Facility\": \"LOCAL0\"\r\n  },\r\n  \"Exec\": {\r\n    \"Command\": \"\",\r\n    \"Splay\": 0,\r\n    \"ReloadSignal\": null,\r\n    \"KillSignal\": 15,\r\n    \"KillTimeout\": 30000000000\r\n  },\r\n  \"MaxStale\": 1000000000,\r\n  \"ConfigTemplates\": [\r\n    {\r\n      \"Source\": \"\",\r\n      \"Destination\": \"/tmp/nomad/alloc/47847e8c-f805-ca4b-e981-3ae38777e036/template-test/local/test.conf\",\r\n      \"EmbeddedTemplate\": \"{{ key \\\"configtest\\\" }}\",\r\n      \"Command\": \"\",\r\n      \"CommandTimeout\": 0,\r\n      \"Perms\": 420,\r\n      \"Backup\": false,\r\n      \"LeftDelim\": \"\",\r\n      \"RightDelim\": \"\",\r\n      \"Wait\": {\r\n        \"min\": 0,\r\n        \"max\": 0\r\n      }\r\n    }\r\n  ],\r\n  \"Retry\": 5000000000,\r\n  \"Wait\": {\r\n    \"min\": 0,\r\n    \"max\": 0\r\n  },\r\n  \"PidFile\": \"\",\r\n  \"LogLevel\": \"WARN\",\r\n  \"Deduplicate\": {\r\n    \"Enabled\": false,\r\n    \"Prefix\": \"consul-template/dedup/\",\r\n    \"TTL\": 15000000000\r\n  }\r\n}\r\n\r\n    2017/01/17 10:17:56 [INFO] (clients) creating consul/api client\r\n    2017/01/17 10:17:56 [INFO] (clients) creating vault/api client\r\n    2017/01/17 10:17:56 [DEBUG] (clients) enabling vault SSL\r\n    2017/01/17 10:17:56 [INFO] (runner) creating Watcher\r\n    2017/01/17 10:17:56 [INFO] (runner) starting\r\n    2017/01/17 10:17:56 [DEBUG] (runner) running initial templates\r\n    2017/01/17 10:17:56 [INFO] (runner) running\r\n    2017/01/17 10:17:56 [DEBUG] (runner) checking template a7c72e8af6bd3ec74e72f63455032e62\r\n    2017/01/17 10:17:56 [INFO] (runner) was not watching 2 dependencies\r\n    2017/01/17 10:17:56 [INFO] (watcher) adding \"key(configtest)\"\r\n    2017/01/17 10:17:56 [DEBUG] (watcher) \"key(configtest)\" starting\r\n    2017/01/17 10:17:56 [INFO] (watcher) adding \"key(configtest)\"\r\n    2017/01/17 10:17:56 [DEBUG] (watcher) \"key(configtest)\" already exists, skipping\r\n    2017/01/17 10:17:56 [INFO] (runner) diffing and updating dependencies\r\n    2017/01/17 10:17:56 [INFO] (runner) watching 1 dependencies\r\n    2017/01/17 10:17:56 [DEBUG] (view) \"key(configtest)\" starting fetch\r\n    2017/01/17 10:17:56 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:0 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:17:56 [DEBUG](view) \"key(configtest)\" data was not present\r\n    2017/01/17 10:17:56 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:17:56.220845 [DEBUG] client: updated allocations at index 10 (pulled 0) (filtered 1)\r\n    2017/01/17 10:17:56.220943 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)\r\n    2017/01/17 10:18:57 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:18:57 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:19:44.923652 [DEBUG] http: Request /v1/jobs?prefix=template-test (1.383435ms)\r\n    2017/01/17 10:19:44.928406 [DEBUG] http: Request /v1/job/template-test (3.432318ms)\r\n    2017/01/17 10:19:44.932277 [DEBUG] http: Request /v1/job/template-test\r\n    2017/01/17 10:20:01 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:20:01 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:21:03 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:21:03 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:22:03 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:22:03 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:22:45.508362 [DEBUG] worker: dequeued evaluation 9c0b83e9-e876-a3af-b95f-f553c405680b\r\n    2017/01/17 10:22:45.508526 [DEBUG] sched.core: eval GC: scanning before index 0 (1h0m0s)\r\n    2017/01/17 10:22:45.508693 [DEBUG] worker: ack for evaluation 9c0b83e9-e876-a3af-b95f-f553c405680b\r\n    2017/01/17 10:22:45.508801 [DEBUG] worker: dequeued evaluation 8c1260fb-3736-73b1-9bd4-94571760db4d\r\n    2017/01/17 10:22:45.508903 [DEBUG] sched.core: job GC: scanning before index 0 (4h0m0s)\r\n    2017/01/17 10:22:45.508997 [DEBUG] worker: ack for evaluation 8c1260fb-3736-73b1-9bd4-94571760db4d\r\n    2017/01/17 10:22:45.509104 [DEBUG] worker: dequeued evaluation 510e8440-ac18-c2a1-dfd5-cc25ede176a6\r\n    2017/01/17 10:22:45.509189 [DEBUG] sched.core: node GC: scanning before index 0 (24h0m0s)\r\n    2017/01/17 10:22:45.509273 [DEBUG] worker: ack for evaluation 510e8440-ac18-c2a1-dfd5-cc25ede176a6\r\n    2017/01/17 10:23:04 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:23:04 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:23:12.283734 [DEBUG] client: updated allocations at index 10 (pulled 0) (filtered 1)\r\n    2017/01/17 10:23:12.284105 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)\r\n    2017/01/17 10:24:08 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:24:08 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:25:11 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:25:11 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:26:00.511325 [DEBUG] http: Request /v1/jobs?prefix=template-test (1.322226ms)\r\n    2017/01/17 10:26:00.514341 [DEBUG] http: Request /v1/job/template-test (773.681µs)\r\n    2017/01/17 10:26:00.523499 [DEBUG] http: Request /v1/job/template-test/allocations (1.560619ms)\r\n    2017/01/17 10:26:00.531481 [DEBUG] http: Request /v1/job/template-test/evaluations (462.904µs)\r\n    2017/01/17 10:26:00.534095 [DEBUG] http: Request /v1/job/template-test/summary (705.178µs)\r\n    2017/01/17 10:26:14 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:26:14 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n    2017/01/17 10:27:15 [DEBUG] (view) \"key(configtest)\" no new data (index was the same)\r\n    2017/01/17 10:27:15 [DEBUG] (\"key(configtest)\") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:9679 WaitTime:1m0s Token: Near:}\r\n  [...]\r\n```"}}
{"actor":{"display_login":"dadgar"}}