{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/31768","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/31768/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/31768/comments","events_url":"https://api.github.com/repos/moby/moby/issues/31768/events","html_url":"https://github.com/moby/moby/issues/31768","id":213630664,"number":31768,"title":"Docker 1.13 unable to kill unresponsive containers ","user":{"login":"DamionWaltermeyer","id":7000179,"avatar_url":"https://avatars1.githubusercontent.com/u/7000179?v=4","gravatar_id":"","url":"https://api.github.com/users/DamionWaltermeyer","html_url":"https://github.com/DamionWaltermeyer","followers_url":"https://api.github.com/users/DamionWaltermeyer/followers","following_url":"https://api.github.com/users/DamionWaltermeyer/following{/other_user}","gists_url":"https://api.github.com/users/DamionWaltermeyer/gists{/gist_id}","starred_url":"https://api.github.com/users/DamionWaltermeyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DamionWaltermeyer/subscriptions","organizations_url":"https://api.github.com/users/DamionWaltermeyer/orgs","repos_url":"https://api.github.com/users/DamionWaltermeyer/repos","events_url":"https://api.github.com/users/DamionWaltermeyer/events{/privacy}","received_events_url":"https://api.github.com/users/DamionWaltermeyer/received_events","type":"User","site_admin":false},"labels":[{"id":73641990,"url":"https://api.github.com/repos/moby/moby/labels/area/runtime","name":"area/runtime","color":"fbca04","default":false},{"id":182417459,"url":"https://api.github.com/repos/moby/moby/labels/status/more-info-needed","name":"status/more-info-needed","color":"C6A49A","default":false},{"id":480238176,"url":"https://api.github.com/repos/moby/moby/labels/version/1.13","name":"version/1.13","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2017-03-12T20:13:47Z","updated_at":"2017-12-07T17:18:47Z","closed_at":null,"author_association":"NONE","body":"-->\r\n\r\n**Daemon unable to kill containers **\r\n\r\n<!--\r\nWhile running docker 1.13, the daemon gets into a state where it will not/can not  restart, stop, or kill a container.   It believes the container no longer exists, but still shows it in the list and the container itself is unresponsive to interaction.\r\n-->\r\n\r\n**Steps to reproduce the issue:**\r\n1. run docker 1.13\r\n2. try to stop a container, have it fail\r\n3. try to kill a container, have it fail.\r\n\r\n**Describe the results you received:**\r\n\r\n```\r\n~# docker kill ss-manager-5-0-0-premium-170302-191041 \r\n~# tailf -n 50  /var/log/upstart/docker.log\r\ntime=\"2017-03-12T19:55:32.674806975Z\" level=info msg=\"Container 4aa932810626 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.674828202Z\" level=info msg=\"Container 8b0adfb7e123 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.676849792Z\" level=info msg=\"Container 94d4b2ed6fba failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.676537069Z\" level=info msg=\"Container bc30500c7abb failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677315493Z\" level=info msg=\"Container 5b142373205f failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677341269Z\" level=info msg=\"Container aeae119b7ab6 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677361586Z\" level=info msg=\"Container 04a96afad4c8 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677610496Z\" level=info msg=\"Container fcc9dbb773a4 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677673471Z\" level=info msg=\"Container 61040a84271e failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677693876Z\" level=info msg=\"Container bc8447a998e9 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677707919Z\" level=info msg=\"Container 643223aca1bd failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677722431Z\" level=info msg=\"Container 722241c15b54 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:32.677804912Z\" level=info msg=\"Container f4131dedc151 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:39.848401195Z\" level=info msg=\"Container 3458aed3490ed82b834af68eef2aff05c7026f2e5e372b6704e621e071a4f24d failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:55:39.849196760Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 3458aed3490ed82b834af68eef2aff05c7026f2e5e372b6704e621e071a4f24d: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:55:49.849866345Z\" level=info msg=\"Container 3458aed3490e failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:55:49.869222190Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container a7c3c05fc8dfb462fb796baf2670837f5bb46fbf08602257c72945b1a3963649: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:55:56.660633065Z\" level=error msg=\"Handler for POST /containers/1ea92200f734d73e501a2289f0ad7b35877cdd5e452d6402901ff7eb9ba9c62b/start returned error: failed to create endpoint simple-ss-windows.9E7097DC-C95F-A2D5-0FD8-6C8BD0DB8716 on network bridge: adding interface vethf174bd5 to bridge docker0 failed: exchange full\" \r\ntime=\"2017-03-12T19:55:59.870261160Z\" level=info msg=\"Container a7c3c05fc8dfb462fb796baf2670837f5bb46fbf08602257c72945b1a3963649 failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:55:59.871896923Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container a7c3c05fc8dfb462fb796baf2670837f5bb46fbf08602257c72945b1a3963649: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:09.740266272Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container b0fa7e915b55a85b1c99decea3675242b81722620393861644c79e8dd9ee184f: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:09.873369121Z\" level=info msg=\"Container a7c3c05fc8df failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:56:09.889438359Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 04a96afad4c8ed5c0ded57507460b83c8a0ef5e4447b205dd702b003fc11bc67: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:19.740961133Z\" level=info msg=\"Container b0fa7e915b55a85b1c99decea3675242b81722620393861644c79e8dd9ee184f failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:56:19.741804980Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container b0fa7e915b55a85b1c99decea3675242b81722620393861644c79e8dd9ee184f: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:19.890020497Z\" level=info msg=\"Container 04a96afad4c8ed5c0ded57507460b83c8a0ef5e4447b205dd702b003fc11bc67 failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:56:19.891176708Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 04a96afad4c8ed5c0ded57507460b83c8a0ef5e4447b205dd702b003fc11bc67: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:29.743224528Z\" level=info msg=\"Container b0fa7e915b55 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:56:29.900078298Z\" level=info msg=\"Container 04a96afad4c8 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:56:29.958444420Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 3458aed3490ed82b834af68eef2aff05c7026f2e5e372b6704e621e071a4f24d: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:39.962634702Z\" level=info msg=\"Container 3458aed3490ed82b834af68eef2aff05c7026f2e5e372b6704e621e071a4f24d failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:56:39.965354089Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 3458aed3490ed82b834af68eef2aff05c7026f2e5e372b6704e621e071a4f24d: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:56:49.966826818Z\" level=info msg=\"Container 3458aed3490e failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:56:49.977822475Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container a7c3c05fc8dfb462fb796baf2670837f5bb46fbf08602257c72945b1a3963649: rpc error: code = 2 desc = containerd: container not found\" \r\nERRO[377793] containerd: deleting container                error=exit status 1: \"container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53 does not exist\\none or more of the container deletions failed\\n\"\r\ntime=\"2017-03-12T19:56:59.978711321Z\" level=info msg=\"Container a7c3c05fc8dfb462fb796baf2670837f5bb46fbf08602257c72945b1a3963649 failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:56:59.980489891Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container a7c3c05fc8dfb462fb796baf2670837f5bb46fbf08602257c72945b1a3963649: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:57:06.500490355Z\" level=info msg=\"Container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53 failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:57:06.502819988Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:57:09.981052370Z\" level=info msg=\"Container a7c3c05fc8df failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:57:16.503962723Z\" level=info msg=\"Container 421284b55846 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T19:57:33.957198699Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:57:43.958293987Z\" level=info msg=\"Container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53 failed to exit within 10 seconds of signal 15 - using the force\" \r\ntime=\"2017-03-12T19:57:43.959917250Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T19:57:53.963104972Z\" level=info msg=\"Container 421284b55846 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T20:00:32.370642737Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T20:00:42.373388749Z\" level=info msg=\"Container 421284b55846 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\ntime=\"2017-03-12T20:01:36.605491268Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T20:03:59.049313727Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container 421284b55846a2443d939f1a1ccf1542b2598e907371fc7df3ff3c56cb51aa53: rpc error: code = 2 desc = containerd: container not found\" \r\ntime=\"2017-03-12T20:04:09.050248921Z\" level=info msg=\"Container 421284b55846 failed to exit within 10 seconds of kill - trying direct SIGKILL\" \r\n```\r\n\r\n**Describe the results you expected:**\r\n\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\n~# docker version\r\nClient:\r\n Version:      1.13.0\r\n API version:  1.25\r\n Go version:   go1.7.3\r\n Git commit:   49bf474\r\n Built:        Tue Jan 17 09:50:17 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.13.0\r\n API version:  1.25 (minimum version 1.12)\r\n Go version:   go1.7.3\r\n Git commit:   49bf474\r\n Built:        Tue Jan 17 09:50:17 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 1024\r\n Running: 1023\r\n Paused: 0\r\n Stopped: 1\r\nImages: 35\r\nServer Version: 1.13.0\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 2120\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins: \r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e\r\nrunc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e\r\ninit version: 949e6fa\r\nSecurity Options:\r\n apparmor\r\nKernel Version: 4.4.0-64-generic\r\nOperating System: Ubuntu 14.04.5 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 3.859 GiB\r\nName: docker-sg-singapore-5.connectify.me\r\nID: 4W6K:UJSX:HMHT:EK47:2IAV:OH6Z:QQXJ:5UMR:QW6V:GJYH:3F4T:6VHN\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No swap limit support\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nSimplercloud VPS"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/350035224","html_url":"https://github.com/moby/moby/issues/31768#issuecomment-350035224","issue_url":"https://api.github.com/repos/moby/moby/issues/31768","id":350035224,"user":{"login":"msavlani","id":25729412,"avatar_url":"https://avatars3.githubusercontent.com/u/25729412?v=4","gravatar_id":"","url":"https://api.github.com/users/msavlani","html_url":"https://github.com/msavlani","followers_url":"https://api.github.com/users/msavlani/followers","following_url":"https://api.github.com/users/msavlani/following{/other_user}","gists_url":"https://api.github.com/users/msavlani/gists{/gist_id}","starred_url":"https://api.github.com/users/msavlani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msavlani/subscriptions","organizations_url":"https://api.github.com/users/msavlani/orgs","repos_url":"https://api.github.com/users/msavlani/repos","events_url":"https://api.github.com/users/msavlani/events{/privacy}","received_events_url":"https://api.github.com/users/msavlani/received_events","type":"User","site_admin":false},"created_at":"2017-12-07T17:18:47Z","updated_at":"2017-12-07T17:18:47Z","author_association":"NONE","body":"@mlaventure  Yes docker-containerd is alive.\r\n\r\nBelow is goroutine stack from dockerd -\r\n[goroutine-stacks-2017-12-07T104714-0600.log](https://github.com/moby/moby/files/1539919/goroutine-stacks-2017-12-07T104714-0600.log)\r\n\r\nI have docker log in debug mode but size of file is 100Mb.\r\nNot sure how can i attach that here and limit is only for 10mb.\r\n\r\n"}}
{"actor":{"display_login":"msavlani"}}