{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/31663","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/31663/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/31663/comments","events_url":"https://api.github.com/repos/docker/docker/issues/31663/events","html_url":"https://github.com/docker/docker/issues/31663","id":212868625,"number":31663,"title":"Error restoring container from checkpoint: \"No parent found for mountpoint\"","user":{"login":"tswift242","id":1853953,"avatar_url":"https://avatars0.githubusercontent.com/u/1853953?v=3","gravatar_id":"","url":"https://api.github.com/users/tswift242","html_url":"https://github.com/tswift242","followers_url":"https://api.github.com/users/tswift242/followers","following_url":"https://api.github.com/users/tswift242/following{/other_user}","gists_url":"https://api.github.com/users/tswift242/gists{/gist_id}","starred_url":"https://api.github.com/users/tswift242/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tswift242/subscriptions","organizations_url":"https://api.github.com/users/tswift242/orgs","repos_url":"https://api.github.com/users/tswift242/repos","events_url":"https://api.github.com/users/tswift242/events{/privacy}","received_events_url":"https://api.github.com/users/tswift242/received_events","type":"User","site_admin":false},"labels":[{"id":190258329,"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false},{"id":214964971,"url":"https://api.github.com/repos/docker/docker/labels/kind/experimental","name":"kind/experimental","color":"bfdadc","default":false},{"id":555642114,"url":"https://api.github.com/repos/docker/docker/labels/version/17.03","name":"version/17.03","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-03-08T21:55:17Z","updated_at":"2017-03-09T16:48:18Z","closed_at":null,"body":"<!--\r\nIf you are reporting a new issue, make sure that we do not have any duplicates\r\nalready open. You can ensure this by searching the issue list for this\r\nrepository. If there is a duplicate, please close your issue and add a comment\r\nto the existing issue instead.\r\n\r\nIf you suspect your issue is a bug, please edit your issue description to\r\ninclude the BUG REPORT INFORMATION shown below. If you fail to provide this\r\ninformation within 7 days, we cannot debug your issue and will close it. We\r\nwill, however, reopen it if you later provide the information.\r\n\r\nFor more information about reporting issues, see\r\nhttps://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues\r\n\r\n---------------------------------------------------\r\nGENERAL SUPPORT INFORMATION\r\n---------------------------------------------------\r\n\r\nThe GitHub issue tracker is for bug reports and feature requests.\r\nGeneral support can be found at the following locations:\r\n\r\n- Docker Support Forums - https://forums.docker.com\r\n- IRC - irc.freenode.net #docker channel\r\n- Post a question on StackOverflow, using the Docker tag\r\n\r\n---------------------------------------------------\r\nBUG REPORT INFORMATION\r\n---------------------------------------------------\r\nUse the commands below to provide key information from your environment:\r\nYou do NOT have to include this information if this is a FEATURE REQUEST\r\n-->\r\n\r\n**Description**\r\n\r\nVery sporadically, I get the following type of error when trying to restore a container from a checkpoint, which causes the restore to fail:\r\n\r\n\"Error (criu/mount.c:360): mnt: No parent found for mountpoint 574 (@./var/lib/docker/checkpoints/test_cp/criu.work/restore-2017-03-08T15:12:42Z/.criu.cgyard.fnEP8f/systemd)\"\r\n\r\nThe full criu restore.log is attached.\r\n\r\n[restore.txt](https://github.com/docker/docker/files/829057/restore.txt)\r\n\r\nIt seems to sporadically happen when I restore several containers from the same checkpoint at the same time. The issue reproduced 8/720 times when I ran the following script, which just starts and stops a bunch of simple Ubuntu containers over and over (renamed to satisfy github):\r\n\r\n[stress_cr_simple_start_stop.txt](https://github.com/docker/docker/files/829065/stress_cr_simple_start_stop.txt)\r\n\r\nI gave the script the following parameters: ```./stress_cr_simple_start_stop.sh 10 720```; this will go through 720 rounds of restoring 10 containers at a time from the same checkpoint.\r\n\r\nI wonder if this has anything to do with the way docker assigns the criu work dir for each restore. Docker creates a separate work directory for each restore; the directory names are differentiated with a datetimestamp with ten-millisecond precision. So, this means that if docker makes 2+ restore calls to criu within ten milliseconds, the work dir for those restores could be the same. As an example, when I ran the script above for 720 runs with 10 concurrent restores at a time (7200 restores total), there were only 1087 different work directories created. The path that the criu error cites appears to be an ephemeral path that is created during the restore and which sits inside the work dir. If there are multiple concurrent restores happening for the same checkpoint, it therefore seems possible that they could be stepping on each others' toes if they are using the same work dir.\r\n\r\n**Steps to reproduce the issue:**\r\n1. ```./stress_cr_simple_start_stop.sh 10 720``` (can be a smaller number than 720, but should probably be at least 100)\r\n2. Observe restore failures during script execution, or grep for them afterward: ```grep -l -i \"error\" /var/lib/docker/checkpoints/test_cp/criu.work/**/*```\r\n\r\n**Describe the results you received:**\r\n\r\nRestore fails with message like \"Error (criu/mount.c:360): mnt: No parent found for mountpoint 574 (@./var/lib/docker/checkpoints/test_cp/criu.work/restore-2017-03-08T15:12:42Z/.criu.cgyard.fnEP8f/systemd)\"\r\n\r\n**Describe the results you expected:**\r\n\r\nRestores should always succeed, especially for a basic Ubuntu container.\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      17.03.0-ce\r\n API version:  1.26\r\n Go version:   go1.7.5\r\n Git commit:   3a232c8\r\n Built:        Tue Feb 28 07:57:58 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      17.03.0-ce\r\n API version:  1.26 (minimum version 1.12)\r\n Go version:   go1.7.5\r\n Git commit:   3a232c8\r\n Built:        Tue Feb 28 07:57:58 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: true\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 27\r\nServer Version: 17.03.0-ce\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host ipvlan macvlan null overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 977c511eda0925a723debdc94d09459af49d082a\r\nrunc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70\r\ninit version: 949e6fa\r\nSecurity Options:\r\n apparmor\r\nKernel Version: 4.10.1-041001-generic\r\nOperating System: Ubuntu 14.04.5 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 16\r\nTotal Memory: 120.1 GiB\r\nName: ip-10-97-0-35\r\nID: PBSJ:KR3H:XP7F:KQMN:CFJS:J75A:ZGUM:ZSC3:5DUR:MCZJ:5ACD:CXM7\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nExperimental: true\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nUsing AWS and criu 2.11."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/285408644","html_url":"https://github.com/docker/docker/issues/31663#issuecomment-285408644","issue_url":"https://api.github.com/repos/docker/docker/issues/31663","id":285408644,"user":{"login":"boucher","id":22065,"avatar_url":"https://avatars3.githubusercontent.com/u/22065?v=3","gravatar_id":"","url":"https://api.github.com/users/boucher","html_url":"https://github.com/boucher","followers_url":"https://api.github.com/users/boucher/followers","following_url":"https://api.github.com/users/boucher/following{/other_user}","gists_url":"https://api.github.com/users/boucher/gists{/gist_id}","starred_url":"https://api.github.com/users/boucher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boucher/subscriptions","organizations_url":"https://api.github.com/users/boucher/orgs","repos_url":"https://api.github.com/users/boucher/repos","events_url":"https://api.github.com/users/boucher/events{/privacy}","received_events_url":"https://api.github.com/users/boucher/received_events","type":"User","site_admin":false},"created_at":"2017-03-09T16:48:18Z","updated_at":"2017-03-09T16:48:18Z","body":"Here's the line that generates the path:\r\nhttps://github.com/docker/containerd/blob/docker-1.13.x/containerd-shim/process.go#L141\r\n\r\nThere's no real reason not to just use something like a uuid, or even just this plus a random number, if that's easy to generate in this context."}}
{"actor":{"display_login":"boucher"}}