{"action":"created","issue":{"url":"https://api.github.com/repos/lxc/lxc/issues/1726","repository_url":"https://api.github.com/repos/lxc/lxc","labels_url":"https://api.github.com/repos/lxc/lxc/issues/1726/labels{/name}","comments_url":"https://api.github.com/repos/lxc/lxc/issues/1726/comments","events_url":"https://api.github.com/repos/lxc/lxc/issues/1726/events","html_url":"https://github.com/lxc/lxc/issues/1726","id":247404407,"number":1726,"title":"Restarting lxc service leaves orphaned cgroups","user":{"login":"tomponline","id":280181,"avatar_url":"https://avatars3.githubusercontent.com/u/280181?v=4","gravatar_id":"","url":"https://api.github.com/users/tomponline","html_url":"https://github.com/tomponline","followers_url":"https://api.github.com/users/tomponline/followers","following_url":"https://api.github.com/users/tomponline/following{/other_user}","gists_url":"https://api.github.com/users/tomponline/gists{/gist_id}","starred_url":"https://api.github.com/users/tomponline/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomponline/subscriptions","organizations_url":"https://api.github.com/users/tomponline/orgs","repos_url":"https://api.github.com/users/tomponline/repos","events_url":"https://api.github.com/users/tomponline/events{/privacy}","received_events_url":"https://api.github.com/users/tomponline/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-08-02T14:15:57Z","updated_at":"2017-08-04T10:30:35Z","closed_at":null,"body":"When restarting lxc service using ```systemctl restart lxc``` the containers seem to leave orphaned cgroups in /sys/fs/cgroups\r\n\r\nThis doesn't happen if the containers are stopped with ```lxc-stop``` or shutdown from the container.\r\n\r\n# Required information\r\n\r\n * Distribution: Debian\r\n * Distribution version: 9\r\n * The output of\r\n   * `lxc-start --version`\r\n2.0.8\r\n   * `lxc-checkconfig` \r\n```\r\nlxc-checkconfig\r\nKernel configuration not found at /proc/config.gz; searching...\r\nKernel configuration found at /boot/config-4.9.0-3-amd64\r\n--- Namespaces ---\r\nNamespaces: enabled\r\nUtsname namespace: enabled\r\nIpc namespace: enabled\r\nPid namespace: enabled\r\nUser namespace: enabled\r\nWarning: newuidmap is not setuid-root\r\nWarning: newgidmap is not setuid-root\r\nNetwork namespace: enabled\r\n\r\n--- Control groups ---\r\nCgroup: enabled\r\nCgroup clone_children flag: enabled\r\nCgroup device: enabled\r\nCgroup sched: enabled\r\nCgroup cpu account: enabled\r\nCgroup memory controller: enabled\r\nCgroup cpuset: enabled\r\n\r\n--- Misc ---\r\nVeth pair device: enabled\r\nMacvlan: enabled\r\nVlan: enabled\r\nBridges: enabled\r\nAdvanced netfilter: enabled\r\nCONFIG_NF_NAT_IPV4: enabled\r\nCONFIG_NF_NAT_IPV6: enabled\r\nCONFIG_IP_NF_TARGET_MASQUERADE: enabled\r\nCONFIG_IP6_NF_TARGET_MASQUERADE: enabled\r\nCONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled\r\nFUSE (for use with lxcfs): enabled\r\n\r\n--- Checkpoint/Restore ---\r\ncheckpoint restore: enabled\r\nCONFIG_FHANDLE: enabled\r\nCONFIG_EVENTFD: enabled\r\nCONFIG_EPOLL: enabled\r\nCONFIG_UNIX_DIAG: enabled\r\nCONFIG_INET_DIAG: enabled\r\nCONFIG_PACKET_DIAG: enabled\r\nCONFIG_NETLINK_DIAG: enabled\r\nFile capabilities: enabled\r\n\r\nNote : Before booting a new kernel, you can check its configuration\r\nusage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig\r\n```\r\n   * `uname -a`\r\n\r\n```\r\nroot@vz01:~# uname -a\r\nLinux vz01 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux\r\n```\r\n   * `cat /proc/self/cgroup`\r\n```\r\nroot@vz01:/sys/fs/cgroup/cpu/lxc# cat /proc/self/cgroup\r\n10:blkio:/user.slice/user-0.slice/session-39.scope\r\n9:pids:/user.slice/user-0.slice/session-39.scope\r\n8:memory:/user.slice/user-0.slice/session-39.scope\r\n7:cpuset:/\r\n6:perf_event:/\r\n5:net_cls,net_prio:/\r\n4:freezer:/user/root/0\r\n3:devices:/user.slice/user-0.slice/session-39.scope\r\n2:cpu,cpuacct:/user.slice/user-0.slice/session-39.scope\r\n1:name=systemd:/user.slice/user-0.slice/session-39.scope\r\n```\r\n   * `cat /proc/1/mounts`\r\n```\r\nroot@vz01:/sys/fs/cgroup/cpu/lxc# cat /proc/1/mounts\r\nsysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0\r\nproc /proc proc rw,nosuid,nodev,noexec,relatime 0 0\r\nudev /dev devtmpfs rw,nosuid,relatime,size=1974912k,nr_inodes=493728,mode=755 0 0\r\ndevpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0\r\ntmpfs /run tmpfs rw,nosuid,noexec,relatime,size=397320k,mode=755 0 0\r\n/dev/sda1 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0\r\nsecurityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0\r\ntmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0\r\ntmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0\r\ntmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0\r\ncgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0\r\npstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0\r\ncgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0\r\ncgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0\r\ncgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0\r\ncgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0\r\ncgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0\r\ncgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset,clone_children 0 0\r\ncgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0\r\ncgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0\r\ncgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0\r\nsystemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10246 0 0\r\nmqueue /dev/mqueue mqueue rw,relatime 0 0\r\ndebugfs /sys/kernel/debug debugfs rw,relatime 0 0\r\nhugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0\r\nlxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0\r\nfusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0\r\ntmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=397316k,mode=700 0 0\r\n```\r\n# Issue description\r\n\r\nRestarting the lxc service causes orphaned cgroups in /sys/fs:\r\n\r\ne.g.\r\n\r\n```\r\nroot@vz01:/sys/fs/cgroup/cpu/lxc# ls -la\r\ntotal 0\r\ndrwxr-xr-x 22 root root    0 Aug  2 14:10 .\r\ndr-xr-xr-x  6 root root    0 Aug  1 15:40 ..\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 cgroup.clone_children\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 cgroup.procs\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.stat\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage_all\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage_percpu\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage_percpu_sys\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage_percpu_user\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage_sys\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpuacct.usage_user\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 cpu.cfs_period_us\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 cpu.cfs_quota_us\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 cpu.shares\r\n-r--r--r--  1 root root    0 Aug  1 15:41 cpu.stat\r\ndrwxrwxr-x  2 root 1258512 0 Aug  1 15:43 deb9build01\r\ndrwxrwxr-x  2 root 1258512 0 Aug  1 15:55 deb9build01-1\r\ndrwxrwxr-x  2 root 1258512 0 Aug  1 15:55 debian\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 notify_on_release\r\n-rw-r--r--  1 root root    0 Aug  1 15:41 tasks\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:30 test1-10\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:30 test1-11\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:51 test1-12\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:51 test1-13\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:51 test1-14\r\ndrwxr-xr-x  2 root root    0 Aug  2 11:53 test1-15\r\ndrwxr-xr-x  2 root root    0 Aug  2 11:53 test1-16\r\ndrwxr-xr-x  2 root root    0 Aug  2 11:53 test1-17\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:54 test1-18\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:11 test1-2\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:12 test1-3\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:26 test1-4\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:26 test1-5\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:26 test1-6\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:27 test1-7\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:27 test1-8\r\ndrwxrwxr-x  2 root 1258512 0 Aug  2 11:28 test1-9\r\n```\r\n\r\nA brief description of what failed or what could be improved.\r\n\r\n# Steps to reproduce\r\n\r\n 1. Step one\r\nRun ```systemctl restart lxc```\r\nthen ```lxc-stop -n container```\r\n 2. Step two\r\nCheck `/sys/fs/cgroup/cpu/lxc`\r\n 3. Step three\r\nYou will a cgroup left in that directory\r\n\r\n# Information to attach\r\n\r\n If you then try to delete the orphaned cgroups, most can be deleted, but some are not removal:\r\n\r\n```\r\nroot@vz01:/sys/fs/cgroup/cpu/lxc# find /sys/fs/cgroup/*/lxc/test1/ -type d |   tac  | xargs rmdir\r\nrmdir: failed to remove '/sys/fs/cgroup/pids/lxc/test1/system.slice/proc-sysrqx2dtrigger.mount': No such file or directory\r\nrmdir: failed to remove '/sys/fs/cgroup/pids/lxc/test1/system.slice': Device or resource busy\r\nrmdir: failed to remove '/sys/fs/cgroup/pids/lxc/test1/': Device or resource busy\r\nrmdir: failed to remove '/sys/fs/cgroup/devices/lxc/test1/system.slice/proc-sysrqx2dtrigger.mount': No such file or directory\r\nrmdir: failed to remove '/sys/fs/cgroup/devices/lxc/test1/system.slice': Device or resource busy\r\nrmdir: failed to remove '/sys/fs/cgroup/devices/lxc/test1/': Device or resource busy\r\n```\r\n\r\nThis seems to be something to do with /sys/fs/cgroup/pids/lxc/test1/system.slice/proc-sysrqx2dtrigger.mount\r\n\r\nExample test1 container config:\r\n\r\n```\r\n# Distribution configuration\r\nlxc.include = /usr/share/lxc/config/debian.common.conf\r\nlxc.include = /usr/share/lxc/config/debian.userns.conf\r\nlxc.arch = x86_64\r\n\r\n# Container specific configuration\r\nlxc.cgroup.memory.limit_in_bytes = 512M\r\nlxc.cgroup.memory.memsw.limit_in_bytes = 1G\r\nlxc.start.auto = 1\r\nlxc.aa_allow_incomplete = 1\r\nlxc.id_map = u 0 1258512 65536\r\nlxc.id_map = g 0 1258512 65536\r\nlxc.rootfs = /dev/VolGroup00/test1\r\nlxc.rootfs.backend = lvm\r\nlxc.utsname = test1\r\n```"},"comment":{"url":"https://api.github.com/repos/lxc/lxc/issues/comments/320215854","html_url":"https://github.com/lxc/lxc/issues/1726#issuecomment-320215854","issue_url":"https://api.github.com/repos/lxc/lxc/issues/1726","id":320215854,"user":{"login":"tomponline","id":280181,"avatar_url":"https://avatars3.githubusercontent.com/u/280181?v=4","gravatar_id":"","url":"https://api.github.com/users/tomponline","html_url":"https://github.com/tomponline","followers_url":"https://api.github.com/users/tomponline/followers","following_url":"https://api.github.com/users/tomponline/following{/other_user}","gists_url":"https://api.github.com/users/tomponline/gists{/gist_id}","starred_url":"https://api.github.com/users/tomponline/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomponline/subscriptions","organizations_url":"https://api.github.com/users/tomponline/orgs","repos_url":"https://api.github.com/users/tomponline/repos","events_url":"https://api.github.com/users/tomponline/events{/privacy}","received_events_url":"https://api.github.com/users/tomponline/received_events","type":"User","site_admin":false},"created_at":"2017-08-04T10:30:35Z","updated_at":"2017-08-04T10:30:35Z","body":"Awesome thanks @brauner !"}}
{"actor":{"display_login":"tomponline"}}