{"action":"closed","issue":{"url":"https://api.github.com/repos/moby/moby/issues/35257","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/35257/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/35257/comments","events_url":"https://api.github.com/repos/moby/moby/issues/35257/events","html_url":"https://github.com/moby/moby/issues/35257","id":267090059,"number":35257,"title":"Docker overlay storage driver unable to delete directories from underlying image layer","user":{"login":"g000444555","id":13013939,"avatar_url":"https://avatars2.githubusercontent.com/u/13013939?v=4","gravatar_id":"","url":"https://api.github.com/users/g000444555","html_url":"https://github.com/g000444555","followers_url":"https://api.github.com/users/g000444555/followers","following_url":"https://api.github.com/users/g000444555/following{/other_user}","gists_url":"https://api.github.com/users/g000444555/gists{/gist_id}","starred_url":"https://api.github.com/users/g000444555/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/g000444555/subscriptions","organizations_url":"https://api.github.com/users/g000444555/orgs","repos_url":"https://api.github.com/users/g000444555/repos","events_url":"https://api.github.com/users/g000444555/events{/privacy}","received_events_url":"https://api.github.com/users/g000444555/received_events","type":"User","site_admin":false},"labels":[{"id":149275894,"url":"https://api.github.com/repos/moby/moby/labels/area/storage/overlay","name":"area/storage/overlay","color":"fbca04","default":false},{"id":182417459,"url":"https://api.github.com/repos/moby/moby/labels/status/more-info-needed","name":"status/more-info-needed","color":"C6A49A","default":false},{"id":712410756,"url":"https://api.github.com/repos/moby/moby/labels/version/17.10","name":"version/17.10","color":"ededed","default":false},{"id":375241375,"url":"https://api.github.com/repos/moby/moby/labels/version/unsupported","name":"version/unsupported","color":"ededed","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-10-20T07:44:52Z","updated_at":"2017-10-21T11:20:18Z","closed_at":"2017-10-21T11:20:18Z","author_association":"NONE","body":"---------------------------------------------------\r\nBUG REPORT INFORMATION\r\n---------------------------------------------------\r\n\r\n**Description**\r\n\r\nThe issue is similar to [overlayfs: Can't delete file moved from base layer to newly created dir even on ext4 #9572](https://github.com/moby/moby/issues/9572)\r\n\r\nHowever, the route cause isn't the same, because `d_type` is enabled in my case.\r\n\r\n**Steps to reproduce the issue:**\r\n\r\nFor example, manually running the container and trying to delete a directory results in the following error\r\n\r\n    $ docker run --rm -it arm64v8/ubuntu:latest bash\r\n    root@95f2e1ed7888:/# rm -rf /usr/share/doc\r\n    rm: cannot remove '/usr/share/doc/sed': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/libpam-modules': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/util-linux': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/mawk': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/gnupg': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/apt': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/libreadline6': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/adduser/examples/adduser.local.conf.examples': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/passwd': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/mount': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/procps': Directory not empty\r\n\r\nIf the directory is empty, then `rm -rf` won't complain but won't delete it either, for example\r\n\r\n    root@95f2e1ed7888:/# ls -la /usr/share/doc/zlib1g\r\n    total 8\r\n    drwxr-xr-x 1 root root 4096 Oct 20 07:29 .\r\n    drwxr-xr-x 1 root root 4096 Oct 20 07:29 ..\r\n    root@95f2e1ed7888:/# rm -rf /usr/share/doc/zlib1g\r\n    root@95f2e1ed7888:/# ls -la /usr/share/doc/zlib1g\r\n    total 8\r\n    drwxr-xr-x 1 root root 4096 Oct 20 07:29 .\r\n    drwxr-xr-x 1 root root 4096 Oct 20 07:29 ..\r\n\r\nOn the other hand, `rmdir` for an empty directory reports the following error\r\n\r\n    root@95f2e1ed7888:/# ls -la /usr/share/doc/zlib1g\r\n    total 8\r\n    drwxr-xr-x 1 root root 4096 Oct 20 07:29 .\r\n    drwxr-xr-x 1 root root 4096 Oct 20 07:29 ..\r\n    root@95f2e1ed7888:/# rmdir /usr/share/doc/zlib1g\r\n    rmdir: failed to remove '/usr/share/doc/zlib1g': Invalid argument\r\n\r\n**Output of `docker version`:**\r\n\r\n    $ docker version\r\n    Client:\r\n     Version:      17.10.0-ce\r\n     API version:  1.33\r\n     Go version:   go1.9.1\r\n     Git commit:   f4ffd2511c\r\n     Built:        Thu Oct 19 00:10:43 2017\r\n     OS/Arch:      linux/arm64\r\n    \r\n    Server:\r\n     Version:      17.10.0-ce\r\n     API version:  1.33 (minimum version 1.12)\r\n     Go version:   go1.9.1\r\n     Git commit:   f4ffd2511c\r\n     Built:        Thu Oct 19 00:11:05 2017\r\n     OS/Arch:      linux/arm64\r\n     Experimental: false\r\n\r\n**Output of `docker info`:**\r\n\r\n    $ docker info\r\n    Containers: 7\r\n     Running: 1\r\n     Paused: 0\r\n     Stopped: 6\r\n    Images: 95\r\n    Server Version: 17.10.0-ce\r\n    Storage Driver: overlay\r\n     Backing Filesystem: extfs\r\n     Supports d_type: true\r\n    Logging Driver: json-file\r\n    Cgroup Driver: cgroupfs\r\n    Plugins:\r\n     Volume: local\r\n     Network: bridge host macvlan null overlay\r\n     Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\n    Swarm: active\r\n     NodeID: mjm8vu1ixmw1ls25dk7gomiva\r\n     Is Manager: true\r\n     ClusterID: lnspgxtrca8d4lm3kfv7zbz77\r\n     Managers: 3\r\n     Nodes: 3\r\n     Orchestration:\r\n      Task History Retention Limit: 5\r\n     Raft:\r\n      Snapshot Interval: 10000\r\n      Number of Old Snapshots to Retain: 0\r\n      Heartbeat Tick: 1\r\n      Election Tick: 3\r\n     Dispatcher:\r\n      Heartbeat Period: 5 seconds\r\n     CA Configuration:\r\n      Expiry Duration: 3 months\r\n      Force Rotate: 0\r\n     Autolock Managers: false\r\n     Root Rotation In Progress: false\r\n     Node Address: 192.168.1.11\r\n     Manager Addresses:\r\n      192.168.1.11:2377\r\n      192.168.1.12:2377\r\n      192.168.1.13:2377\r\n    Runtimes: runc\r\n    Default Runtime: runc\r\n    Init Binary: docker-init\r\n    containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0\r\n    runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d\r\n    init version: 949e6fa\r\n    Security Options:\r\n     seccomp\r\n      Profile: default\r\n    Kernel Version: 3.14.79-27-ARCH\r\n    Operating System: Arch Linux ARM\r\n    OSType: linux\r\n    Architecture: aarch64\r\n    CPUs: 4\r\n    Total Memory: 1.928GiB\r\n    Name: alarm-docker-manager-host\r\n    ID: Y5ZZ:5G3O:KIE2:MSQX:7MBB:XWOZ:CYLL:2SSY:GTMB:YCBG:4TTT:EJVN\r\n    Docker Root Dir: /var/lib/docker\r\n    Debug Mode (client): false\r\n    Debug Mode (server): false\r\n    Registry: https://index.docker.io/v1/\r\n    Experimental: false\r\n    Insecure Registries:\r\n     127.0.0.0/8\r\n    Live Restore Enabled: false\r\n\r\nNot sure if this is relevant, however note that the underlying partition is encrypted with LUKS on LVM.\r\n\r\nDigging into the issue further, the permissions of the directory I try to delete are ok\r\n\r\n    root@95f2e1ed7888:/# ls -lah /usr/share\r\n    total 160K\r\n    drwxr-xr-x  1 root root 4.0K Oct 19 05:39 .\r\n    drwxr-xr-x  1 root root 4.0K Oct 19 05:39 ..\r\n    drwxr-xr-x  2 root root 4.0K Oct 19 05:39 adduser\r\n    drwxr-xr-x  3 root root 4.0K Oct 19 05:39 apport\r\n    drwxr-xr-x  2 root root 4.0K Oct 19 05:39 base-files\r\n    drwxr-xr-x  2 root root 4.0K Oct 19 05:39 base-passwd\r\n    drwxr-xr-x  3 root root 4.0K Oct 19 05:39 bash-completion\r\n    drwxr-xr-x  6 root root 4.0K Oct 19 05:39 bug\r\n    drwxr-xr-x  2 root root 4.0K Oct 19 05:39 common-licenses\r\n    drwxr-xr-x  4 root root 4.0K Oct 19 05:39 dbus-1\r\n    drwxr-xr-x  2 root root 4.0K Oct 19 05:39 debconf\r\n    drwxr-xr-x  2 root root 4.0K Oct 19 05:39 debianutils\r\n    drwxr-xr-x  2 root root 4.0K Apr 12  2016 dict\r\n    drwxr-xr-x  1 root root 4.0K Oct 20 07:29 doc\r\n    ...\r\n\r\nAlso, the directory is not being used by another process\r\n\r\n    root@95f2e1ed7888:/# lsof /usr/share/doc\r\n\r\nMoreover, running the container with sudo has same effect\r\n\r\n    $ sudo docker run --rm -it arm64v8/ubuntu:latest rm -rf /usr/share/doc\r\n    rm: cannot remove '/usr/share/doc/sed': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/libpam-modules': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/util-linux': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/mawk': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/gnupg': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/apt': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/libreadline6': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/adduser/examples/adduser.local.conf.examples': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/passwd': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/mount': Directory not empty\r\n    rm: cannot remove '/usr/share/doc/procps': Directory not empty\r\n\r\nFinally, more debugging information in regards to this issue can be found at https://stackoverflow.com/questions/46824425/docker-is-unable-to-delete-a-directory-when-building-images\r\n"}}
{"actor":{"display_login":"thaJeztah"}}