{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/34218","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/34218/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/34218/comments","events_url":"https://api.github.com/repos/moby/moby/issues/34218/events","html_url":"https://github.com/moby/moby/pull/34218","id":244817042,"number":34218,"title":"[WIP] Close readclosers returned by DecompressStream","user":{"login":"yosifkit","id":896423,"avatar_url":"https://avatars3.githubusercontent.com/u/896423?v=4","gravatar_id":"","url":"https://api.github.com/users/yosifkit","html_url":"https://github.com/yosifkit","followers_url":"https://api.github.com/users/yosifkit/followers","following_url":"https://api.github.com/users/yosifkit/following{/other_user}","gists_url":"https://api.github.com/users/yosifkit/gists{/gist_id}","starred_url":"https://api.github.com/users/yosifkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosifkit/subscriptions","organizations_url":"https://api.github.com/users/yosifkit/orgs","repos_url":"https://api.github.com/users/yosifkit/repos","events_url":"https://api.github.com/users/yosifkit/events{/privacy}","received_events_url":"https://api.github.com/users/yosifkit/received_events","type":"User","site_admin":false},"labels":[{"id":175375407,"url":"https://api.github.com/repos/moby/moby/labels/status/2-code-review","name":"status/2-code-review","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars1.githubusercontent.com/u/585223?v=4","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},"assignees":[{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars1.githubusercontent.com/u/585223?v=4","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},{"login":"mlaventure","id":16065150,"avatar_url":"https://avatars1.githubusercontent.com/u/16065150?v=4","gravatar_id":"","url":"https://api.github.com/users/mlaventure","html_url":"https://github.com/mlaventure","followers_url":"https://api.github.com/users/mlaventure/followers","following_url":"https://api.github.com/users/mlaventure/following{/other_user}","gists_url":"https://api.github.com/users/mlaventure/gists{/gist_id}","starred_url":"https://api.github.com/users/mlaventure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlaventure/subscriptions","organizations_url":"https://api.github.com/users/mlaventure/orgs","repos_url":"https://api.github.com/users/mlaventure/repos","events_url":"https://api.github.com/users/mlaventure/events{/privacy}","received_events_url":"https://api.github.com/users/mlaventure/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2017-07-22T02:06:55Z","updated_at":"2017-11-10T01:20:33Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/moby/moby/pulls/34218","html_url":"https://github.com/moby/moby/pull/34218","diff_url":"https://github.com/moby/moby/pull/34218.diff","patch_url":"https://github.com/moby/moby/pull/34218.patch"},"body":"**- What I did**\r\n\r\n@tianon mentioned that he has seen this for quite a while, but never bothered to find the root cause. I build many images on my local machine (every Linux amd64 image in the official images) and, over time, docker leaves many `xz` processes waiting.\r\n\r\n```console\r\n$ ps aux | grep xz | grep -V grep\r\nroot       981  0.0  0.0  76052   884 ?        S    Jul19   0:00 xz -d -c -q\r\nroot      2973  0.0  0.0  76052   852 ?        S    Jul20   0:00 xz -d -c -q\r\nroot      7361  0.0  0.0  76052   840 ?        S    Jul20   0:00 xz -d -c -q\r\nroot      8006  0.0  0.0  76052   944 ?        S    Jul20   0:00 xz -d -c -q\r\nroot     17969  0.0  0.0  76052   856 ?        S    Jul18   0:00 xz -d -c -q\r\nroot     18836  0.0  0.0  76052   848 ?        S    Jul18   0:00 xz -d -c -q\r\nroot     19855  0.0  0.0  76052   860 ?        S    Jul18   0:00 xz -d -c -q\r\nroot     20062  0.0  0.0  18708   856 ?        S    Jul20   0:00 xz -d -c -q\r\nroot     27810  0.0  0.0  76052   864 ?        S    Jul20   0:00 xz -d -c -q\r\nroot     28826  0.0  0.0  18708   896 ?        S    Jul19   0:00 xz -d -c -q\r\nroot     32101  0.0  0.0  76052   912 ?        S    Jul19   0:00 xz -d -c -q\r\nroot     32759  0.0  0.0  76052   860 ?        S    Jul19   0:00 xz -d -c -q\r\n```\r\n\r\nYou can reproduce it by building an image that uses an `ADD` with an `xz` file:\r\n```console\r\n$ # get count of current xz processes\r\n$ pidof xz | wc -w\r\n12\r\n$ docker build --no-cache https://github.com/debuerreotype/docker-debian-artifacts.git#dist-amd64:jessie/slim\r\nSending build context to Docker daemon  16.25MB\r\nStep 1/3 : FROM scratch\r\n ---> \r\nStep 2/3 : ADD rootfs.tar.xz /\r\n ---> cab71017cf4e\r\nRemoving intermediate container a37f42abdd32\r\nStep 3/3 : CMD bash\r\n ---> Running in d5a8f1e3d97b\r\n ---> 36f80bc07497\r\nRemoving intermediate container d5a8f1e3d97b\r\nSuccessfully built 36f80bc07497\r\n$ pidof xz | wc -w\r\n13\r\n```\r\n\r\n**- How I did it**\r\nFound out who calls `xz -d -c -q` and found it in `pkg/archive/archive.go`, walked up the chain and found who receives a `ReadCloser` and doesn't `Close`.  This is the same pattern applied in these files:\r\n\r\n- [pkg/chrootarchive/archive.go](https://github.com/moby/moby/blob/4ac4c8ef4b06ffa8ccd4fd3ae0f76e4ca820c1bc/pkg/chrootarchive/archive.go#L61-L66)\r\n- [pkg/chrootarchive/diff_unix.go](https://github.com/moby/moby/blob/4ac4c8ef4b06ffa8ccd4fd3ae0f76e4ca820c1bc/pkg/chrootarchive/diff_unix.go#L88-L94)\r\n- [pkg/chrootarchive/diff_windows.go](https://github.com/moby/moby/blob/4ac4c8ef4b06ffa8ccd4fd3ae0f76e4ca820c1bc/pkg/chrootarchive/diff_windows.go#L24-L30)\r\n- [pkg/archive/archive_test.go](https://github.com/moby/moby/blob/4ac4c8ef4b06ffa8ccd4fd3ae0f76e4ca820c1bc/pkg/archive/archive_test.go#L108-L117)\r\n\r\n**- How to verify it**\r\nI tried applying this diff, restarted docker, and then ran the same `docker build` from above and it hangs on `ADD rootfs.tar.xz /`.\r\n\r\nI'd love some help to find where I am going wrong.  Feel free to push onto my branch if you'd like to help! \r\n\r\n**- Description for the changelog**\r\nClose ReadClosers to prevent `xz` zombies\r\n\r\n**- A picture of a cute animal (not mandatory but encouraged)**\r\n![](https://media.giphy.com/media/sueyjcZ4vSnQc/giphy.gif)\r\n"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/343345656","html_url":"https://github.com/moby/moby/pull/34218#issuecomment-343345656","issue_url":"https://api.github.com/repos/moby/moby/issues/34218","id":343345656,"user":{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars1.githubusercontent.com/u/585223?v=4","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},"created_at":"2017-11-10T01:20:33Z","updated_at":"2017-11-10T01:20:33Z","author_association":"MEMBER","body":"Green again. LGTM"}}
{"actor":{"display_login":"tonistiigi"}}