{"action":"created","issue":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/764","repository_url":"https://api.github.com/repos/aws/amazon-ecs-agent","labels_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/764/labels{/name}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/764/comments","events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/764/events","html_url":"https://github.com/aws/amazon-ecs-agent/issues/764","id":221300523,"number":764,"title":"ECS Agent filling up root drive on ECS-optimized instances","user":{"login":"cwndrws","id":1382553,"avatar_url":"https://avatars2.githubusercontent.com/u/1382553?v=3","gravatar_id":"","url":"https://api.github.com/users/cwndrws","html_url":"https://github.com/cwndrws","followers_url":"https://api.github.com/users/cwndrws/followers","following_url":"https://api.github.com/users/cwndrws/following{/other_user}","gists_url":"https://api.github.com/users/cwndrws/gists{/gist_id}","starred_url":"https://api.github.com/users/cwndrws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cwndrws/subscriptions","organizations_url":"https://api.github.com/users/cwndrws/orgs","repos_url":"https://api.github.com/users/cwndrws/repos","events_url":"https://api.github.com/users/cwndrws/events{/privacy}","received_events_url":"https://api.github.com/users/cwndrws/received_events","type":"User","site_admin":false},"labels":[{"id":205830716,"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/more%20info%20needed","name":"more info needed","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-04-12T15:26:40Z","updated_at":"2017-04-24T16:58:48Z","closed_at":"2017-04-24T16:58:48Z","body":"Yesterday, I started receiving alerts that some of my ECS container instances' root drives were filling up. I have a block device mapping configured for the `/dev/xvdcz` device, which Docker device mapper uses. That device looked normal, but `/dev/xvda1`, which is the root drive on these boxes was receiving a lot of writes. Using Datadog, I was able to determine the only thing writing at higher rate than normal was the ecs-agent container. \r\n\r\n![screen shot 2017-04-12 at 10 20 24 am](https://cloud.githubusercontent.com/assets/1382553/24965478/bf58ba70-1f69-11e7-8d99-e900fc37c4c1.png)\r\n\r\nUsing `du`, I narrowed the disk usage increase to `/var/lib/docker` but wasn't able to find out what the container was trying to write to. I am not certain what changed to cause this increase in disk usage. As a temporary fix, I am now mounting a larger EBS volume (100GB) to the root drive, but I would like to go back to the default 8GB root drive if possible. What can I do to further debug this disk usage problem."},"comment":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/comments/296743749","html_url":"https://github.com/aws/amazon-ecs-agent/issues/764#issuecomment-296743749","issue_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/764","id":296743749,"user":{"login":"cwndrws","id":1382553,"avatar_url":"https://avatars2.githubusercontent.com/u/1382553?v=3","gravatar_id":"","url":"https://api.github.com/users/cwndrws","html_url":"https://github.com/cwndrws","followers_url":"https://api.github.com/users/cwndrws/followers","following_url":"https://api.github.com/users/cwndrws/following{/other_user}","gists_url":"https://api.github.com/users/cwndrws/gists{/gist_id}","starred_url":"https://api.github.com/users/cwndrws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cwndrws/subscriptions","organizations_url":"https://api.github.com/users/cwndrws/orgs","repos_url":"https://api.github.com/users/cwndrws/repos","events_url":"https://api.github.com/users/cwndrws/events{/privacy}","received_events_url":"https://api.github.com/users/cwndrws/received_events","type":"User","site_admin":false},"created_at":"2017-04-24T16:58:48Z","updated_at":"2017-04-24T16:58:48Z","body":"@cabbruzzese Yes it did. I believe the solution I was given was the correct one. I don't believe it was caused by images being pulled, since these machines are running many containers that use the same image. The only images that would be pulled in the life time of these machines would be the ecs-agent image and the application image that is running on them. We do not update the application image often (maybe once or twice per week), so I don't believe the image pulling is the culprit here. I implemented the change to the `ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION` setting and it seems to be doing fine now."}}
{"actor":{"display_login":"cwndrws"}}