{"action":"created","issue":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/943","repository_url":"https://api.github.com/repos/aws/amazon-ecs-agent","labels_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/943/labels{/name}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/943/comments","events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/943/events","html_url":"https://github.com/aws/amazon-ecs-agent/issues/943","id":252530542,"number":943,"title":"CannotPullContainerError, ECS Service task fails to start","user":{"login":"L226","id":7380860,"avatar_url":"https://avatars1.githubusercontent.com/u/7380860?v=4","gravatar_id":"","url":"https://api.github.com/users/L226","html_url":"https://github.com/L226","followers_url":"https://api.github.com/users/L226/followers","following_url":"https://api.github.com/users/L226/following{/other_user}","gists_url":"https://api.github.com/users/L226/gists{/gist_id}","starred_url":"https://api.github.com/users/L226/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/L226/subscriptions","organizations_url":"https://api.github.com/users/L226/orgs","repos_url":"https://api.github.com/users/L226/repos","events_url":"https://api.github.com/users/L226/events{/privacy}","received_events_url":"https://api.github.com/users/L226/received_events","type":"User","site_admin":false},"labels":[{"id":205830716,"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/more%20info%20needed","name":"more info needed","color":"fbca04","default":false},{"id":593677626,"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/scope/ECR","name":"scope/ECR","color":"c5def5","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-08-24T08:49:57Z","updated_at":"2017-09-12T00:04:01Z","closed_at":"2017-09-12T00:04:01Z","author_association":"NONE","body":"So here's a funny one, repeated CannotPullContainerError: API error (404) for one of our services. We have multiple containers running in production, attempted to deploy another today using our CI stack. Keep getting the above error even though we're using the exact same process for our existing services.\r\n\r\n1. Application docker image built on CI stack\r\n2. Image pushed to ECR and tagged with the deploy version\r\n3. CI stack crafts task definition from provided JSON skeleton, secret envvars from S3, terraform outputs and deploy version\r\n4. New task definition sent to ECS service\r\n5. ECS attempts to run the task with latest definition\r\n6. Repeated failures to pull\r\n\r\nWe have a framework to handle all of this, allowing us to drop in new service app code and just push onto new ECS services. This system has been working fine so far....\r\n\r\nThings I have tried: \r\n\r\n- running the 'bad' task definition on a different ECS cluster - fails to deploy\r\n- running a known good task definition on the 'bad' ECS cluster - task runs successfully\r\n- manually crafting task definition JSON identical to the 'bad' one but pointing to a known good image - task runs successfully\r\n- manually pushing a known good image tagged with the deploy version of the 'bad' task definition to that task's ECR - task fails to deploy\r\n- terminating the EC2 instance and redeploying the ECS service task - still fails to deploy\r\n- using a bastion to ssh into the instance and trying a manual pull of suspect image - fails\r\n- using a bastion to ssh into the instance and trying a manual pull of some other image (alpine) - success\r\n\r\nI can also successfully pull the suspect image locally. I have also tried using a completely new ECR for the suspect image.\r\n\r\nThe suspect image is only 1.27GB, being pulled onto an m3.medium with nothing else running on it.\r\n\r\nIs this an issue with the docker image (runs fine locally and on CI), with the task definition (runs fine for other images) or with AWS (?) ?\r\n\r\nOur ECS clusters and services are managed with terraform - we have other systems running with the exact same configuration. IAM access is setup correctly.\r\n\r\nVersions:\r\n- ECS AMI - ami-c3233ba0\r\n- docker - 17.03.1-ce\r\n- ecs-agent - 1.14.3\r\n- terraform - v0.9.5\r\n- instance - m3.medium\r\n\r\nThe app is python 2.7.\r\n\r\nAny suggestions?\r\n\r\ntask def:\r\n```json\r\n{\r\n  \"requiresAttributes\": [\r\n    {\r\n      \"value\": null,\r\n      \"name\": \"com.amazonaws.ecs.capability.logging-driver.syslog\",\r\n      \"targetId\": null,\r\n      \"targetType\": null\r\n    },\r\n    {\r\n      \"value\": null,\r\n      \"name\": \"com.amazonaws.ecs.capability.ecr-auth\",\r\n      \"targetId\": null,\r\n      \"targetType\": null\r\n    },\r\n    {\r\n      \"value\": null,\r\n      \"name\": \"com.amazonaws.ecs.capability.task-iam-role\",\r\n      \"targetId\": null,\r\n      \"targetType\": null\r\n    },\r\n    {\r\n      \"value\": null,\r\n      \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.19\",\r\n      \"targetId\": null,\r\n      \"targetType\": null\r\n    }\r\n  ],\r\n  \"taskDefinitionArn\": \"arn:aws:ecs:ap-southeast-2:123456789:task-definition\",\r\n  \"networkMode\": \"bridge\",\r\n  \"status\": \"ACTIVE\",\r\n  \"revision\": 11,\r\n  \"taskRoleArn\": \"arn:aws:iam::123456789:role/ECSRole\",\r\n  \"containerDefinitions\": [\r\n    {\r\n      \"volumesFrom\": [],\r\n      \"memory\": 2048,\r\n      \"extraHosts\": null,\r\n      \"dnsServers\": null,\r\n      \"disableNetworking\": null,\r\n      \"dnsSearchDomains\": null,\r\n      \"portMappings\": [],\r\n      \"hostname\": null,\r\n      \"essential\": true,\r\n      \"entryPoint\": null,\r\n      \"mountPoints\": [],\r\n      \"name\": \"appname\",\r\n      \"ulimits\": null,\r\n      \"dockerSecurityOptions\": null,\r\n      \"environment\": [\r\n        {\r\n          \"name\": \"AWS_REGION\",\r\n          \"value\": \"ap-southeast-2\"\r\n        }\r\n      ],\r\n      \"links\": null,\r\n      \"workingDirectory\": null,\r\n      \"readonlyRootFilesystem\": null,\r\n      \"image\": \"xxxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/repo_name:deploy_version\",\r\n      \"command\": null,\r\n      \"user\": null,\r\n      \"dockerLabels\": null,\r\n      \"logConfiguration\": {\r\n        \"logDriver\": \"syslog\",\r\n        \"options\": {\r\n          \"syslog-address\": \"udp://logs5.papertrailapp.com:xxxxx\"\r\n        }\r\n      },\r\n      \"cpu\": 1024,\r\n      \"privileged\": null,\r\n      \"memoryReservation\": null\r\n    }\r\n  ],\r\n  \"placementConstraints\": [],\r\n  \"volumes\": [],\r\n  \"family\": \"appname\"\r\n}\r\n```\r\n\r\necs-agent log:\r\n```bash\r\n2017-08-24T08:06:50Z [INFO] Error transitioning container module=\"TaskEngine\" task=\"appname:11 arn:aws:ecs:ap-southeast-2:123456789:task/717fe788-eadb-4cfa-a33c-d9528b60834b, Status: (NONE->RUNNING) Containers: [appname (NONE->RUNNING),]\" container=\"appname(xxxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/image_name:deploy_version) (NONE->RUNNING)\" state=\"PULLED\" error=\"API error (404): repository xxxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/image_name not found: does not exist or no pull access\r\n```"},"comment":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/comments/328692387","html_url":"https://github.com/aws/amazon-ecs-agent/issues/943#issuecomment-328692387","issue_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/943","id":328692387,"user":{"login":"L226","id":7380860,"avatar_url":"https://avatars1.githubusercontent.com/u/7380860?v=4","gravatar_id":"","url":"https://api.github.com/users/L226","html_url":"https://github.com/L226","followers_url":"https://api.github.com/users/L226/followers","following_url":"https://api.github.com/users/L226/following{/other_user}","gists_url":"https://api.github.com/users/L226/gists{/gist_id}","starred_url":"https://api.github.com/users/L226/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/L226/subscriptions","organizations_url":"https://api.github.com/users/L226/orgs","repos_url":"https://api.github.com/users/L226/repos","events_url":"https://api.github.com/users/L226/events{/privacy}","received_events_url":"https://api.github.com/users/L226/received_events","type":"User","site_admin":false},"created_at":"2017-09-12T00:04:01Z","updated_at":"2017-09-12T00:04:01Z","author_association":"NONE","body":"This seems to have been an issue with our CI workflow. Closing for now. Thanks for your help!"}}
{"actor":{"display_login":"L226"}}