{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/contrib/issues/2444","repository_url":"https://api.github.com/repos/kubernetes/contrib","labels_url":"https://api.github.com/repos/kubernetes/contrib/issues/2444/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/contrib/issues/2444/comments","events_url":"https://api.github.com/repos/kubernetes/contrib/issues/2444/events","html_url":"https://github.com/kubernetes/contrib/issues/2444","id":211681438,"number":2444,"title":"Nginx-Ingress: Docker Registry behind Ingress - docker push don't work","user":{"login":"muffin87","id":22658582,"avatar_url":"https://avatars1.githubusercontent.com/u/22658582?v=4","gravatar_id":"","url":"https://api.github.com/users/muffin87","html_url":"https://github.com/muffin87","followers_url":"https://api.github.com/users/muffin87/followers","following_url":"https://api.github.com/users/muffin87/following{/other_user}","gists_url":"https://api.github.com/users/muffin87/gists{/gist_id}","starred_url":"https://api.github.com/users/muffin87/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/muffin87/subscriptions","organizations_url":"https://api.github.com/users/muffin87/orgs","repos_url":"https://api.github.com/users/muffin87/repos","events_url":"https://api.github.com/users/muffin87/events{/privacy}","received_events_url":"https://api.github.com/users/muffin87/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2017-03-03T12:58:31Z","updated_at":"2017-11-27T01:02:11Z","closed_at":"2017-03-04T11:56:10Z","author_association":"NONE","body":"Hi everyone,\r\n\r\nim running K8s version 1.5.2 on CentOS.\r\nI wanted to deploy a vanilla docker registry and expose the service with Nginx Ingress controller.\r\n\r\n# Issue\r\n\r\n`docker push` does not work with the nginx-ingress-controller in front of the docker-registry.\r\n\r\nIt seems to me that something in the nginx-ingress-controller config is blocking the `docker push` payload upload (tar file).\r\n\r\nAny idea whats happening here and how I can solve this?\r\n\r\n# Error Message:\r\n\r\ndocker-client:\r\n`Error: Status 404 trying to push repository nginx: \"404 page not found\\n\"`\r\n\r\ndockerd\r\n`Upload failed: EOF`\r\n\r\n```\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.177922934Z\" level=debug msg=\"Pushing layer: sha256:809c8c0dd73c9081a8ab9795b5fb9beab60c7f602a0886dbbdf48e939d608163\"\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.401819584Z\" level=debug msg=\"Assembling tar data for d388be1b57d4bbb37c16836b3fafba89e433a413f7448680d6992ecfe4f07089\"\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.416382683Z\" level=debug msg=\"Assembling tar data for 6780c17ba8c81862656e46d0812c0eaf7fc2eb55447cf1f71b1e3e64e548256b\"\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.416555479Z\" level=debug msg=\"Assembling tar data for 6c84bf0c3c99e5454f5536068db727174c4dea0add6c7c54052604dbbba4a46a\"\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.519050420Z\" level=error msg=\"Upload failed, retrying: EOF\"\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.537034379Z\" level=error msg=\"Upload failed, retrying: EOF\"\r\nMar  3 11:29:51 moby root: time=\"2017-03-03T11:29:51.541124148Z\" level=error msg=\"Upload failed, retrying: EOF\"\r\nMar  3 11:29:53 moby root: time=\"2017-03-03T11:29:53.983465337Z\" level=debug msg=\"Pushing layer: sha256:e8fa134cb7b8a5e2943c9ea6e48a90c9f7fc80fd26780ce9d74bbab927a6cfc5\"\r\nMar  3 11:29:54 moby root: time=\"2017-03-03T11:29:54.227004087Z\" level=debug msg=\"Assembling tar data for b3c45395c4b867925ed531458101d8c5cb8c5462f06a678c8132f06e2db8c588\"\r\nMar  3 11:29:54 moby root: time=\"2017-03-03T11:29:54.345157833Z\" level=error msg=\"Upload failed, retrying: EOF\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.520650325Z\" level=debug msg=\"Pushing layer: sha256:7cbcbac42c44c6c38559e5df3a494f44987333c8023a40fec48df2fce1fc146b\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.537773800Z\" level=debug msg=\"Pushing layer: sha256:e8d45b8ab3ca8a83ba4e340f4b901c593da00f7f2bb068937a8b32b3beb606af\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.541707653Z\" level=debug msg=\"Pushing layer: sha256:809c8c0dd73c9081a8ab9795b5fb9beab60c7f602a0886dbbdf48e939d608163\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.744771310Z\" level=debug msg=\"Assembling tar data for 6c84bf0c3c99e5454f5536068db727174c4dea0add6c7c54052604dbbba4a46a\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.756433036Z\" level=debug msg=\"Assembling tar data for 6780c17ba8c81862656e46d0812c0eaf7fc2eb55447cf1f71b1e3e64e548256b\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.760178212Z\" level=debug msg=\"Assembling tar data for d388be1b57d4bbb37c16836b3fafba89e433a413f7448680d6992ecfe4f07089\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.851262730Z\" level=error msg=\"Upload failed: EOF\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.856225668Z\" level=error msg=\"Upload failed: EOF\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.863372727Z\" level=error msg=\"Upload failed: EOF\"\r\nMar  3 11:30:11 moby root: time=\"2017-03-03T11:30:11.863480799Z\" level=error msg=\"Attempting next endpoint for push after error: EOF\"\r\n```\r\n\r\nnginx-ingress log:\r\n\r\n```\r\n<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] \"HEAD /v2/nginx/blobs/sha256:ba1ed986f6a0df1b3fa02fe8cf76e45e7f7ae248c2a2d9f6ce901585c38d5a79 HTTP/1.1\" 404 0 \"-\" \"docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\x5C(darwin\\x5C))\" \"-\"\r\n<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] \"POST /v2/nginx/blobs/uploads/ HTTP/1.1\" 202 0 \"-\" \"docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\x5C(darwin\\x5C))\" \"-\"\r\n<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] \"POST /v2/nginx/blobs/uploads/ HTTP/1.1\" 202 0 \"-\" \"docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\x5C(darwin\\x5C))\" \"-\"\r\n<censored> - ucpapp [03/Mar/2017:11:29:21 +0000] \"POST /v2/nginx/blobs/uploads/ HTTP/1.1\" 202 0 \"-\" \"docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\x5C(darwin\\x5C))\" \"-\"\r\n<censored> - ucpapp [03/Mar/2017:11:29:21 +0000] \"HEAD /v2/nginx/blobs/sha256:a362ec94dcabb258895c0cde3905dafd6c40df5ffa4a668fdf6b175655774192 HTTP/1.1\" 404 0 \"-\" \"docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\x5C(darwin\\x5C))\" \"-\"\r\n```\r\n\r\n# Setup\r\n\r\ndockerd -> nginx-ingress -> k8s-service-object -> nginx-for-auth -> registry\r\n\r\n* `docker push` through the nginx results in error.\r\n* Skipping the nginx-ingress and pushing to the `k8s-service-object` via NodePort works fine\r\n* `docker login` works fine\r\n* `docker push` manifest files end up in the registry fine\r\n\r\n# Configs\r\n\r\nrendered nginx-config\r\n\r\n```\r\nupstream tools-docker-registry-registry.<censored>-docker-registry {\r\n\r\n  server 10.254.4.9:80;\r\n}\r\n\r\nserver {\r\n  listen 80;\r\n\r\n  server_name registry.<censored>;\r\n\r\n  location / {\r\n    proxy_http_version 1.1;\r\n\r\n    proxy_connect_timeout 60s;\r\n    proxy_read_timeout 60s;\r\n    client_max_body_size 1m;\r\n    proxy_set_header Host $host;\r\n    proxy_set_header X-Real-IP $remote_addr;\r\n    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\r\n    proxy_set_header X-Forwarded-Host $host;\r\n    proxy_set_header X-Forwarded-Port $server_port;\r\n    proxy_set_header X-Forwarded-Proto $scheme;\r\n\r\n    proxy_buffering on;\r\n\r\n    proxy_pass http://tools-docker-registry-registry.<censored>-docker-registry;\r\n\r\n  }\r\n}\r\n```\r\n\r\nregistry ingress conf\r\n\r\n```\r\napiVersion: extensions/v1beta1\r\nkind: Ingress\r\nmetadata:\r\n  name: docker-registry\r\n  namespace: tools\r\nspec:\r\n  rules:\r\n    - host: registry.<censored>\r\n      http:\r\n        paths:\r\n        - backend:\r\n            serviceName: docker-registry\r\n            servicePort: 80\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/contrib/issues/comments/347054049","html_url":"https://github.com/kubernetes/contrib/issues/2444#issuecomment-347054049","issue_url":"https://api.github.com/repos/kubernetes/contrib/issues/2444","id":347054049,"user":{"login":"shenshouer","id":1491181,"avatar_url":"https://avatars1.githubusercontent.com/u/1491181?v=4","gravatar_id":"","url":"https://api.github.com/users/shenshouer","html_url":"https://github.com/shenshouer","followers_url":"https://api.github.com/users/shenshouer/followers","following_url":"https://api.github.com/users/shenshouer/following{/other_user}","gists_url":"https://api.github.com/users/shenshouer/gists{/gist_id}","starred_url":"https://api.github.com/users/shenshouer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shenshouer/subscriptions","organizations_url":"https://api.github.com/users/shenshouer/orgs","repos_url":"https://api.github.com/users/shenshouer/repos","events_url":"https://api.github.com/users/shenshouer/events{/privacy}","received_events_url":"https://api.github.com/users/shenshouer/received_events","type":"User","site_admin":false},"created_at":"2017-11-27T01:02:11Z","updated_at":"2017-11-27T01:02:11Z","author_association":"NONE","body":"I think the `ingress.kubernetes.io/proxy-body-size: \"0\"` int annotations of nginx ingress doesn`t work fine!"}}
{"actor":{"display_login":"shenshouer"}}