{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/kube-aws/issues/290","repository_url":"https://api.github.com/repos/coreos/kube-aws","labels_url":"https://api.github.com/repos/coreos/kube-aws/issues/290/labels{/name}","comments_url":"https://api.github.com/repos/coreos/kube-aws/issues/290/comments","events_url":"https://api.github.com/repos/coreos/kube-aws/issues/290/events","html_url":"https://github.com/coreos/kube-aws/issues/290","id":204434653,"number":290,"title":"latest pre-release 0.9.3 kube's private internal subnets route table uses igw instead of NAT","user":{"login":"cmcconnell1","id":14812439,"avatar_url":"https://avatars.githubusercontent.com/u/14812439?v=3","gravatar_id":"","url":"https://api.github.com/users/cmcconnell1","html_url":"https://github.com/cmcconnell1","followers_url":"https://api.github.com/users/cmcconnell1/followers","following_url":"https://api.github.com/users/cmcconnell1/following{/other_user}","gists_url":"https://api.github.com/users/cmcconnell1/gists{/gist_id}","starred_url":"https://api.github.com/users/cmcconnell1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmcconnell1/subscriptions","organizations_url":"https://api.github.com/users/cmcconnell1/orgs","repos_url":"https://api.github.com/users/cmcconnell1/repos","events_url":"https://api.github.com/users/cmcconnell1/events{/privacy}","received_events_url":"https://api.github.com/users/cmcconnell1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-01-31T21:53:17Z","updated_at":"2017-02-01T06:40:58Z","closed_at":null,"body":"Hello,\r\n\r\nAs of the pre-release version 0.9.3 and building kube-aws binary from forked master branch, it seems as though the networking/routes for using private, internal NAT subnets as configured in cluster.yaml for kube-aws are no longer working for us.\r\nWe are using the all of the same values (in our cluster.yaml) that we've been using successfully--up until this pre-release version 0.9.3, including:\r\n- vpcId\r\n- routeTableId\r\n- vpcCIDR\r\n- subnets and their associated instanceCIDR ranges\r\n- serviceCIDR\r\n- podCID\r\n\r\nNote: Some of the content for this has come from my below two comments from one of our existing issues:\r\n- https://github.com/coreos/kube-aws/issues/189#issuecomment-275816250\r\n- https://github.com/coreos/kube-aws/issues/189#issuecomment-276199562\r\n\r\nRoute tables from kube-aws-v0.9.3-rc.5 kubernetes subnets correctly generated and sent traffic to our NAT instance:\r\n```\r\n# v0.9.3-rc.5 myapp-kube-Subnet0 sends all traffic to EC2 NAT instance\r\n 0.0.0.0/0     eni-def0049a / i-abc12345\r\n\r\n# v0.9.3-rc.5 myapp-kube-Subnet1 sends all traffic to EC2 NAT instance\r\n 0.0.0.0/0     eni-def0049a / i-abc12345\r\n```\r\n\r\nNote that this is correct and works for us and we confirm that this matches our other existing internal non-kube-aws/kubernetes subnets as shown below (wherein the route table sends all traffic to our EC2 NAT instance).\r\n```\r\n0.0.0.0/0     eni-def0049a / i-abc12345\r\n```\r\n\r\nNow with the latest pre-release 0.9.3 version of kube-aws, what we see (with us using the same settings and configurations that we've been using for all releases up to and including v0.9.3-rc.5) is that all of the kube-aws provisioned kubernetes nodes cannot access the internet. Note that many of the specific and supporting details are documented in the above two comments/links.\r\n\r\nWhat appears to be the problem for us, is that now kube-aws is configuring the kubernetes subnets to try to send traffic to either a specified existing, or newly created internet gateway, instead of forwarding the private subnet traffic to (our EC2 instance which does the) NAT.\r\n\r\n```\r\n# pre-release 0.9.3 myapp-kube-Subnet0 route table sends traffic to an igw instead of the NAT instance\r\n0.0.0.0/0     igw-ccb87ca4 \r\n\r\n# pre-release 0.9.3 myapp-kube-Subnet0 route table sends traffic to an igw instead of the NAT instance\r\n0.0.0.0/0     igw-ccb87ca4 \r\n```\r\n\r\nAlso from the new pre-release kube-aws version's generated cluster.yaml file, noting that we now see config options for specifying an existing ```natGateway``` wherein it appears as though **perhaps a very important distinction should/must be made that this new configuration option**, and its related code, appear to be designed to work with the newer _**AWS NAT Gateway service**_ (and **perhaps/maybe not** with the (older) _**EC2 NAT instances**?_  Because if it did support actual EC2 NAT instances,  we'd need to specify an actual EC2 instance-id of the current NAT instance?  I don't see this noted in the comments and **perhaps I am in error by not noting this distinction** when I tested building and deploying the pre-release binary?\r\n\r\n```\r\n# Kubernetes subnets with their CIDRs and availability zones. Differentiating availability zone for 2 or more subnets result in high-availability (failures of a single availability zone won't result in immediate downtimes)\r\n# subnets:\r\n#   - availabilityZone: us-west-1a\r\n#     instanceCIDR: \"10.0.0.0/24\"\r\n#     subnetId: \"subnet-xxxxxxxx\" #optional\r\n#   - availabilityZone: us-west-1b\r\n#     instanceCIDR: \"10.0.1.0/24\"\r\n#     # natGateway:\r\n#     #   # Pre-allocated NAT Gateway. Used with private subnets.\r\n#     #   id: \"ngw-abcdef12\"\r\n#     #   # Pre-allocated EIP for NAT Gateways. Used with private subnets.\r\n#     #   eipAllocationId: \"eipalloc-abcdef12\"\r\n#     subnetId: \"subnet-xxxxxxxx\" #optional\r\n```\r\n\r\nTo summarize:\r\nThe kube-aws provisioned kubernetes subnet routing has changed for us between the latest (and last working version for us) v0.9.3-rc.5 and the current state of the master branch with pre-release 0.9.3\r\nI'm confused as to why we would be forced into specifying (and if not, having kube-aws auto generating) an igw and then have kube-aws configure the routing of all traffic to it, if we are deploying into a private subnet which utilizes (and has been working correctly with previous kube-aws versions) NAT?\r\n\r\nA few questions that would greatly help us move forward:\r\nAre the new configuration options (and related/supported code) _only_ applicable for the (perhaps newer) **AWS NAT Gateway service** and **NOT** applicable for (older) infrastructures which may utilize an **actual NAT EC2 instance?**  If so, this will now be a problem for us as.\r\n\r\nIf the changes are by design, then with the latest pre-release and onward, we will need to know how to correctly configure our cluster.yaml per our requirements, including: \r\n- deploying into existing VPC with existing igw.\r\n- deploying into private subnets which **use an actual EC2 NAT instance and not the NAT Service for all NAT traffic** (ensuring the requisite route table is specified in cluster.yaml) \r\n- want multi-AZ H/A\r\n\r\nThank you!"},"comment":{"url":"https://api.github.com/repos/coreos/kube-aws/issues/comments/276584528","html_url":"https://github.com/coreos/kube-aws/issues/290#issuecomment-276584528","issue_url":"https://api.github.com/repos/coreos/kube-aws/issues/290","id":276584528,"user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars.githubusercontent.com/u/22009?v=3","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"created_at":"2017-02-01T06:40:58Z","updated_at":"2017-02-01T06:40:58Z","body":"Hi @cmcconnell1, thanks as always!\r\n\r\nI'm running short on time but AFAICS right now, the current master branch is for deverlopers only; it isn't a v0.9.3 candidate but a v0.9.4-rc.1 candidate.\r\n\r\nv0.9.3 is still pending due to the binary signing process but you can build it your own by checking out the `v0.9.3` tag of the git repo.\r\n\r\nLet me also add that I believe the bug you've mentioned will be fixed once #284 is merged."}}
{"actor":{"display_login":"mumoshu"}}