title: Scalability of kube-proxy
body: <p>I have encountered a scalability problem when trying out the kubernetes cluster. To simplify the topology in my test machine, NodePort type is used to expose the individual service externally. The baremetal to host the node and master is a RHEL 7 with 24 CPUs and 32G RAM; I don't yet have a dedicated load balancer, or a cloud provider like infrastructure. A snippet of the service definition looks just like below</p>
  
  <pre><code>    "spec": {       
           "ports": [{
               "port": 10443,             
               "targetPort": 10443,               
               "protocol": "TCP",                                
               "nodePort": 30443
      } ],
     "type": "NodePort",
  </code></pre>
  
  <p>Through this way the application can be accessible via <code>https://[node_machine]:30443/[a_service]</code> </p>
  
  <p>Such service is only backed by one Pod. Ideally I would want to have several services deployed on the same node (but using different NodePort's), and and running concurrently.</p>
  
  <p>Things were working well until it became evident that for a similar workload, increasing the number of services deployed (therefore backend pods as well) makes the applications degrade in performance. Surprisingly, when breaking down the service loading time, I noticed there's dramatic degradation in 'Connection Time' which seems to indicate there is a slowdown somewhere in the 'network' layer. Please note that the load isn't high enough to drive much of the CPU on the node yet. I read about the <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/services.md#shortcomings" rel="nofollow">shortcomings</a> in the doc, but not sure if what I hit is exactly the limitation of the kube-proxy/Service described there. </p>
  
  <p>The questions are:</p>
  
  <ol>
  <li><p>Is there any suggestion on how to make it more scalable? I.e. to be able to support more services/Pods without scarifying the applications' performance? The NodePort type is the easiest way to setup the 'public' address for our services but is there any limitation for scalability or performance if all services and Pods are setting up this way?</p></li>
  <li><p>Would there be any difference if we change the type to LoadBalancer? 
  "type": "LoadBalancer"</p></li>
  <li><p>Further more, is there a benefit to have a dedicated LoadBalancer or reverse proxy to improve the scalability, e.g. HAProxy or alike, that routes traffic from external to the backend Pods (or Services)? I noticed there's some work done for Nginx darkgaro/kubernetes-reverseproxy - unfortunately the doc seems incomplete and there's no concrete example.  In some of the other threads folks talked about Vulcan - is it the recommended LB tool for kubernetes? </p></li>
  </ol>
  
  <p>Your recommendation and help are highly appreciated!</p>