title: Does ALB over grpc protocol return network related errors when scaling concurrent load?
body: <p>We were experimenting load balancing startegies for grpc based services in aws cloud. In addition to client side load balancing recommened in grpc platform, we also wanted to try the ALB offered in aws over <a href="https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/" rel="nofollow noreferrer">the grpc protocol</a>. We created a grpc service written in golang with two instances and followed all the steps like creating Target groups, configuring an ALB over grpc protocol and health checks. We wrote a load generation[in golang] tool to send concurrent requests to the service. The load generation tool creates a single grpc client connection and uses the same to send concurrent requests. When the concurrency[workers] is increased[~1000] and run for a period of time, some requests are failing with below error.</p>
  <blockquote>
  <p>code = Unavailable desc = transport is closing</p>
  </blockquote>
  <p>For 250K requests to the ALB in 20mins, around 1k requests were failing in small batches with the above error.</p>
  <p>Then to identify the root cause, we used a NLB to test the same load and didn't get any errors.</p>
  <p>Note: We are aware that NLB won't load balance requests over single client to multiple instances. This is done just to identify the cause of error.</p>
  <p>We added <a href="https://pkg.go.dev/google.golang.org/grpc@v1.37.0/channelz/service" rel="nofollow noreferrer">channelz</a> to the service and monitored the number of failed messages in all channels/sockets. The number of failures are below hunder[~70] in the channelz stats.</p>
  <p>We also noticed that the monitoring stats for the alb showed 4xx error codes.</p>
  <p>Please share suggestions to debug the failures from ALB or articles around the internals of AWS ALB to figure out the solution.</p>