title: Azure Function runs into System.Net.Sockets.SocketException
body: <p>I have AzureFunctions app with 2 HTTP trigered functions. Both are direved from same class but use different urls to fetch data.
  Every day Azure Data Factory pipeline triggers 1st HTTP function
  and another pipeline calls 2nd function within 1 minute</p>
  
  <p>Each function makes around 1300 HTTP requests to 3rd party website and stores each response as separate json file in Blob storage.</p>
  
  <p>The problem is almost each time (but not always) 2nd function throws System.Net.Sockets.SocketException because few outbound requests run into common 21 seconds TCP timeout. 
  Strange thing I've notices - it is likely Azure throttles for some reason my outbound requests: first batch takes near 300ms, next sequence takes 4.3sec, then 9.5sec and next batch reaches 21 sec with exception</p>
  
  <p><a href="https://i.stack.imgur.com/RwGkl.jpg" rel="nofollow noreferrer">Here is image of timing increasing of outbound requests</a></p>
  
  <p>Exception stacktrace:</p>
  
  <blockquote>
    <p>System.Net.Http.HttpRequestException: A connection attempt failed
    because the connected party did not properly respond after a period of
    time, or established connection failed because connected host has
    failed to respond ---> System.Net.Sockets.SocketException: A
    connection attempt failed because the connected party did not properly
    respond after a period of time, or established connection failed
    because connected host has failed to respond    at
    System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port,
    CancellationToken cancellationToken)    --- End of inner exception
    stack trace ---    at
    System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port,
    CancellationToken cancellationToken)    at
    System.Threading.Tasks.ValueTask<code>1.get_Result()    at
    System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage
    request, CancellationToken cancellationToken)    at
    System.Threading.Tasks.ValueTask</code>1.get_Result()    at
    System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask<code>1
    creationTask)    at System.Threading.Tasks.ValueTask</code>1.get_Result()<br>
    at
    System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage
    request, Boolean doRequestAuth, CancellationToken cancellationToken)<br>
    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage
    request, CancellationToken cancellationToken)    at
    System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage
    request, CancellationToken cancellationToken)    at
    System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,
    HttpRequestMessage request, CancellationTokenSource cts, Boolean
    disposeCts)    at
    FunctionApp.BaseFunc.&lt;>c__DisplayClass7_2.&lt;b__0>d.MoveNext()
    in
    E:\vsts-agent-win-1_work\339\s\Services\Host\Controllers\BaseFunc.cs:line
    102
    --- End of stack trace from previous location where exception was thrown ---    at FunctionApp.BaseFunc.ProcessRun(ILogger
    log, String runId) in
    E:\vsts-agent-win-1_work\339\s\Services\Host\Controllers\BaseFunc.cs:line
    122.</p>
  </blockquote>
  
  <p>FunctionApp is hosted on AppService plan S1 so there is no limit of outbound connections of 600 (I believe so)</p>
  
  <p>Metrics of TCP connections during exception (max was 498):
  <a href="https://i.stack.imgur.com/Lkour.jpg" rel="nofollow noreferrer">Metrics of AzureFunction App</a></p>
  
  <p>TCP Connections from "Solve problem" helper of AzureFunction App
  <a href="https://i.stack.imgur.com/Cvt7h.jpg" rel="nofollow noreferrer">Max TCP connections in all states was 502</a></p>
  
  <p>CPU and Memory of App service plan during exception:
  <a href="https://i.stack.imgur.com/HbBGb.jpg" rel="nofollow noreferrer">App Service Plan metrics</a></p>
  
  <p>App is .Net Core 2.2</p>
  
  <p>I didn't manage to reproduce this on my local PC. But on Azure it happens almost every day on each environment (dev, test, prod).
  After such fail Azure Data Factory makes retry in 5 minutes and it is successfull each time.</p>
  
  <p>Here is code of Base class that is used by both functions:</p>
  
  <pre><code> public abstract class BaseFunc
  {
      protected abstract string BlobFolderName { get; }
      protected TelemetryClient telemetryClient;
      private static HttpClient _httpClient;
  
      static BaseFunc()
      {
          HttpClientHandler handler = new HttpClientHandler();
          handler.MaxConnectionsPerServer = 300;
          _httpClient = new HttpClient(handler);
      }
      protected async Task ProcessRun(ILogger log, string runId)
      {
          int processedItems = 0;
          try
          {
              Stopwatch sw = Stopwatch.StartNew();
              ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
  
              await Authentication("url", log, runId); //sets default Authorization header
  
              string getIdeaResult = await _httpClient.GetStringAsync("url");
              JObject jsonObject = JObject.Parse(getIdeaResult);
              int ideaCount = (int)jsonObject.SelectToken("total_count");
  
              List&lt;Task&gt; tasks = new List&lt;Task&gt;();
              string DataPulledDate = DateTime.Now.ToString("dd-MMM-yyyy");
              CloudStorageAccount storageAccount = CloudStorageAccount.Parse("connection string");
              CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
              CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("container");
  
              string getIdsUri = "url" + $"&amp;limit={batchSize}&amp;offset=";
              int iterations = (int)Math.Ceiling((decimal)ideaCount/batchSize);
  
              for (int i = 0; i &lt; iterations; i++)
              {
                  string result = await _httpClient.GetStringAsync("url" + i * 50);
                  JObject jsonIdsObject = JObject.Parse(result);
                  int[] ideaIds = jsonIdsObject["content"].Children().Values&lt;int&gt;("id").ToArray();
                  foreach (int id in ideaIds)
                  {
                      tasks.Add(Task.Run(async () =&gt;
                      {
                          string content = null;
                          using (var response = await _httpClient.SendAsync(new HttpRequestMessage(HttpMethod.Get, "url"+ id))) //Exception is thrown on this line
                          {
                              content = await response.Content.ReadAsStringAsync();
                              response.EnsureSuccessStatusCode();
                          }
                          CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference($"{DataPulledDate}/{BlobFolderName}/ideaId-{id}.json");
                          await cloudBlobContainer.CreateIfNotExistsAsync();
                          await cloudBlockBlob.UploadTextAsync(content);
                          Interlocked.Increment(ref processedItems);
                      }));
                  }
              }
              await Task.WhenAll(tasks);
              sw.Stop();
          }
          catch (Exception ex)
          {
              log.LogError(ex, "{RunId}: Run failed. {Items} items processed successfully, Exception: {Exception}.", runId, processedItems, ex.ToString());
              throw;
          }
          finally
          {
              if (telemetryClient != null)
              {
                  telemetryClient.Flush();
                  Thread.Sleep(3000);
              }
          }
      }
  }
  </code></pre>
  
  <p>Code of Function itself:</p>
  
  <pre><code>namespace FunctionApp
  {
      public class GetIdeas : BaseFunc
      {
          public GetIdeas(TelemetryClient telemetryClient)
          {
              this.telemetryClient = telemetryClient;
          }
  
          protected override string BlobFolderName { get =&gt; "folder"; }
          protected override string GetItemUrl { get =&gt; "url"; }
  
          [FunctionName("GetIdeasFn")]
          public async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req, ILogger log)
          {
              await ProcessRun(log, $"GetIdeasFn - {DateTime.UtcNow.Ticks}");
          }
      }
  }
  </code></pre>
  
  <p>Appreciate any help.</p>