title: load balance UDP syslog with custom health check
body: <p>I setup Nginx Plus to load-balance UDP syslog traffic. Here's a snippet from <code>nginx.conf</code>:</p>
  
  <pre><code>stream {
  
      upstream syslog_standard {
          zone syslog_zone 64k;
          server cp01.woolford.io:1514 max_fails=1 fail_timeout=10s;
          server cp02.woolford.io:1514 max_fails=1 fail_timeout=10s;
          server cp03.woolford.io:1514 max_fails=1 fail_timeout=10s;
      }
  
      server {
          listen 514 udp;
          proxy_pass syslog_standard;
          proxy_bind $remote_addr transparent;
          health_check udp;
      }
  
  }
  </code></pre>
  
  <p>I was a little surprised to hear that NGINX Plus could perform health checks on UDP since UDP is, by design, unreliable. Since there is no acknowledgment in UDP, the messages effectively go into a black hole.</p>
  
  <p>I'm trying to set up a somewhat fault-tolerant and scalable syslog ingestion pipeline. The loss of a node should be detected, by a health check, and be temporarily removed from the list of available servers.</p>
  
  <p><a href="https://i.stack.imgur.com/jueNs.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jueNs.png" alt="nginx big picture"></a></p>
  
  <p>This didn't work, despite the UDP health check. I think the UDP health check only works for services that respond to the caller (e.g. DNS). Since syslog doesn't respond, there's no way to check for errors, e.g. using <a href="https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#defining-custom-conditions" rel="nofollow noreferrer">match</a>.</p>
  
  <p>The process that's ingesting the syslog messages listens on port 1514 and has a REST interface on port 8073:</p>
  
  <p>If the ingest process is healthy a <code>GET</code> request to <code>/connectors/syslog/status</code> on port 8073 returns:</p>
  
  <pre><code>{
    "name": "syslog",
    "connector": {
      "state": "RUNNING",
      "worker_id": "10.0.1.41:8073"
    },
    "tasks": [
      {
        "id": 0,
        "state": "RUNNING",
        "worker_id": "10.0.1.41:8073"
      }
    ],
    "type": "source"
  }
  </code></pre>
  
  <p>I'd like to create a custom check to see that ingest is running. Is that possible with NGINX Plus? Can we check the health on a completely different port?</p>