title: Docker swarm mode on RHEL
body: <p>I have been trying to run a single node docker swarm for testing on RHEL 7.6. <code>firewalld</code> is disabled and not running.  Services are running on an <code>overlay</code> network.  I noticed that I can't connect to the published port either from the host or outside.  This behaviour is consistent for a few RHEL instances I tried.  I do use docker swarm on Ubuntu 16.04LTS and 18.04LTS without any glitches. </p>
  
  <p>Given below is my <code>docker info</code></p>
  
  <pre><code>Client:
   Debug Mode: false
  
  Server:
   Containers: 14
    Running: 3
    Paused: 0
    Stopped: 11
   Images: 4
   Server Version: 19.03.3
   Storage Driver: overlay2
    Backing Filesystem: xfs
    Supports d_type: true
    Native Overlay Diff: true
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: active
    NodeID: fhewk7l15g42o36henpfigwjk
    Is Manager: true
    ClusterID: kegypzam66ehi6s50utrsff1l
    Managers: 1
    Nodes: 1
    Default Address Pool: 10.0.0.0/8
    SubnetSize: 24
    Data Path Port: 4789
    Orchestration:
     Task History Retention Limit: 5
    Raft:
     Snapshot Interval: 10000
     Number of Old Snapshots to Retain: 0
     Heartbeat Tick: 1
     Election Tick: 10
    Dispatcher:
     Heartbeat Period: 5 seconds
    CA Configuration:
     Expiry Duration: 3 months
     Force Rotate: 0
  Autolock Managers: false
    Root Rotation In Progress: false
    Node Address: 10.0.1.125
    Manager Addresses:
     10.0.1.125:2377
   Runtimes: runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
   runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
   init version: fec3683
   Security Options:
    seccomp
     Profile: default
   Kernel Version: 3.10.0-957.5.1.el7.x86_64
   Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
   OSType: linux
   Architecture: x86_64
   CPUs: 4
   Total Memory: 15.33GiB
   Name: rhel-test.dev.koopid.io
   ID: IM3X:THRY:FYUO:L7XI:VJW6:5B4Y:VZOX:YL43:E7WR:U5GM:3BQK:NLKP
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Registry: https://index.docker.io/v1/
   Labels:
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false
  </code></pre>
  
  <p>And my <code>overlaynet</code></p>
  
  <pre><code>[
      {
          "Name": "overlaynet",
          "Id": "4g4dphekzyshqpcp0fjfmc877",
          "Created": "2019-10-18T14:29:06.284905975Z",
          "Scope": "swarm",
          "Driver": "overlay",
          "EnableIPv6": false,
          "IPAM": {
              "Driver": "default",
              "Options": null,
              "Config": [
                  {
                      "Subnet": "172.20.0.0/24",
                      "Gateway": "172.20.0.1"
                  }
              ]
          },
          "Internal": false,
          "Attachable": true,
          "Ingress": false,
          "ConfigFrom": {
              "Network": ""
          },
          "ConfigOnly": false,
          "Containers": {
              "142c22a7e517f463f37c89cfb58dcde37f9529c9b469357b37868057be044e48": {
                  "Name": "dbsvcs_redis.1.0lsxkr88eq89igid7w7ifk3wq",
                  "EndpointID": "167fbdfb2146f09bb20c258fea52d9f8ca886cf1d264b1d8cd9169532c26b9db",
                  "MacAddress": "02:42:ac:14:00:03",
                  "IPv4Address": "172.20.0.3/24",
                  "IPv6Address": ""
              },
              "2e70a7589f13c74be66149d5bbf9504b5b74aee1ad6711f82ec4b02011c00cc1": {
                  "Name": "dbpg_postgresql-rw.1.9keeuowk9zk5e6f8bq5a0itij",
                  "EndpointID": "44a2376b4d0d2bdb8787c9cc18726da140ca0f9a8e97e54a6a78b2206e10a13b",
                  "MacAddress": "02:42:ac:14:00:06",
                  "IPv4Address": "172.20.0.6/24",
                  "IPv6Address": ""
              },
              "d9119bb3d605aa9b2df23985cd884afa941499d888937e3c34f4ec08dac14c73": {
                  "Name": "dbsvcs_influxdb.1.ap5cg0se1rntdbsopxbm7whma",
                  "EndpointID": "d2a5c093a0721291a114309ef1fd690510b03007fdaf83c8d77e00870a1568cd",
                  "MacAddress": "02:42:ac:14:00:04",
                  "IPv4Address": "172.20.0.4/24",
                  "IPv6Address": ""
              },
              "lb-overlaynet": {
                  "Name": "overlaynet-endpoint",
                  "EndpointID": "2bdf0d2370856d9a4b2da1e86d65521585ffc89c778f5db1d3f4b2fd39da7c8b",
                  "MacAddress": "02:42:ac:14:00:08",
                  "IPv4Address": "172.20.0.8/24",
                  "IPv6Address": ""
              }
          },
          "Options": {
              "com.docker.network.driver.overlay.vxlanid_list": "4097"
          },
          "Labels": {},
          "Peers": [
              {
                  "Name": "80ab8f4e3bcd",
                  "IP": "10.0.1.125"
              }
          ]
      }
  ]
  </code></pre>
  
  <p>I've the following services and as you notice, all of them publish one or two ports.</p>
  
  <pre><code>4j7p43udxkoc        dbpg_postgresql-rw   replicated          1/1                 myregistry/postgres   *:5432-&gt;5432/tcp
  hu0wkspwc7j3        dbsvcs_influxdb      replicated          1/1                 myregistry/influxdb   *:8086-&gt;8086/tcp
  dlte2nzg226x        dbsvcs_redis         replicated          1/1                 myregistry/redis      *:6379-&gt;6379/tcp
  </code></pre>
  
  <p>And you can see that port 5432 is open for INADDR_ANY on the host</p>
  
  <pre><code>tcp6       1      0 :::5432                 :::*                    LISTEN
  </code></pre>
  
  <p>However, I can't connect to port 5432 from the host our outside.  <code>psql</code> client times out as if some firewall is blocking the connection.  </p>
  
  <p>I can see the following errors if I enable <code>firewalld</code></p>
  
  <pre><code>firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker_gwbridge -o docker_gwbridge -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -nL DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -nL DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -nL DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
  firewalld[2809]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -nL DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
  </code></pre>
  
  <p>Is it something I should worry about?  Do I need to fiddle around with <code>iptables</code> on RHEL to get docker swarm working.  There are some reports to add docker control ports to <code>iptables</code> for multi-node cluster configuration.  My <code>iptable</code> configuration is something like this...</p>
  
  <pre><code>$ iptables -L -v -n --line-numbers
  Chain INPUT (policy ACCEPT 82507 packets, 8110K bytes)
  num   pkts bytes target     prot opt in     out     source               destination
  
  Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
  num   pkts bytes target     prot opt in     out     source               destination
  1       30  5664 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  2       30  5664 DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  3       30  5664 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  4        0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
  5        0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
  6        0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
  7        0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0
  8       14  4064 ACCEPT     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
  9        0     0 DOCKER     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0
  10      16  1600 ACCEPT     all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0
  11       0     0 DROP       all  --  docker_gwbridge docker_gwbridge  0.0.0.0/0            0.0.0.0/0
  
  Chain OUTPUT (policy ACCEPT 82105 packets, 8106K bytes)
  num   pkts bytes target     prot opt in     out     source               destination
  
  Chain DOCKER (2 references)
  num   pkts bytes target     prot opt in     out     source               destination
  
  Chain DOCKER-INGRESS (1 references)
  num   pkts bytes target     prot opt in     out     source               destination
  1        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5432
  2        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:5432
  3        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6379
  4        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:6379
  5        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8086
  6        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8086
  7       30  5664 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
  
  Chain DOCKER-ISOLATION-STAGE-1 (1 references)
  num   pkts bytes target     prot opt in     out     source               destination
  1        0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
  2       16  1600 DOCKER-ISOLATION-STAGE-2  all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0
  3       30  5664 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
  
  Chain DOCKER-ISOLATION-STAGE-2 (2 references)
  num   pkts bytes target     prot opt in     out     source               destination
  1        0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
  2        0     0 DROP       all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0
  3       16  1600 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
  
  Chain DOCKER-USER (1 references)
  num   pkts bytes target     prot opt in     out     source               destination
  1       30  5664 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
  </code></pre>
  
  <p>Appreciate some help/direction to get this working on RHEL as I'm stuck on this for the last couple of weeks.  Configuring and running <code>docker swarm</code> on Ubuntu was a breeze!!!</p>