title: Hiding a docker container behind OpenVPN, in docker swarm, with an overlay network
body: <blockquote>
  <p>The goal: To deploy on docker swarm a set of services, one of which is only available for me when I am connected to the OpenVPN server which has also been spun up on docker swarm.</p>
  <p>How can I, <em>step by step</em>, only connect to a whoami example container, with a domain in the browser, when I am connected to a VPN?</p>
  </blockquote>
  <h2>Background</h2>
  <p>The general idea would be have, say, kibana and elasticsearch running internally which can only be accessed when on the VPN (rather like a corporate network), with other services running perfectly fine publicly as normal. <em>These will all be on separate nodes, so I am using an overlay network</em>.</p>
  <p>I do indeed have OpenVPN running on docker swarm along with a <a href="https://github.com/containous/whoami" rel="noreferrer">whoami</a> container, and I can connect to the VPN, however it doesn't look like the IP is changing and I have no idea how to make it so that the whoami container is only available when on the VPN, especially considering I'm using an <a href="https://docs.docker.com/network/overlay/" rel="noreferrer">overlay network</a> which is multi-host. I'm also using <a href="https://docs.traefik.io/" rel="noreferrer">traefik</a>, a reverse proxy which provides me with a mostly automatic letsencrypt setup (via DNS challenge) for wildcard domains. With this I can get:</p>
  <blockquote>
  <p><a href="https://traefik.mydomain.com" rel="noreferrer">https://traefik.mydomain.com</a></p>
  </blockquote>
  <p>But I also want to connect to <code>vpn.mydomain.com</code> (which I can do right now), and then be able to visit:</p>
  <blockquote>
  <p><a href="https://whoami.mydomain.com" rel="noreferrer">https://whoami.mydomain.com</a></p>
  </blockquote>
  <p>...which I cannot. Yet. I've posted my <a href="https://pastebin.com/bidbGCcL" rel="noreferrer">traefik configuration</a> in a different place in case you want to take a look, as this thread will grow too big if I post it here.</p>
  <p>Let's start with where I am right now.</p>
  <h2>OpenVPN</h2>
  <p>Firstly, the interesting thing about OpenVPN and docker swarm is that OpenVPN needs to run in privileged mode because it has to make network interfaces changes amongst other things, and <a href="https://github.com/moby/moby/issues/25885" rel="noreferrer">swarm doesn't have CAP_ADD</a> capabilities yet. So the idea is to launch the container via a sort of <a href="https://github.com/ix-ai/swarm-launcher" rel="noreferrer">'proxy container'</a> that will run the container manually with these privileges added for you. It's a workaround for now, but it means you can deploy the service with swarm.</p>
  <p>Here's my docker-compose for OpenVPN:</p>
  <pre><code>    vpn-udp:
          image: ixdotai/swarm-launcher:latest
          hostname: mainnode
          environment:
              LAUNCH_IMAGE: ixdotai/openvpn:latest
              LAUNCH_PULL: 'true'
              LAUNCH_EXT_NETWORKS: 'app-net'
              LAUNCH_PROJECT_NAME: 'vpn'
              LAUNCH_SERVICE_NAME: 'vpn-udp'
              LAUNCH_CAP_ADD: 'NET_ADMIN'
              LAUNCH_PRIVILEGED: 'true'
              LAUNCH_ENVIRONMENTS: 'OVPN_NATDEVICE=eth1'
              LAUNCH_VOLUMES: '/etc/openvpn:/etc/openvpn:rw'
          volumes:
              - '/var/run/docker.sock:/var/run/docker.sock:rw'
          networks:
              - my-net
          deploy:
              placement:
                  constraints:
                      - node.hostname==mainnode
  </code></pre>
  <p>I can deploy the above with: <code>docker stack deploy --with-registry-auth --compose-file docker/docker-compose.prod.yml my-app-name</code> and this is what I'm using for the rest. Importantly I cannot just deploy this as it won't load yet. OpenVPN configuration needs to exist in <code>/etc/openvpn</code> on the node, which is then mounted in the container, and I do this during provisioning:</p>
  <pre><code>// Note that you have to create the overlay network with --attachable for standalone containers
  docker network create -d overlay app-net --attachable
  
  // Create the config
  docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn ovpn_genconfig -u udp://vpn.mydomain.com:1194 -b
  
  // Generate all the vpn files, setup etc
  docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn bash -c 'yes yes | EASYRSA_REQ_CN=vpn.mydomain.com ovpn_initpki nopass'
  
  // Setup a client config and grab the .ovpn file used for connecting
  docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn easyrsa build-client-full client nopass
  
  docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn ovpn_getclient client &gt; client.ovpn
  </code></pre>
  <p>So now, I have an attachable overlay network, and when I deploy this, OpenVPN is up and running on the first node. I can grab a copy of <code>client.ovpn</code> and connect to the VPN. Even if I check &quot;send all traffic through the VPN&quot; though, it looks like the IP isn't being changed, and I'm still nowhere near hiding a container behind it.</p>
  <hr />
  <h2>Whoami</h2>
  <p>This simple container can be deployed with the following in docker-compose:</p>
  <pre><code>    whoami:
          image: &quot;containous/whoami&quot;
          hostname: mainnode
          networks:
              - ${DOCKER_NETWORK_NAME}
          ports:
              - 1337:80
          deploy:
              placement:
                  constraints:
                      - node.hostname==mainnode
  </code></pre>
  <p>I put port <code>1337</code> there for testing, as I can visit my IP:1337 and see it, but this doesn't achieve my goal of having <code>whoami.mydomain.com</code> only resolving when connected to OpenVPN.</p>
  <hr />
  <h2>I can ping a <code>192.168</code> address when connected to the vpn</h2>
  <p>I ran the following on the host node:</p>
  <blockquote>
  <p>ip -4 address add 192.168.146.16/24 dev eth0</p>
  </blockquote>
  <p>Then when connected to the VPN, I can resolve this address! So it looks like <em>something</em> is working at least.</p>
  <p>How can I achieve the goal stated at the top? What is required? What OpenVPN configuration needs to exist, what network configuration, and what container configuration? Do I need a custom DNS solution as I suggest below? What better alternatives are there?</p>
  <hr />
  <p>Some considerations:</p>
  <ul>
  <li><p>I can have the domains, including the private one <code>whoami.mydomain.com</code> public. This means I would have https and get wildcard certificates for them easily, I suppose? But my confusion here is - how can I get those domains only on the VPN but also have tls certs for them without using a self-signed certificate?</p>
  </li>
  <li><p>I can also run my own DNS server for resolving. I have tried this but I just couldn't get it working, probably because the VPN part isn't working properly yet. I found <a href="http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html" rel="noreferrer">dnsmasq</a> for this and I had to add the aforementioned local ip to <code>resolve.conf</code> to get anything working locally for this. But domains would still not resolve when connected to the VPN, so it doesn't look like DNS traffic was going over the VPN either (even though I set it as such - my client is <a href="https://www.sparklabs.com/viscosity/" rel="noreferrer">viscosity</a>.</p>
  </li>
  <li><p>Some mention using a bridge network, but a bridge network does not work for multi-host</p>
  </li>
  </ul>
  <h2>Resources thus far (I will update with more)</h2>
  <p><em>- <a href="https://alex.thom.ae/2020/02/22/openvpn-server-docker-swarm/" rel="noreferrer">Using swarm-launcher to deploy OpenVPN</a></em></p>
  <p><em>- <a href="https://security.stackexchange.com/questions/97542/hide-docker-containers-behind-vpn">A completely non-explanatory answer on stackexchange</a> which I have seen referenced as basically unhelpful by multiple people across other Github threads, and one of the links is dead</em></p>