title: How to connect mongo replicaset on kubernetes cluster from another cluster pods
body: <p>I have a mongo db replicaset running on kubernetes cluster ( on AWS EKS ) , say cluster-1. This is running within VPC-1 having cidr 192.174.0.0/16.</p>
  <p>I have another cluster in a separate VPC , say VPC-2, where I'll be running some applications on top of the mongo cluster. This VPC cidr range is 192.176.0.0/16. All VPC peering and security group ingress/egress rules are working fine and I am able to ping cluster nodes across the two VPCs.</p>
  <p>I am using NodePort type service and StatefulSet for the mongo cluster :</p>
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: mongodb-service
    labels:
      name: mongodb
  spec:
    selector:
      role: mongo
    type: NodePort
    ports:
    - port: 26017
      targetPort: 27017
      nodePort: 30017
  </code></pre>
  <p>Here are the nodes &amp; pods in mongo cluster, cluster-1 :</p>
  <pre><code>ubuntu@ip-192-174-5-253:/st_config/kubeobj$ kubectl get nodes -o wide
  NAME                                             STATUS   ROLES    AGE   VERSION              INTERNAL-IP       EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
  ip-192-174-187-133.ap-south-1.compute.internal   Ready    &lt;none&gt;   19h   v1.16.8-eks-e16311   192.174.187.133   13.232.195.39    Amazon Linux 2   4.14.181-140.257.amzn2.x86_64   docker://19.3.6
  ip-192-174-23-229.ap-south-1.compute.internal    Ready    &lt;none&gt;   19h   v1.16.8-eks-e16311   192.174.23.229    13.234.111.139   Amazon Linux 2   4.14.181-140.257.amzn2.x86_64   docker://19.3.6
  ubuntu@ip-192-174-5-253:/st_config/kubeobj$
  
  
  ubuntu@ip-192-174-5-253:/st_config/kubeobj$ kubectl get pods -o wide
  NAME       READY   STATUS    RESTARTS   AGE   IP                NODE                                             NOMINATED NODE   READINESS GATES
  mongod-0   1/1     Running   0          45m   192.174.8.10      ip-192-174-23-229.ap-south-1.compute.internal    &lt;none&gt;           &lt;none&gt;
  mongod-1   1/1     Running   0          44m   192.174.133.136   ip-192-174-187-133.ap-south-1.compute.internal   &lt;none&gt;           &lt;none&gt;
  ubuntu@ip-192-174-5-253:/st_config/kubeobj$
  </code></pre>
  <p>If I try to connect using a specific node address, OR both node addresses, kubernetes is perhaps load-balancing or rotating the connection in a round robin fashion:</p>
  <pre><code>ubuntu@ip-192-176-42-206:~$ mongo mongodb://192.174.23.229:30017
  MongoDB shell version v3.6.3
  connecting to: mongodb://192.174.23.229:30017
  MongoDB server version: 3.4.24
  WARNING: shell and server versions do not match
  test_rs0:PRIMARY&gt;
  
  ubuntu@ip-192-176-42-206:~$ mongo mongodb://192.174.23.229:30017
  MongoDB shell version v3.6.3
  connecting to: mongodb://192.174.23.229:30017
  MongoDB server version: 3.4.24
  WARNING: shell and server versions do not match
  test_rs0:SECONDARY&gt;
  
  ubuntu@ip-192-176-42-206:~$ mongo mongodb://192.174.23.229:30017,192.174.187.133:30017
  MongoDB shell version v3.6.3
  connecting to: mongodb://192.174.23.229:30017,192.174.187.133:30017
  MongoDB server version: 3.4.24
  WARNING: shell and server versions do not match
  test_rs0:PRIMARY&gt;
  </code></pre>
  <p>I wish to leverage the replicaset features. So when I used the the connect string as - mongodb://192.174.23.229:30017,192.174.187.133:30017/?replicaSet=test_rs0 , it is actually getting the FQDNs of the pods which are not resolved from the node in cluster-2 nodes/pods in VPC-2.</p>
  <pre><code>ubuntu@ip-192-176-42-206:~$ mongo mongodb://192.174.23.229:30017,192.174.187.133:30017/?replicaSet=test_rs0
  MongoDB shell version v3.6.3
  connecting to: mongodb://192.174.23.229:30017,192.174.187.133:30017/?replicaSet=test_rs0
  2020-06-23T15:59:07.407+0000 I NETWORK  [thread1] Starting new replica set monitor for test_rs0/192.174.23.229:30017,192.174.187.133:30017
  2020-06-23T15:59:07.409+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to 192.174.23.229:30017 (1 connections now open to 192.174.23.229:30017 with a 5 second timeout)
  2020-06-23T15:59:07.409+0000 I NETWORK  [thread1] Successfully connected to 192.174.187.133:30017 (1 connections now open to 192.174.187.133:30017 with a 5 second timeout)
  2020-06-23T15:59:07.410+0000 I NETWORK  [thread1] changing hosts to test_rs0/mongod-0.mongodb-service.default.svc.cluster.local:27017,mongod-1.mongodb-service.default.svc.cluster.local:27017 from test_rs0/192.174.187.133:30017,192.174.23.229:30017
  2020-06-23T15:59:07.415+0000 I NETWORK  [thread1] getaddrinfo(&quot;mongod-1.mongodb-service.default.svc.cluster.local&quot;) failed: Name or service not known
  2020-06-23T15:59:07.415+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] getaddrinfo(&quot;mongod-0.mongodb-service.default.svc.cluster.local&quot;) failed: Name or service not known
  2020-06-23T15:59:07.917+0000 I NETWORK  [thread1] getaddrinfo(&quot;mongod-0.mongodb-service.default.svc.cluster.local&quot;) failed: Name or service not known
  2020-06-23T15:59:07.918+0000 I NETWORK  [thread1] getaddrinfo(&quot;mongod-1.mongodb-service.default.svc.cluster.local&quot;) failed: Name or service not known
  2020-06-23T15:59:07.918+0000 W NETWORK  [thread1] Unable to reach primary for set test_rs0
  2020-06-23T15:59:07.918+0000 I NETWORK  [thread1] Cannot reach any nodes for set test_rs0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
  </code></pre>
  <p>Do I need to have some additional DNS service so that the names get resolved in VPC-2 nodes ? What would be the best approach ?</p>
  <p>Also how can I use connect string can be based on the service name eg. mongodb://mongodb-service.default.svc.cluster.local:/?replicaSet=test_rs0 from any node in VPC-2 ? It works from any pod in VPC-1. But I need to get this working from pods in cluster in VPC-2 so that I don't have to specify specific pod/node IP in connect string. All my kubernetes objects are in default namespace.</p>
  <p>Really appreciate some help here.
  **Please Note: I am NOT using helm **</p>