{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/33589","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/33589/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/33589/comments","events_url":"https://api.github.com/repos/moby/moby/issues/33589/events","html_url":"https://github.com/moby/moby/issues/33589","id":234528671,"number":33589,"title":"Docker Swarm Mode service discovery is totally unstable","user":{"login":"soar","id":30570,"avatar_url":"https://avatars0.githubusercontent.com/u/30570?v=4","gravatar_id":"","url":"https://api.github.com/users/soar","html_url":"https://github.com/soar","followers_url":"https://api.github.com/users/soar/followers","following_url":"https://api.github.com/users/soar/following{/other_user}","gists_url":"https://api.github.com/users/soar/gists{/gist_id}","starred_url":"https://api.github.com/users/soar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soar/subscriptions","organizations_url":"https://api.github.com/users/soar/orgs","repos_url":"https://api.github.com/users/soar/repos","events_url":"https://api.github.com/users/soar/events{/privacy}","received_events_url":"https://api.github.com/users/soar/received_events","type":"User","site_admin":false},"labels":[{"id":237472034,"url":"https://api.github.com/repos/moby/moby/labels/area/networking","name":"area/networking","color":"fbca04","default":false},{"id":392678049,"url":"https://api.github.com/repos/moby/moby/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false},{"id":190258329,"url":"https://api.github.com/repos/moby/moby/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false},{"id":583300157,"url":"https://api.github.com/repos/moby/moby/labels/version/17.05","name":"version/17.05","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2017-06-08T13:32:33Z","updated_at":"2017-09-18T15:01:12Z","closed_at":null,"author_association":"NONE","body":"**Description**\r\n\r\nI have about 10 nodes, 4 master nodes, 7 overlay networks and about 70 services. And swarm is totally unstable now: it accidentally stops resolving names of my services:\r\n\r\n```console\r\nroot@balancer:/# nslookup backend 127.0.0.11\r\nServer:\t\t127.0.0.11\r\nAddress:\t127.0.0.11#53\r\n\r\n** server can't find backend: NXDOMAIN\r\n```\r\n\r\nAnd this problem can exist for 10-20 minutes! All this time you'll see `502 Bad Gateway` error. And then you'll see messages like this:\r\n\r\n```text\r\nJun  8 15:04:16 node4 dockerd[840]: time=\"2017-06-08T15:04:16.415822659+02:00\" level=warning msg=\"Neighbor entry already present for IP 10.10.4.50, mac 02:42:0a:0a:04:32\"\r\nJun  8 15:04:16 node4 dockerd[840]: time=\"2017-06-08T15:04:16.415898743+02:00\" level=warning msg=\"Neighbor entry already present for IP <external-ip-here>, mac 02:42:0a:0a:04:32\"\r\nJun  8 15:04:16 node4 dockerd[840]: time=\"2017-06-08T15:04:16.415971947+02:00\" level=warning msg=\"Neighbor entry already present for IP 10.10.4.45, mac 02:42:0a:0a:04:2d\"\r\nJun  8 15:04:16 node4 dockerd[840]: time=\"2017-06-08T15:04:16.415997559+02:00\" level=warning msg=\"Neighbor entry already present for IP <external-ip-here>, mac 02:42:0a:0a:04:2d\"\r\n...\r\n```\r\n\r\n... and then it will recover itself:\r\n\r\n```console\r\nroot@balancer:/# nslookup backend 127.0.0.11\r\nServer:\t\t127.0.0.11\r\nAddress:\t127.0.0.11#53\r\n\r\nNon-authoritative answer:\r\nName:\tbackend\r\nAddress: 10.0.0.16\r\n```\r\n\r\nFor 1-2 hours. And then - cluster will become broken again.\r\n\r\n**Steps to reproduce the issue:**\r\n1. Create swarm\r\n2. Create services and networks\r\n3. Try to use it in production\r\n\r\n**Describe the results you received:**\r\nAt this moment Swarm Mode is unusable for production environments.\r\n\r\n**Describe the results you expected:**\r\nI expected that it will!\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      17.05.0-ce\r\n API version:  1.29\r\n Go version:   go1.7.5\r\n Git commit:   89658be\r\n Built:        Thu May  4 22:04:27 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      17.05.0-ce\r\n API version:  1.29 (minimum version 1.12)\r\n Go version:   go1.7.5\r\n Git commit:   89658be\r\n Built:        Thu May  4 22:04:27 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 3\r\n Running: 2\r\n Paused: 0\r\n Stopped: 1\r\nImages: 3\r\nServer Version: 17.05.0-ce\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 36\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins: \r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\nSwarm: active\r\n NodeID: yidccbcki64epay4p4ugq6xm1\r\n Is Manager: true\r\n ClusterID: hqvohft3etj4ajnkgubbnjwzp\r\n Managers: 4\r\n Nodes: 15\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Number of Old Snapshots to Retain: 0\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n Node Address: <ip1>\r\n Manager Addresses:\r\n  <ip2>:2377\r\n  <ip3>:2377\r\n  <ip1>:2377\r\n  <ip4>:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 9048e5e50717ea4497b757314bad98ea3763c145\r\nrunc version: 9c2d8d184e5da67c95d601382adf14862e4f2228\r\ninit version: 949e6fa\r\nKernel Version: 3.16.0-4-amd64\r\nOperating System: Debian GNU/Linux 8 (jessie)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 12\r\nTotal Memory: 63.02GiB\r\nName: node5\r\nID: JRVQ:2D26:NA5B:TJ5L:VWSC:TEAE:G4B2:UXYF:3TRR:F7YY:LRW5:3IWA\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nUsername: filiatixbot\r\nRegistry: https://index.docker.io/v1/\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\nWARNING: No kernel memory limit support\r\nWARNING: No cpu cfs quota support\r\nWARNING: No cpu cfs period support\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/330251501","html_url":"https://github.com/moby/moby/issues/33589#issuecomment-330251501","issue_url":"https://api.github.com/repos/moby/moby/issues/33589","id":330251501,"user":{"login":"jgranstrom","id":3751473,"avatar_url":"https://avatars1.githubusercontent.com/u/3751473?v=4","gravatar_id":"","url":"https://api.github.com/users/jgranstrom","html_url":"https://github.com/jgranstrom","followers_url":"https://api.github.com/users/jgranstrom/followers","following_url":"https://api.github.com/users/jgranstrom/following{/other_user}","gists_url":"https://api.github.com/users/jgranstrom/gists{/gist_id}","starred_url":"https://api.github.com/users/jgranstrom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgranstrom/subscriptions","organizations_url":"https://api.github.com/users/jgranstrom/orgs","repos_url":"https://api.github.com/users/jgranstrom/repos","events_url":"https://api.github.com/users/jgranstrom/events{/privacy}","received_events_url":"https://api.github.com/users/jgranstrom/received_events","type":"User","site_admin":false},"created_at":"2017-09-18T15:01:12Z","updated_at":"2017-09-18T15:01:12Z","author_association":"NONE","body":"@taiidani what cloud provider are you using? Are you ever seeing any sporadic connectivity issues outside of docker swarm on the host itself? For example when you simply try a lot of pings or similar between the nodes or to some other external source over a few hours to a day."}}
{"actor":{"display_login":"jgranstrom"}}