{"action":"created","issue":{"url":"https://api.github.com/repos/joyent/illumos-joyent/issues/137","repository_url":"https://api.github.com/repos/joyent/illumos-joyent","labels_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/137/labels{/name}","comments_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/137/comments","events_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/137/events","html_url":"https://github.com/joyent/illumos-joyent/issues/137","id":206169448,"number":137,"title":"netatalk causes a CN to become unresponsive under certain conditions","user":{"login":"themisanthrope","id":8216030,"avatar_url":"https://avatars1.githubusercontent.com/u/8216030?v=4","gravatar_id":"","url":"https://api.github.com/users/themisanthrope","html_url":"https://github.com/themisanthrope","followers_url":"https://api.github.com/users/themisanthrope/followers","following_url":"https://api.github.com/users/themisanthrope/following{/other_user}","gists_url":"https://api.github.com/users/themisanthrope/gists{/gist_id}","starred_url":"https://api.github.com/users/themisanthrope/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/themisanthrope/subscriptions","organizations_url":"https://api.github.com/users/themisanthrope/orgs","repos_url":"https://api.github.com/users/themisanthrope/repos","events_url":"https://api.github.com/users/themisanthrope/events{/privacy}","received_events_url":"https://api.github.com/users/themisanthrope/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-02-08T11:23:39Z","updated_at":"2017-09-01T21:44:49Z","closed_at":"2017-09-01T21:44:49Z","author_association":"NONE","body":"Reproducible on at least two platforms, the most recent being `20170119T014129Z`.\r\n\r\n```\r\n{\r\n  \"brand\": \"joyent\",\r\n  \"limit_priv\": \"default\",\r\n  \"v\": 1,\r\n  \"create_timestamp\": \"2016-10-20T16:11:16.479Z\",\r\n  \"image_uuid\": \"adf9565c-8be6-11e6-a077-57637270218d\",\r\n  \"max_msg_ids\": 4096,\r\n  \"max_sem_ids\": 4096,\r\n  \"max_shm_ids\": 4096,\r\n  \"max_shm_memory\": 1536,\r\n  \"nics\": [\r\n    {\r\n      \"interface\": \"net0\",\r\n      \"gateway\": \"10.0.48.129\",\r\n      \"gateways\": [\r\n        \"10.0.48.129\"\r\n      ],\r\n      \"netmask\": \"255.255.255.128\",\r\n      \"ip\": \"10.0.48.133\",\r\n      \"ips\": [\r\n        \"10.0.48.133/25\"\r\n      ],\r\n      \"mtu\": 1500,\r\n      \"primary\": true\r\n    }\r\n  ],\r\n  \"resolvers\": [\r\n    \"10.0.0.133\",\r\n    \"10.0.0.134\"\r\n  ],\r\n  \"cpu_shares\": 12,\r\n  \"max_lwps\": 4000,\r\n  \"zfs_io_priority\": 16000,\r\n  \"max_physical_memory\": 1536,\r\n  \"max_locked_memory\": 1536,\r\n  \"max_swap\": 3072,\r\n  \"cpu_cap\": 500,\r\n  \"zone_state\": \"running\",\r\n  \"zoneid\": 1,\r\n  \"last_modified\": \"2017-02-05T06:44:04.000Z\",\r\n  \"firewall_enabled\": false,\r\n  \"platform_buildstamp\": \"20170119T014129Z\",\r\n  \"state\": \"running\",\r\n  \"boot_timestamp\": \"2017-01-25T19:19:23.000Z\",\r\n  \"pid\": 5040,\r\n  \"tags\": {\r\n    \"triton.cns.services\": \"unix,images-hot,afp-images\",\r\n    \"os\": 20113,\r\n    \"backup\": true\r\n  },\r\n  \"quota\": 650,\r\n  \"zfs_root_compression\": \"lz4\",\r\n  \"zfs_root_recsize\": 131072,\r\n  \"zpool\": \"zones\",\r\n}\r\n```\r\n_Some information has been removed from this zone description_\r\n\r\nExtra packages installed from `pkgsrc`: `ansible`, `gcc47`, `gmake`, `libevent`, `libgcrypt`, `openssl`, `syncthing`\r\n\r\nSoftware installed from source: `http://downloads.sourceforge.net/project/netatalk/netatalk/3.1.10/netatalk-3.1.10.tar.gz`\r\n\r\nServices enabled: `smb/server`, `smb/client`, `rpc/bind`, `idmap`, `netatalk`, `syncthing (custom manifest)`\r\n\r\nExtra `pam` services: `pam_smb_passwd`\r\n\r\nZone purpose: Act as an AFP and SMB fileserver, hosting assets used by the Design and Marketing teams. The vast majority of users connect with AFP, with a handful of SMB connections for those unfortunate enough to use Windows.\r\n\r\n--------------------------\r\n\r\nActing as a standalone fileserver, this zone functions as expected. Syncthing is used to replicate assets with another fileserver (of the same configuration) running on a different CN.\r\n\r\nThere are another six file servers (three pairs) with the same configuration, but hosting different files.\r\n\r\nTo improve performance, load is balanced between the pairs of hosts using `haproxy`, which runs on our routers (`FreeBSD 10.3-RELEASE-p9`). This also works fine, until we start using TCP availability checks.\r\nThe availability checks run against port 548, and only check that the port is open. If `netatalk` dies, the port gets closed and the fileserver is removed from the pool.\r\n\r\nEach CN has four of these zones running, with an availability check running every 1000ms. After about 10 hours, the CNs became unresponsive;\r\n\r\n- Unable to connect by SSH\r\n- CN agent stopped responding, causing the host to be marked **UNAVAILABLE** by `Triton`\r\n- `Zabbix` agent stopped responding, causing the host to be marked as **DOWN** by `Zabbix`\r\n- Some other zones on these CNs stopped responding altogether\r\n- Other zones on these CNs started to act unpredictably\r\n- The CNs still responded to ping\r\n- Logging in to the CNs was still possible via IPMI SOL and local console\r\n- When logged in, some commands (like `vmadm *`) cause the console to hang and never return\r\n- When logged in, some commands (like `zpool *`) function without issue and provide sane information\r\n\r\nI could only find two ways to get the CNs out of this state: Issue `reboot` from the IPMI SOL or local console (which would cause the CN to switch back to **RUNNING** in `Triton` momentarily before the reboot, or to send an IPMI NMI.\r\n\r\nI did try manually killing the zone processes, but this didn't help.\r\n\r\nI can make the dumps produced by the NMI instruction available, from both CNs. There are several.\r\n\r\n--------------------------\r\n\r\nI've resolved this issue internally by using a different method for health checking and feeding the information to `haproxy`, and I suspect that root of the issue may actually be with `netatalk` (I'll be raising an issue with them, too).\r\n\r\nI'm raising this issue here because a fault inside a zone or with a zone process shouldn't (in my mind) have the potential to take out a whole host machine, and I'm hoping that the information here (along with the dumps) might help with isolating a potential OS availability bug too."},"comment":{"url":"https://api.github.com/repos/joyent/illumos-joyent/issues/comments/326690389","html_url":"https://github.com/joyent/illumos-joyent/issues/137#issuecomment-326690389","issue_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/137","id":326690389,"user":{"login":"themisanthrope","id":8216030,"avatar_url":"https://avatars1.githubusercontent.com/u/8216030?v=4","gravatar_id":"","url":"https://api.github.com/users/themisanthrope","html_url":"https://github.com/themisanthrope","followers_url":"https://api.github.com/users/themisanthrope/followers","following_url":"https://api.github.com/users/themisanthrope/following{/other_user}","gists_url":"https://api.github.com/users/themisanthrope/gists{/gist_id}","starred_url":"https://api.github.com/users/themisanthrope/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/themisanthrope/subscriptions","organizations_url":"https://api.github.com/users/themisanthrope/orgs","repos_url":"https://api.github.com/users/themisanthrope/repos","events_url":"https://api.github.com/users/themisanthrope/events{/privacy}","received_events_url":"https://api.github.com/users/themisanthrope/received_events","type":"User","site_admin":false},"created_at":"2017-09-01T21:44:49Z","updated_at":"2017-09-01T21:44:49Z","author_association":"NONE","body":"This appears to have been rectified in a platform update."}}
{"actor":{"display_login":"themisanthrope"}}