{"action":"created","issue":{"url":"https://api.github.com/repos/NagiosEnterprises/nagioscore/issues/403","repository_url":"https://api.github.com/repos/NagiosEnterprises/nagioscore","labels_url":"https://api.github.com/repos/NagiosEnterprises/nagioscore/issues/403/labels{/name}","comments_url":"https://api.github.com/repos/NagiosEnterprises/nagioscore/issues/403/comments","events_url":"https://api.github.com/repos/NagiosEnterprises/nagioscore/issues/403/events","html_url":"https://github.com/NagiosEnterprises/nagioscore/issues/403","id":246049281,"number":403,"title":"Nagios 4.3.2 not starting anymore --> wproc: 'Core Worker xxx' seems to be choked","user":{"login":"Fr3dY2","id":17852642,"avatar_url":"https://avatars2.githubusercontent.com/u/17852642?v=4","gravatar_id":"","url":"https://api.github.com/users/Fr3dY2","html_url":"https://github.com/Fr3dY2","followers_url":"https://api.github.com/users/Fr3dY2/followers","following_url":"https://api.github.com/users/Fr3dY2/following{/other_user}","gists_url":"https://api.github.com/users/Fr3dY2/gists{/gist_id}","starred_url":"https://api.github.com/users/Fr3dY2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fr3dY2/subscriptions","organizations_url":"https://api.github.com/users/Fr3dY2/orgs","repos_url":"https://api.github.com/users/Fr3dY2/repos","events_url":"https://api.github.com/users/Fr3dY2/events{/privacy}","received_events_url":"https://api.github.com/users/Fr3dY2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-07-27T13:59:56Z","updated_at":"2017-07-27T21:25:25Z","closed_at":null,"body":"Hi,\r\n\r\nI've been using Nagios v4.3.2 on CentOS 7.3 without problems, but I just rebooted the server after some package updates (including nagios-commons) and it fails to start now. There were some 'needs-restarting' packages, maybe from earlier updates, affecting httpd as well (maybe kernel too).\r\n\r\nNow the service is not starting anymore, with the following error messages in nagios.log:\r\n```\r\n[1501162592] SERVICE ALERT: S-JRCSVQFS006P;Volume incremental usage [1 hour max] - Users;OK;SOFT;2;Interval reached (60 minutes) or disk usage reduced, size reset to 3889GB (from 3892GB)\r\n[1501162592] Caught SIGTERM, shutting down...\r\n[1501162592] Caught SIGTERM, shutting down...\r\n[1501162592] wproc: Socket to worker Core Worker 1028 broken, removing\r\n[1501162592] wproc: Socket to worker Core Worker 1029 broken, removing\r\n[1501162592] wproc: Socket to worker Core Worker 1026 broken, removing\r\n[1501162592] wproc: 'Core Worker 1027' seems to be choked. ret = -1; bufsize = 125: written = 0; errno = 32 (Broken pipe)\r\n[1501162592] wproc: Socket to worker Core Worker 1027 broken, removing\r\n[1501162593] Successfully shutdown... (PID=1025)\r\n[1501162593] Event broker module 'NERD' deinitialized successfully.\r\n```\r\n\r\n\r\nI tried again and it fails again, without the core worker messages:\r\n\r\n```\r\n[1501163556] Successfully launched command file worker with pid 4205\r\n[1501163584] HOST ALERT: S-JRCSVQBK002P;DOWN;SOFT;1;CRITICAL - Host Unreachable (139.191.152.38)\r\n[1501163621] SERVICE ALERT: S-JRCSVQBK002P;Cpu_Load;CRITICAL;SOFT;1;connect to address 139.191.152.38 and port 12489: No route to host\r\n[1501163624] HOST ALERT: S-JRCSVQBK002P;DOWN;HARD;2;CRITICAL - Host Unreachable (139.191.152.38)\r\n[1501163646] Caught SIGTERM, shutting down...\r\n[1501163646] Caught SIGTERM, shutting down...\r\n[1501163646] Successfully shutdown... (PID=4200)\r\n[1501163646] Event broker module 'NERD' deinitialized successfully.\r\n```\r\n\r\n\r\n[root@S-JRCSVQNG001P nagios]# systemctl status nagios.service\r\n```\r\n● nagios.service - Nagios Network Monitoring\r\n   Loaded: loaded (/usr/lib/systemd/system/nagios.service; disabled; vendor preset: disabled)\r\n   Active: failed (Result: timeout) since Thu 2017-07-27 15:54:06 CEST; 16s ago\r\n     Docs: https://www.nagios.org/documentation/\r\n  Process: 4199 ExecStart=/usr/sbin/nagios -d /etc/nagios/nagios.cfg (code=exited, status=0/SUCCESS)\r\n  Process: 4196 ExecStartPre=/usr/sbin/nagios -v /etc/nagios/nagios.cfg (code=exited, status=0/SUCCESS)\r\n\r\nJul 27 15:53:41 S-JRCSVQNG001P nagios[4200]: SERVICE ALERT: S-JRCSVQBK002P;Cpu_Load;CRITICAL;SOFT;1;connect to address 139.191.152.38 and port 12489: No route to host\r\nJul 27 15:53:44 S-JRCSVQNG001P nagios[4200]: HOST ALERT: S-JRCSVQBK002P;DOWN;HARD;2;CRITICAL - Host Unreachable (139.191.152.38)\r\nJul 27 15:54:06 S-JRCSVQNG001P systemd[1]: nagios.service start operation timed out. Terminating.\r\nJul 27 15:54:06 S-JRCSVQNG001P nagios[4200]: Caught SIGTERM, shutting down...\r\nJul 27 15:54:06 S-JRCSVQNG001P nagios[4205]: Caught SIGTERM, shutting down...\r\nJul 27 15:54:06 S-JRCSVQNG001P nagios[4200]: Successfully shutdown... (PID=4200)\r\nJul 27 15:54:06 S-JRCSVQNG001P nagios[4200]: Event broker module 'NERD' deinitialized successfully.\r\nJul 27 15:54:06 S-JRCSVQNG001P systemd[1]: Failed to start Nagios Network Monitoring.\r\nJul 27 15:54:06 S-JRCSVQNG001P systemd[1]: Unit nagios.service entered failed state.\r\nJul 27 15:54:06 S-JRCSVQNG001P systemd[1]: nagios.service failed.\r\n```\r\n\r\nAnother issue I've noticed is the following error message:\r\n\r\n`[1501162920] Error: Could not open check result queue directory '/var/nagios/spool/checkresults' for reading.\r\n`\r\n\r\nThat folder is included in the nagios.cfg file, but it wasn't in the filesystem. Was it deleted somehow?\r\nI just created it but service fails with above messages.\r\n\r\nUPDATE: It seems CentOS repository now includes Nagios 4.3.2 officially and it got updated (I was using a compiled version). I'll try to look into it, but this update of the official package over the compiled version broke something and maybe it overwrote my nagios.cfg file...  it seems I've lost the historical alert logs and everything is being checked as for the first time :S\r\n\r\nIs it possible to recover my historical alert logs? The /var/log/nagios/archives is populated with files, but from Nagios I can only see updates from today :(\r\n\r\n\r\n\r\nRegards,"},"comment":{"url":"https://api.github.com/repos/NagiosEnterprises/nagioscore/issues/comments/318491008","html_url":"https://github.com/NagiosEnterprises/nagioscore/issues/403#issuecomment-318491008","issue_url":"https://api.github.com/repos/NagiosEnterprises/nagioscore/issues/403","id":318491008,"user":{"login":"box293","id":8356113,"avatar_url":"https://avatars0.githubusercontent.com/u/8356113?v=4","gravatar_id":"","url":"https://api.github.com/users/box293","html_url":"https://github.com/box293","followers_url":"https://api.github.com/users/box293/followers","following_url":"https://api.github.com/users/box293/following{/other_user}","gists_url":"https://api.github.com/users/box293/gists{/gist_id}","starred_url":"https://api.github.com/users/box293/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/box293/subscriptions","organizations_url":"https://api.github.com/users/box293/orgs","repos_url":"https://api.github.com/users/box293/repos","events_url":"https://api.github.com/users/box293/events{/privacy}","received_events_url":"https://api.github.com/users/box293/received_events","type":"User","site_admin":false},"created_at":"2017-07-27T21:25:25Z","updated_at":"2017-07-27T21:25:25Z","body":"> Is it possible to recover my historical alert logs? The /var/log/nagios/archives is populated with files, but from Nagios I can only see updates from today :(\r\n\r\nIt sounds like the `log_archive_path=` in your nagios.cfg is not pointing to the correct location.\r\n\r\nIn this sort of scenario it would help if you could roll back your server to a backup before this happened. Installing a repo over a compiled can add all sorts of problems.\r\n\r\nIf this is not possible, I've had to fix a similar issue before and the steps you would need to follow are:\r\n\r\n- Take copies of all config files and files / directories referenced by nagios.cfg\r\n- Uninstall the repo install of Nagios via yum\r\n- Re-install nagios from source\r\n- Replace your original nagios config files you took a backup of\r\n- cross fingers\r\n\r\nHopefully you at least have a copy of your nagios.cfg in a backup as this may have been overwritten by the repo install."}}
{"actor":{"display_login":"box293"}}