{"action":"created","issue":{"url":"https://api.github.com/repos/kamailio/kamailio/issues/1172","repository_url":"https://api.github.com/repos/kamailio/kamailio","labels_url":"https://api.github.com/repos/kamailio/kamailio/issues/1172/labels{/name}","comments_url":"https://api.github.com/repos/kamailio/kamailio/issues/1172/comments","events_url":"https://api.github.com/repos/kamailio/kamailio/issues/1172/events","html_url":"https://github.com/kamailio/kamailio/issues/1172","id":239190560,"number":1172,"title":"Kamailio 5.0.2 suddenly stops processing traffic, then generates a core when restarting.","user":{"login":"joelsdc","id":16212586,"avatar_url":"https://avatars2.githubusercontent.com/u/16212586?v=3","gravatar_id":"","url":"https://api.github.com/users/joelsdc","html_url":"https://github.com/joelsdc","followers_url":"https://api.github.com/users/joelsdc/followers","following_url":"https://api.github.com/users/joelsdc/following{/other_user}","gists_url":"https://api.github.com/users/joelsdc/gists{/gist_id}","starred_url":"https://api.github.com/users/joelsdc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joelsdc/subscriptions","organizations_url":"https://api.github.com/users/joelsdc/orgs","repos_url":"https://api.github.com/users/joelsdc/repos","events_url":"https://api.github.com/users/joelsdc/events{/privacy}","received_events_url":"https://api.github.com/users/joelsdc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-06-28T14:50:23Z","updated_at":"2017-06-29T08:08:29Z","closed_at":null,"body":"### Description\r\n\r\nKamailio suddenly stops working (this is a production server so debug logging is disabled), when we restart it a core is generated and a segfault is logged.\r\n\r\n### Troubleshooting\r\n\r\nThe only thing I have seen is some reference to IPv6 in the backtrace. But no idea where to start looking. I'm pretty sure we will need more information, but I don't know how to read or understand backtraces, please let me know what you see in it so I can dig deeper into this problem.\r\n\r\n#### Reproduction\r\n\r\nI don't know how to reproduce yet, first I have to understand the cause.\r\n\r\n#### Debugging Data\r\n\r\nBacktrace:\r\n```\r\n[New LWP 12052]\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\r\nCore was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/sbc.cfg -'.\r\nProgram terminated with signal SIGSEGV, Segmentation fault.\r\n#0  0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1\r\n#0  0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1\r\nNo symbol table info available.\r\n#1  0x00007efcf3b5e234 in OPENSSL_cleanup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1\r\nNo symbol table info available.\r\n#2  0x00007efcf7721910 in __run_exit_handlers (status=0, listp=0x7efcf7a855d8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83\r\n        atfct = <optimized out>\r\n        onfct = <optimized out>\r\n        cxafct = <optimized out>\r\n        f = <optimized out>\r\n#3  0x00007efcf772196a in __GI_exit (status=<optimized out>) at exit.c:105\r\nNo locals.\r\n#4  0x000055f0a9483d54 in handle_sigs () at main.c:699\r\n        chld = 845005216\r\n        chld_status = 845005248\r\n        any_chld_stopped = 32766\r\n        memlog = 0\r\n        __func__ = \"handle_sigs\"\r\n#5  0x000055f0a948ea43 in main_loop () at main.c:1758\r\n        i = 8\r\n        pid = 12091\r\n        si = 0x0\r\n        si_desc = \"udp receiver child=7 sock=[2602:FF37:0:1:0:0:C601:377A]:5060\\000\\367v\\215\\200\\262+\\363\\372~\\000\\000\\025\\067\\236\\003\\000\\000\\000\\000\\300\\017H\\251\\360U\\000\\000\\340\\305]2\\376\\177\", '\\000' <repeats 18 times>, \"\\020\\303]2\\376\\177\\000\\000a\\202g\\251\\360U\\000\"\r\n        nrprocs = 8\r\n        woneinit = 1\r\n        __func__ = \"main_loop\"\r\n#6  0x000055f0a9494fb9 in main (argc=13, argv=0x7ffe325dc5e8) at main.c:2646\r\n        cfg_stream = 0x55f0aa7fb010\r\n        c = -1\r\n        r = 0\r\n        tmp = 0x7ffe325dceb6 \"\"\r\n        tmp_len = -133383536\r\n        port = 32508\r\n        proto = 845006016\r\n        options = 0x55f0a97d80e8 \":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:\"\r\n        ret = -1\r\n        seed = 3679304743\r\n        rfd = 4\r\n        debug_save = 0\r\n        debug_flag = 0\r\n        dont_fork_cnt = 0\r\n        n_lst = 0x0\r\n        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n        st = {st_dev = 19, st_ino = 14769, st_nlink = 2, st_mode = 16832, st_uid = 109, st_gid = 113, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1498523393, tv_nsec = 731915583}, st_mtim = {tv_sec = 1498533865, tv_nsec = 274210743}, st_ctim = {tv_sec = 1498533865, tv_nsec = 274210743}, __glibc_reserved = {0, 0, 0}}\r\n        __func__ = \"main\"\r\n```\r\n\r\nInfo locals:\r\n```\r\n[New LWP 12052]\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\r\nCore was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/sbc.cfg -'.\r\nProgram terminated with signal SIGSEGV, Segmentation fault.\r\n#0  0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1\r\nNo symbol table info available.\r\n```\r\n\r\nList:\r\n```\r\n[New LWP 12052]\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\r\nCore was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/sbc.cfg -'.\r\nProgram terminated with signal SIGSEGV, Segmentation fault.\r\n#0  0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1\r\n```\r\n\r\n#### Log Messages\r\n\r\nWe have regular logging with no issues, then suddenly we get:\r\n\r\n```\r\nJun 28 01:57:51 kam-1 sbc[12075]: WARNING: <script>: [DISPATCHER] - Destination down: OPTIONS sip:A.A.A.A:5060 (<null>)\r\nJun 28 01:57:51 kam-1 sbc[12075]: WARNING: <script>: [DISPATCHER] - Destination down: OPTIONS sip:B.B.B.B:5060 (<null>)\r\nJun 28 01:58:40 kam-1 sbc[12076]: NOTICE: <script>: Node unavailable, trying next sip:A.A.A.A:5060 - M=INVITE R=sip:+12345678900@A.A.A.A:5060 ID=334f3vqDtj\r\nJun 28 01:58:40 kam-1 sbc[12076]: NOTICE: <script>: Forwarding request - M=INVITE R=sip:+12345678900@A.A.A.A:5060 ID=334f3vqDtj\r\nJun 28 01:58:55 kam-1 sbc[12075]: WARNING: dialog [dlg_handlers.c:1574]: dlg_ontimeout(): timeout for dlg with CallID '00g4gNy5KZ' and tags 'FHmrD-CM3' 'SZ3a5Sg58c5pB'\r\nJun 28 01:58:55 kam-1 sbc[12075]: WARNING: dialog [dlg_handlers.c:1574]: dlg_ontimeout(): timeout for dlg with CallID 'Kj7uLRzJfd' and tags 'wRSVjfGaT' 'yQHvjNaZ5Z3rc'\r\nJun 28 01:58:55 kam-1 sbc[12075]: WARNING: dialog [dlg_handlers.c:1574]: dlg_ontimeout(): timeout for dlg with CallID 'S5QUIXtS6-' and tags 'FV4ySEouz' 'eS7F910ycNg8H'\r\nJun 28 01:59:10 kam-1 sbc[12075]: ERROR: <script>: No available gateways - M=INVITE R=sip:+12345678900@B.B.B.B:5060 ID=334f3vqDtj\r\nJun 28 02:01:45 kam-1 sbc[12080]: NOTICE: dialog [dlg_hash.c:248]: dlg_clean_run(): dialog in delete state is too old (0x7efaf3904008 ref 1)\r\nJun 28 02:03:15 kam-1 sbc[12080]: NOTICE: dialog [dlg_hash.c:248]: dlg_clean_run(): dialog in delete state is too old (0x7efaf40f0f50 ref 1)\r\nJun 28 02:04:45 kam-1 sbc[12080]: NOTICE: dialog [dlg_hash.c:248]: dlg_clean_run(): dialog in delete state is too old (0x7efaf4d92108 ref 1)\r\nJun 28 02:55:59 kam-1 sbc[12071]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received\r\nJun 28 02:55:59 kam-1 sbc[12052]: NOTICE: <core> [main.c:696]: handle_sigs(): Thank you for flying kamailio!!!\r\n```\r\n\r\nNOTE: The process stops logging at 02:04:45, then at 02:55:59 we manually restart kamailio and a core is generated.\r\n\r\nAlso, in `dmesg` we have:\r\n\r\n```\r\n[138953.744199] kamailio[12052]: segfault at 7efaf32f51b8 ip 00007efcf3b5df69 sp 00007ffe325dc0c0 error 4 in libcrypto.so.1.1[7efcf39f6000+265000]\r\n```\r\n\r\n\r\n#### SIP Traffic\r\n\r\nI don't have sip traffic, If required I can start capturing it.\r\n\r\n### Additional Information\r\n\r\n```\r\n# kamailio -v\r\nversion: kamailio 5.0.2 (x86_64/linux)\r\nflags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES\r\nADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB\r\npoll method support: poll, epoll_lt, epoll_et, sigio_rt, select.\r\nid: unknown\r\ncompiled with gcc 6.3.0\r\n#\r\n```\r\n\r\n```\r\n# dpkg -l | grep kamailio\r\nii  kamailio                         5.0.2+stretch                  amd64        very fast, dynamic and configurable SIP server\r\nii  kamailio-dbg:amd64               5.0.2+stretch                  amd64        very fast and configurable SIP server [debug symbols]\r\nii  kamailio-extra-modules:amd64     5.0.2+stretch                  amd64        Extra modules for the Kamailio SIP Server\r\nii  kamailio-tls-modules:amd64       5.0.2+stretch                  amd64        TLS support for the Kamailio SIP server (authentication, transport)\r\n#\r\n```\r\n\r\n```\r\n# dpkg -l |grep  libssl\r\nii  libssl1.0.2:amd64                1.0.2l-2                       amd64        Secure Sockets Layer toolkit - shared libraries\r\nii  libssl1.1:amd64                  1.1.0f-3                       amd64        Secure Sockets Layer toolkit - shared libraries#\r\n```\r\n\r\n* **Operating System**:\r\n\r\n```\r\n# uname -a\r\nLinux kam-1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1 (2017-06-18) x86_64 GNU/Linux\r\n```\r\n\r\n```\r\n# lsb_release -a\r\nNo LSB modules are available.\r\nDistributor ID:\tDebian\r\nDescription:\tDebian GNU/Linux 9.0 (stretch)\r\nRelease:\t9.0\r\nCodename:\tstretch\r\n#\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kamailio/kamailio/issues/comments/311893980","html_url":"https://github.com/kamailio/kamailio/issues/1172#issuecomment-311893980","issue_url":"https://api.github.com/repos/kamailio/kamailio/issues/1172","id":311893980,"user":{"login":"joelsdc","id":16212586,"avatar_url":"https://avatars2.githubusercontent.com/u/16212586?v=3","gravatar_id":"","url":"https://api.github.com/users/joelsdc","html_url":"https://github.com/joelsdc","followers_url":"https://api.github.com/users/joelsdc/followers","following_url":"https://api.github.com/users/joelsdc/following{/other_user}","gists_url":"https://api.github.com/users/joelsdc/gists{/gist_id}","starred_url":"https://api.github.com/users/joelsdc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joelsdc/subscriptions","organizations_url":"https://api.github.com/users/joelsdc/orgs","repos_url":"https://api.github.com/users/joelsdc/repos","events_url":"https://api.github.com/users/joelsdc/events{/privacy}","received_events_url":"https://api.github.com/users/joelsdc/received_events","type":"User","site_admin":false},"created_at":"2017-06-29T08:08:29Z","updated_at":"2017-06-29T08:08:29Z","body":"Ok! I'll wait until it happens again. \r\n\r\nSome extra info:\r\n\r\n1- Pike is enabled, but we have logging when it blocks something (and no logs appeared).\r\n2- CPU is always < 1% usage.\r\n3- I have enabled kernel.core_uses_pid=1. \r\n4- Our traffic is:\r\n\r\n[clients]--> IPv4/IPv6 tls_443 [Kamailio] IPv4 udp_5060 --> IPv4 udp_5060 [dispatcher setid 4].\r\n\r\nIf I run the bt full right now to a worker process, should we get random stuff?\r\n\r\nBecase right now I'm getting mostly the same on each process:\r\n\r\n```\r\nroot:~# for i in `kamctl ps | grep PID | awk '{print $2}' | tr -d \",\"`; do gdb /usr/sbin/kamailio -ex \"bt full\" --batch $i >> $i.txt 2>&1; done\r\nroot:~# \r\n\r\nroot:~# ls -l\r\ntotal 140\r\n-rw-r--r-- 1 root root 1769 Jun 29 01:04 23585.txt\r\n-rw-r--r-- 1 root root 3300 Jun 29 01:04 23586.txt\r\n-rw-r--r-- 1 root root 3300 Jun 29 01:04 23587.txt\r\n-rw-r--r-- 1 root root 3301 Jun 29 01:04 23588.txt\r\n-rw-r--r-- 1 root root 3300 Jun 29 01:04 23589.txt\r\n-rw-r--r-- 1 root root 3301 Jun 29 01:04 23590.txt\r\n-rw-r--r-- 1 root root 3300 Jun 29 01:04 23591.txt\r\n-rw-r--r-- 1 root root 3301 Jun 29 01:04 23592.txt\r\n-rw-r--r-- 1 root root 3301 Jun 29 01:04 23593.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23594.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23595.txt\r\n-rw-r--r-- 1 root root 3431 Jun 29 01:04 23596.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23597.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23598.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23599.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23600.txt\r\n-rw-r--r-- 1 root root 3432 Jun 29 01:04 23601.txt\r\n-rw-r--r-- 1 root root 2388 Jun 29 01:04 23602.txt\r\n-rw-r--r-- 1 root root 1836 Jun 29 01:04 23603.txt\r\n-rw-r--r-- 1 root root 2245 Jun 29 01:04 23604.txt\r\n-rw-r--r-- 1 root root 4023 Jun 29 01:04 23605.txt\r\n-rw-r--r-- 1 root root 2950 Jun 29 01:04 23606.txt\r\n-rw-r--r-- 1 root root 2837 Jun 29 01:04 23607.txt\r\n-rw-r--r-- 1 root root 4691 Jun 29 01:04 23608.txt\r\n-rw-r--r-- 1 root root 2740 Jun 29 01:04 23609.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23610.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23611.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23612.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23613.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23614.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23615.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23616.txt\r\n-rw-r--r-- 1 root root 2691 Jun 29 01:04 23617.txt\r\n-rw-r--r-- 1 root root 2151 Jun 29 01:04 23618.txt\r\nroot:~#\r\n\r\nroot:~# grep -n ffff *\r\n23585.txt:31:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23586.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23587.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23588.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23589.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23590.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23591.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23592.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23593.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23594.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23595.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23596.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23597.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23598.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23599.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23600.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23601.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23602.txt:40:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23603.txt:33:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23604.txt:39:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23605.txt:75:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23606.txt:51:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23607.txt:49:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23608.txt:75:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23609.txt:50:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23610.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23611.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23612.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23613.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23614.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23615.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23616.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23617.txt:47:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\n23618.txt:41:        p = 0xffffffff <error: Cannot access memory at address 0xffffffff>\r\nroot:~#\r\n```\r\n\r\nAm I missing something or am I doing anything wrong here?"}}
{"actor":{"display_login":"joelsdc"}}