{"action":"created","issue":{"url":"https://api.github.com/repos/tempesta-tech/tempesta/issues/696","repository_url":"https://api.github.com/repos/tempesta-tech/tempesta","labels_url":"https://api.github.com/repos/tempesta-tech/tempesta/issues/696/labels{/name}","comments_url":"https://api.github.com/repos/tempesta-tech/tempesta/issues/696/comments","events_url":"https://api.github.com/repos/tempesta-tech/tempesta/issues/696/events","html_url":"https://github.com/tempesta-tech/tempesta/issues/696","id":213171272,"number":696,"title":"kernel BUG at /root/ab/tempesta-ab/tempesta_fw/http.c:1321","user":{"login":"keshonok","id":6352580,"avatar_url":"https://avatars0.githubusercontent.com/u/6352580?v=3","gravatar_id":"","url":"https://api.github.com/users/keshonok","html_url":"https://github.com/keshonok","followers_url":"https://api.github.com/users/keshonok/followers","following_url":"https://api.github.com/users/keshonok/following{/other_user}","gists_url":"https://api.github.com/users/keshonok/gists{/gist_id}","starred_url":"https://api.github.com/users/keshonok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keshonok/subscriptions","organizations_url":"https://api.github.com/users/keshonok/orgs","repos_url":"https://api.github.com/users/keshonok/repos","events_url":"https://api.github.com/users/keshonok/events{/privacy}","received_events_url":"https://api.github.com/users/keshonok/received_events","type":"User","site_admin":false},"labels":[{"id":123036026,"url":"https://api.github.com/repos/tempesta-tech/tempesta/labels/bug","name":"bug","color":"ff8030","default":true},{"id":130671020,"url":"https://api.github.com/repos/tempesta-tech/tempesta/labels/crucial","name":"crucial","color":"ff1010","default":false}],"state":"open","locked":false,"assignee":{"login":"keshonok","id":6352580,"avatar_url":"https://avatars0.githubusercontent.com/u/6352580?v=3","gravatar_id":"","url":"https://api.github.com/users/keshonok","html_url":"https://github.com/keshonok","followers_url":"https://api.github.com/users/keshonok/followers","following_url":"https://api.github.com/users/keshonok/following{/other_user}","gists_url":"https://api.github.com/users/keshonok/gists{/gist_id}","starred_url":"https://api.github.com/users/keshonok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keshonok/subscriptions","organizations_url":"https://api.github.com/users/keshonok/orgs","repos_url":"https://api.github.com/users/keshonok/repos","events_url":"https://api.github.com/users/keshonok/events{/privacy}","received_events_url":"https://api.github.com/users/keshonok/received_events","type":"User","site_admin":false},"assignees":[{"login":"keshonok","id":6352580,"avatar_url":"https://avatars0.githubusercontent.com/u/6352580?v=3","gravatar_id":"","url":"https://api.github.com/users/keshonok","html_url":"https://github.com/keshonok","followers_url":"https://api.github.com/users/keshonok/followers","following_url":"https://api.github.com/users/keshonok/following{/other_user}","gists_url":"https://api.github.com/users/keshonok/gists{/gist_id}","starred_url":"https://api.github.com/users/keshonok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keshonok/subscriptions","organizations_url":"https://api.github.com/users/keshonok/orgs","repos_url":"https://api.github.com/users/keshonok/repos","events_url":"https://api.github.com/users/keshonok/events{/privacy}","received_events_url":"https://api.github.com/users/keshonok/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/tempesta-tech/tempesta/milestones/3","html_url":"https://github.com/tempesta-tech/tempesta/milestone/3","labels_url":"https://api.github.com/repos/tempesta-tech/tempesta/milestones/3/labels","id":1015684,"number":3,"title":"0.5.0 Web Server","description":"","creator":{"login":"krizhanovsky","id":884694,"avatar_url":"https://avatars0.githubusercontent.com/u/884694?v=3","gravatar_id":"","url":"https://api.github.com/users/krizhanovsky","html_url":"https://github.com/krizhanovsky","followers_url":"https://api.github.com/users/krizhanovsky/followers","following_url":"https://api.github.com/users/krizhanovsky/following{/other_user}","gists_url":"https://api.github.com/users/krizhanovsky/gists{/gist_id}","starred_url":"https://api.github.com/users/krizhanovsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krizhanovsky/subscriptions","organizations_url":"https://api.github.com/users/krizhanovsky/orgs","repos_url":"https://api.github.com/users/krizhanovsky/repos","events_url":"https://api.github.com/users/krizhanovsky/events{/privacy}","received_events_url":"https://api.github.com/users/krizhanovsky/received_events","type":"User","site_admin":false},"open_issues":38,"closed_issues":179,"state":"open","created_at":"2015-03-10T23:31:23Z","updated_at":"2017-03-09T21:30:33Z","due_on":"2017-03-31T07:00:00Z","closed_at":null},"comments":1,"created_at":"2017-03-09T21:30:33Z","updated_at":"2017-03-09T21:55:41Z","closed_at":null,"body":"The setup:\r\n* Tempesta's current master at 56b0ecb;\r\n* Tempesta kernel 4.8.15 at [5b3aa7f](https://github.com/tempesta-tech/linux-4.8.15-tfw/commit/5b3aa7fe30ce46bb8f112a9544548da926cb3714);\r\n* Both Tempesta and Nginx run on the same physical test server with 12 CPU cores and 64Gb RAM;\r\n* Simplest Tempesta configuration:\r\n```\r\nserver 127.0.0.1:8080\r\n```\r\n\r\nThe configuration means that Tempesta bypasses cache on all requests. All requests are forwarded to the back-end server without exceptions.\r\n\r\nThe test is a simple `wrk -t 8 -c 1024 -d 600s http://192.168.0.1`. It runs on another physical machine with 8 CPU cores and 32Gb RAM. It is connected with the test server via 10G interface.\r\n\r\n\r\n```\r\n[  632.352381] kernel BUG at /root/ab/tempesta-ab/tempesta_fw/http.c:1321!\r\n[  632.359233] invalid opcode: 0000 [#1] SMP\r\n[  632.363473] Modules linked in: tfw_sched_rr(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) joydev hid_generic usbhid hid mlx4_en iTCO_wdt iTCO_vendor_support intel_rapl evdev ehci_pci sb_edac edac_core x86_pkg_temp_thermal ast intel_powerclamp igb coretemp ehci_hcd ttm snd_pcm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_timer aesni_intel drm_kms_helper ptp snd aes_x86_64 glue_helper lrw gf128mul ablk_helper xhci_pci xhci_hcd drm ahci cryptd pps_core mlx4_core soundcore libahci usbcore pcspkr i2c_algo_bit libata fb_sys_fops sg i2c_i801 syscopyarea sysfillrect i2c_smbus sysimgblt mei_me ioatdma i2c_core mei lpc_ich dca usb_common mfd_core wmi ipmi_si ipmi_msghandler acpi_power_meter acpi_pad tpm_tis tpm_tis_core tpm button autofs4 ext4 crc16 jbd2 mbcache sd_mod mpt3sas raid_class scsi_transport_sas scsi_mod shpchp\r\n[  632.444601] CPU: 7 PID: 52 Comm: ksoftirqd/7 Tainted: G           O    4.8.15-debug-ab+ #7\r\n[  632.453339] Hardware name: Supermicro Super Server/X10SRH-CF, BIOS 2.0 12/17/2015\r\n[  632.461315] task: ffff881017839780 task.stack: ffff881017848000\r\n[  632.467743] RIP: 0010:[<ffffffffa0a90b21>]  [<ffffffffa0a90b21>] tfw_http_conn_drop+0x151/0x220 [tempesta_fw]\r\n[  632.478231] RSP: 0018:ffff88101784b710  EFLAGS: 00010287\r\n[  632.484101] RAX: ffff880f2ec4b020 RBX: ffff880f3487a490 RCX: ffff880f2ec4b3e0\r\n[  632.491785] RDX: ffff880f2ea223e0 RSI: 0000000000000001 RDI: ffff880f3487a560\r\n[  632.499484] RBP: ffff88101784b760 R08: 0000000000000001 R09: 0000000000000000\r\n[  632.507192] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880f3487a550\r\n[  632.514910] R13: ffff88101784b710 R14: ffff880f3487a560 R15: ffff880f47830480\r\n[  632.522645] FS:  0000000000000000(0000) GS:ffff88103f3c0000(0000) knlGS:0000000000000000\r\n[  632.531355] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033\r\n[  632.537734] CR2: 00007fa37942f5d8 CR3: 000000100260f000 CR4: 00000000001406e0\r\n[  632.545521] Stack:\r\n[  632.548222]  ffff880f2ec4b020 ffff880f2ec4b020 0000000000000296 000000004c4adbe6\r\n[  632.556425]  ffff881017839780 ffff880f3487a490 ffff8810033b4010 ffff880f3487a4c8\r\n[  632.564617]  0000000000000000 ffff880f47830480 ffff88101784b780 ffffffffa0a8cf9a\r\n[  632.572805] Call Trace:\r\n[  632.575962]  [<ffffffffa0a8cf9a>] tfw_connection_drop+0x2a/0x40 [tempesta_fw]\r\n[  632.583827]  [<ffffffff815fa417>] ? _raw_spin_unlock+0x27/0x40\r\n[  632.590402]  [<ffffffffa0aab045>] tfw_sock_clnt_do_drop.isra.8+0x75/0xb0 [tempesta_fw]\r\n[  632.599085]  [<ffffffffa0aab08e>] tfw_sock_clnt_error+0xe/0x10 [tempesta_fw]\r\n[  632.606912]  [<ffffffffa0aa9713>] ss_linkerror+0x33/0x70 [tempesta_fw]\r\n[  632.614229]  [<ffffffffa0aaa0a8>] ss_tcp_state_change+0xa8/0x2d0 [tempesta_fw]\r\n[  632.622251]  [<ffffffff81531d86>] tcp_fin+0x146/0x3c0\r\n[  632.628126]  [<ffffffff8153278c>] tcp_data_queue+0x78c/0xf00\r\n[  632.634603]  [<ffffffff814ef2ba>] ? sk_filter_trim_cap+0x4a/0x390\r\n[  632.641508]  [<ffffffff81533132>] tcp_rcv_established+0x232/0x730\r\n[  632.648446]  [<ffffffff8153eb31>] tcp_v4_do_rcv+0x141/0x190\r\n[  632.654840]  [<ffffffff8153ffd4>] tcp_v4_rcv+0xa64/0xca0\r\n[  632.660997]  [<ffffffff8151617f>] ? ip_local_deliver_finish+0x2f/0x380\r\n[  632.668405]  [<ffffffff81516225>] ip_local_deliver_finish+0xd5/0x380\r\n[  632.675604]  [<ffffffff8151617f>] ? ip_local_deliver_finish+0x2f/0x380\r\n[  632.682994]  [<ffffffff8151671f>] ip_local_deliver+0x6f/0xf0\r\n[  632.689528]  [<ffffffff8153f4ea>] ? tcp_v4_early_demux+0xfa/0x180\r\n[  632.696537]  [<ffffffff81515cb7>] ip_rcv_finish+0x197/0x630\r\n[  632.703002]  [<ffffffff81516a2b>] ip_rcv+0x28b/0x3d0\r\n[  632.708905]  [<ffffffff81515b20>] ? inet_del_offload+0x40/0x40\r\n[  632.715634]  [<ffffffff814d2b1a>] __netif_receive_skb_core+0x31a/0x9c0\r\n[  632.723051]  [<ffffffff814d4800>] ? netif_receive_skb_internal+0xe0/0x200\r\n[  632.730723]  [<ffffffff814cc878>] ? get_rps_cpu+0x78/0x480\r\n[  632.737087]  [<ffffffff814d31d8>] __netif_receive_skb+0x18/0x60\r\n[  632.743851]  [<ffffffff814d4792>] netif_receive_skb_internal+0x72/0x200\r\n[  632.751286]  [<ffffffff814d4745>] ? netif_receive_skb_internal+0x25/0x200\r\n[  632.758875]  [<ffffffff814d5d87>] napi_gro_frags+0xf7/0x2c0\r\n[  632.765222]  [<ffffffffa04207c6>] mlx4_en_process_rx_cq+0x496/0xf20 [mlx4_en]\r\n[  632.773133]  [<ffffffffa04212e0>] mlx4_en_poll_rx_cq+0x30/0x90 [mlx4_en]\r\n[  632.780608]  [<ffffffff814d549f>] net_rx_action+0x22f/0x460\r\n[  632.786904]  [<ffffffff810e7924>] ? rcu_process_callbacks+0x6b4/0x800\r\n[  632.794056]  [<ffffffff815fd39d>] __do_softirq+0xcd/0x4ad\r\n[  632.800173]  [<ffffffff81096734>] ? smpboot_thread_fn+0x34/0x1e0\r\n[  632.806868]  [<ffffffff81096829>] ? smpboot_thread_fn+0x129/0x1e0\r\n[  632.813634]  [<ffffffff8107393f>] run_ksoftirqd+0x1f/0x60\r\n[  632.819747]  [<ffffffff81096824>] smpboot_thread_fn+0x124/0x1e0\r\n[  632.826331]  [<ffffffff81096700>] ? sort_range+0x30/0x30\r\n[  632.832329]  [<ffffffff81092c2e>] kthread+0xfe/0x120\r\n[  632.837946]  [<ffffffff815fad6f>] ret_from_fork+0x1f/0x40\r\n[  632.843993]  [<ffffffff81092b30>] ? kthread_create_on_node+0x240/0x240\r\n[  632.851164] Code: 08 eb d1 4c 89 f7 e8 ef 98 b6 e0 48 8b 45 b0 4c 39 e8 0f 84 05 ff ff ff 48 8b 90 c0 03 00 00 48 8d 88 c0 03 00 00 48 39 ca 74 14 <0f> 0b 4c 3b a7 c8 00 00 00 0f 85 62 ff ff ff e9 de fe ff ff 48 \r\n[  632.872455] RIP  [<ffffffffa0a90b21>] tfw_http_conn_drop+0x151/0x220 [tempesta_fw]\r\n[  632.880752]  RSP <ffff88101784b710>\r\n```"},"comment":{"url":"https://api.github.com/repos/tempesta-tech/tempesta/issues/comments/285496125","html_url":"https://github.com/tempesta-tech/tempesta/issues/696#issuecomment-285496125","issue_url":"https://api.github.com/repos/tempesta-tech/tempesta/issues/696","id":285496125,"user":{"login":"keshonok","id":6352580,"avatar_url":"https://avatars0.githubusercontent.com/u/6352580?v=3","gravatar_id":"","url":"https://api.github.com/users/keshonok","html_url":"https://github.com/keshonok","followers_url":"https://api.github.com/users/keshonok/followers","following_url":"https://api.github.com/users/keshonok/following{/other_user}","gists_url":"https://api.github.com/users/keshonok/gists{/gist_id}","starred_url":"https://api.github.com/users/keshonok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keshonok/subscriptions","organizations_url":"https://api.github.com/users/keshonok/orgs","repos_url":"https://api.github.com/users/keshonok/repos","events_url":"https://api.github.com/users/keshonok/events{/privacy}","received_events_url":"https://api.github.com/users/keshonok/received_events","type":"User","site_admin":false},"created_at":"2017-03-09T21:55:41Z","updated_at":"2017-03-09T21:55:41Z","body":"The bug is caused by a combination of two major factors:\r\n* `TfwCliConn{}`  has a union for `resp` and `reason`.\r\n*  When a client connection is dropped, `req->resp` is used to distinguish connections that are NOT in the forwarding queue.\r\n\r\nThe assumption was that `resp` and `reason` are never used both at the same time. That's true indeed. When a client connection is dropped some requests may be \"in flight\" - about to be dropped with an HTTP error response sent to the client. In that case `req->resp` is actually `req->reason`. \r\n\r\n`req->resp` is updated under `req->msg.seq_lock`, and the same lock is used in `tfw_http_conn_cli_drop()`. That is intended to ensure that when `req->resp` is set, then the request is definitely NOT in a forwarding queue.\r\n\r\nHowever, `tfw_http_conn_cli_drop()` has no ability to know that `req->resp` may actually be `req->reason`. Moreover, `req->reason` is usually updated under a completely different lock.\r\n\r\n`req->resp` and `req-reason` may not be shared. A proper fix is in the works."}}
{"actor":{"display_login":"keshonok"}}