{"push_id":2207203595,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"ff9e69ec3cac123c81aedb9f1b2c8cc1354bdf18","before":"bbffb59efc49baba1d131d497202e8cda0068c3d","commits":[{"sha":"ff9e69ec3cac123c81aedb9f1b2c8cc1354bdf18","author":{"name":"Li Dongyang","email":"812ffd889148a220e8ee7a84e222663719d68d7a@anu.edu.au"},"message":"Call commit callbacks from the tail of the list\n\nOur zfs backed Lustre MDT had soft lockups while under heavy metadata\nworkload:\n[ 3597.867291] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [tx_commit_cb:67888]\n[ 3597.867329] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dm_mirror dm_region_hash dm_log sb_edac zfs(POE) edac_core intel_powerclamp coretemp zunicode(POE) zavl(POE) intel_rapl icp(POE) iosf_mbi kvm_intel zcommon(POE) znvpair(POE) spl(OE) kvm iTCO_wdt iTCO_vendor_support irqbypass dm_round_robin crc32_pclmul ghash_clmulni_intel sg ipmi_si hpilo hpwdt aesni_intel ipmi_devintf lrw gf128mul glue_helper ipmi_msghandler ioatdma ablk_helper pcspkr i2c_i801 wmi cryptd nfsd dm_multipath lpc_ich shpchp dca acpi_cpufreq acpi_power_meter dm_mod\n[ 3597.867344]  auth_rpcgss nfs_acl lockd grace sunrpc knem(OE) ip_tables xfs libcrc32c mlx5_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 mlx5_core(OE) devlink mlx_compat(OE) crct10dif_pclmul ptp crct10dif_common serio_raw crc32c_intel i2c_core pps_core hpsa(OE) scsi_transport_sas\n[ 3597.867346] CPU: 20 PID: 67888 Comm: tx_commit_cb Tainted: P           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1\n[ 3597.867348] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/25/2017\n[ 3597.867349] task: ffff889c3d6f1fa0 ti: ffff8898b7dec000 task.ti: ffff8898b7dec000\n[ 3597.867357] RIP: 0010:[<ffffffff810fa326>]  [<ffffffff810fa326>] native_queued_spin_lock_slowpath+0x116/0x1e0\n[ 3597.867358] RSP: 0018:ffff8898b7defcd0  EFLAGS: 00000246\n[ 3597.867358] RAX: 0000000000000000 RBX: 000000010001a588 RCX: 0000000000a10000\n[ 3597.867359] RDX: ffff88befed97880 RSI: 0000000000c10001 RDI: ffff885e58a78138\n[ 3597.867359] RBP: ffff8898b7defcd0 R08: ffff88befec97880 R09: 0000000000000000\n[ 3597.867360] R10: 00000000ea2b3a01 R11: ffffea017ba8acc0 R12: ffff885efbf32f40\n[ 3597.867361] R13: ffff88bebc38a48d R14: 0000000000016d39 R15: ffffffff811de591\n[ 3597.867361] FS:  0000000000000000(0000) GS:ffff88befec80000(0000) knlGS:0000000000000000\n[ 3597.867362] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033\n[ 3597.867362] CR2: 00007fda2a9a2090 CR3: 0000005eef36d000 CR4: 00000000003407e0\n[ 3597.867363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000\n[ 3597.867363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400\n[ 3597.867364] Stack:\n[ 3597.867365]  ffff8898b7defce0 ffffffff8169e61f ffff8898b7defcf0 ffffffff816abb70\n[ 3597.867366]  ffff8898b7defd68 ffffffffc136314e ffff885e58a78138 ffff885c6376a650\n[ 3597.867367]  ffff8898b7defd10 ffff8898b7defd10 0000000000000000 0000000000000000\n[ 3597.867367] Call Trace:\n[ 3597.867375]  [<ffffffff8169e61f>] queued_spin_lock_slowpath+0xb/0xf\n[ 3597.867378]  [<ffffffff816abb70>] _raw_spin_lock+0x20/0x30\n[ 3597.867445]  [<ffffffffc136314e>] ptlrpc_commit_replies+0x7e/0x380 [ptlrpc]\n[ 3597.867481]  [<ffffffffc13af3a2>] tgt_cb_last_committed+0x2c2/0x3d0 [ptlrpc]\n[ 3597.867489]  [<ffffffffc114773b>] osd_trans_commit_cb+0x14b/0x490 [osd_zfs]\n[ 3597.867529]  [<ffffffffc0b91ba4>] dmu_tx_do_callbacks+0x44/0x70 [zfs]\n[ 3597.867554]  [<ffffffffc0bdd224>] txg_do_callbacks+0x14/0x30 [zfs]\n[ 3597.867561]  [<ffffffffc02c3ed6>] taskq_thread+0x246/0x470 [spl]\n[ 3597.867564]  [<ffffffff810c4810>] ? wake_up_state+0x20/0x20\n[ 3597.867568]  [<ffffffffc02c3c90>] ? taskq_thread_spawn+0x60/0x60 [spl]\n[ 3597.867571]  [<ffffffff810b098f>] kthread+0xcf/0xe0\n[ 3597.867573]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40\n[ 3597.867576]  [<ffffffff816b4f58>] ret_from_fork+0x58/0x90\n[ 3597.867577]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40\n[ 3597.867588] Code: 0d 48 98 83 e2 30 48 81 c2 80 78 01 00 48 03 14 c5 e0 fd b0 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2\n\nThis patch makes zfs commit callbacks work on highest transaction\nnumber first, saving the subsequent calls to ptlrpc_commit_replies\nin Lustre, which makes the problem go away.\n\nA similar issue for ext4/ldiskfs is described on:\nhttps://jira.hpdd.intel.com/browse/LU-6527\n\nSigned-off-by: Li Dongyang <dongyang.li@anu.edu.au>","distinct":true,"url":"https://api.github.com/repos/lidongyang/zfs/commits/ff9e69ec3cac123c81aedb9f1b2c8cc1354bdf18"}]}
{"actor":{"display_login":"lidongyang"}}