| Summary: | SAN HBA: using sysfs to remove and add a HBA causes system to crash | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Bruno Goncalves <bgoncalv> |
| Component: | kernel | Assignee: | Ewan D. Milne <emilne> |
| kernel sub component: | Storage Drivers | QA Contact: | Bruno Goncalves <bgoncalv> |
| Status: | CLOSED WONTFIX | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | bgoncalv |
| Version: | 7.3 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-15 07:45:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
using kernel 3.10.0-495.el7.x86_64.debug
[ 88.337019] [ INFO: possible circular locking dependency detected ]
[ 88.365653] 3.10.0-495.el7.x86_64.debug #1 Not tainted
[ 88.388757] -------------------------------------------------------
[ 88.417012] bash/10652 is trying to acquire lock:
[ 88.438150] (pci_rescan_remove_lock){+.+.+.}, at: [<ffffffff813cc8e7>] pci_lock_rescan_remove+0x17/0x20
[ 88.480780]
[ 88.480780] but task is already holding lock:
[ 88.507582] (s_active#414){++++.+}, at: [<ffffffff812de4e3>] sysfs_write_file+0xc3/0x150
[ 88.547068]
[ 88.547068] which lock already depends on the new lock.
[ 88.547068]
[ 88.584221]
[ 88.584221] the existing dependency chain (in reverse order) is:
[ 88.617933]
-> #1 (s_active#414){++++.+}:
[ 88.636779] [<ffffffff81120e06>] __lock_acquire+0xd06/0x16c0
[ 88.664700] [<ffffffff811221c9>] lock_acquire+0x99/0x1e0
[ 88.691391] [<ffffffff812df1ae>] sysfs_deactivate+0x13e/0x1a0
[ 88.720053] [<ffffffff812dfe1b>] sysfs_addrm_finish+0x3b/0x70
[ 88.748640] [<ffffffff812ddf13>] sysfs_hash_and_remove+0x63/0xc0
[ 88.778645] [<ffffffff812e1ba1>] remove_files.isra.2+0x41/0x80
[ 88.807676] [<ffffffff812e1cf7>] sysfs_remove_group+0x47/0xd0
[ 88.839920] [<ffffffff812e1db3>] sysfs_remove_groups+0x33/0x50
[ 88.870392] [<ffffffff814a69ac>] device_remove_attrs+0x5c/0xa0
[ 88.899378] [<ffffffff814a765a>] device_del+0x13a/0x210
[ 88.926072] [<ffffffff813cfc40>] pci_remove_bus_device+0x70/0xd0
[ 88.955795] [<ffffffff813cfce2>] pci_stop_and_remove_bus_device_locked+0x22/0x30
[ 88.991812] [<ffffffff813d8cc5>] remove_callback+0x15/0x20
[ 89.019189] [<ffffffff812de114>] sysfs_schedule_callback_work+0x14/0x60
[ 89.051736] [<ffffffff810c0ab6>] process_one_work+0x226/0x720
[ 89.080249] [<ffffffff810c10d6>] worker_thread+0x126/0x3b0
[ 89.107858] [<ffffffff810c9fdd>] kthread+0xed/0x100
[ 89.132493] [<ffffffff8176a258>] ret_from_fork+0x58/0x90
[ 89.159163]
-> #0 (pci_rescan_remove_lock){+.+.+.}:
[ 89.181855] [<ffffffff8111f627>] check_prevs_add+0x987/0x990
[ 89.209819] [<ffffffff81120e06>] __lock_acquire+0xd06/0x16c0
[ 89.238095] [<ffffffff811221c9>] lock_acquire+0x99/0x1e0
[ 89.264661] [<ffffffff8175ba79>] mutex_lock_nested+0x99/0x530
[ 89.293245] [<ffffffff813cc8e7>] pci_lock_rescan_remove+0x17/0x20
[ 89.323934] [<ffffffff813d7aa5>] dev_rescan_store+0x65/0x90
[ 89.354257] [<ffffffff814a64d8>] dev_attr_store+0x18/0x30
[ 89.384165] [<ffffffff812de4fb>] sysfs_write_file+0xdb/0x150
[ 89.413020] [<ffffffff81251200>] vfs_write+0xc0/0x1f0
[ 89.439185] [<ffffffff81251d8a>] SyS_write+0x8a/0xf0
[ 89.465188] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 89.494857]
[ 89.494857] other info that might help us debug this:
[ 89.494857]
[ 89.530817] Possible unsafe locking scenario:
[ 89.530817]
[ 89.558731] CPU0 CPU1
[ 89.580949] ---- ----
[ 89.602187] lock(s_active#414);
[ 89.617180] lock(pci_rescan_remove_lock);
[ 89.647337] lock(s_active#414);
[ 89.673489] lock(pci_rescan_remove_lock);
[ 89.692273]
[ 89.692273] *** DEADLOCK ***
[ 89.692273]
[ 89.718530] 3 locks held by bash/10652:
[ 89.735695] #0: (sb_writers#3){.+.+.+}, at: [<ffffffff812512fb>] vfs_write+0x1bb/0x1f0
[ 89.772186] #1: (&buffer->mutex){+.+.+.}, at: [<ffffffff812de45c>] sysfs_write_file+0x3c/0x150
[ 89.811832] #2: (s_active#414){++++.+}, at: [<ffffffff812de4e3>] sysfs_write_file+0xc3/0x150
[ 89.853320]
[ 89.853320] stack backtrace:
[ 89.876340] CPU: 0 PID: 10652 Comm: bash Not tainted 3.10.0-495.el7.x86_64.debug #1
[ 89.910747] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 89.940448] ffffffff82c04ff0 000000003d7dc360 ffff880206343bf8 ffffffff817562ae
[ 89.973719] ffff880206343c38 ffffffff817501db ffff880206343c60 0000000000000002
[ 90.007090] ffff8802063ec000 ffff8802063ec9f8 ffff8802063ec9f8 0000000000000003
[ 90.041449] Call Trace:
[ 90.052488] [<ffffffff817562ae>] dump_stack+0x19/0x1b
[ 90.075834] [<ffffffff817501db>] print_circular_bug+0x1f9/0x207
[ 90.102952] [<ffffffff8111f627>] check_prevs_add+0x987/0x990
[ 90.128806] [<ffffffff81120e06>] __lock_acquire+0xd06/0x16c0
[ 90.154742] [<ffffffff811221c9>] lock_acquire+0x99/0x1e0
[ 90.178986] [<ffffffff813cc8e7>] ? pci_lock_rescan_remove+0x17/0x20
[ 90.207607] [<ffffffff8175ba79>] mutex_lock_nested+0x99/0x530
[ 90.233909] [<ffffffff813cc8e7>] ? pci_lock_rescan_remove+0x17/0x20
[ 90.262496] [<ffffffff813cc8e7>] ? pci_lock_rescan_remove+0x17/0x20
[ 90.291117] [<ffffffff813cc8e7>] pci_lock_rescan_remove+0x17/0x20
[ 90.318983] [<ffffffff813d7aa5>] dev_rescan_store+0x65/0x90
[ 90.344497] [<ffffffff814a64d8>] dev_attr_store+0x18/0x30
[ 90.372322] [<ffffffff812de4fb>] sysfs_write_file+0xdb/0x150
[ 90.400641] [<ffffffff81251200>] vfs_write+0xc0/0x1f0
[ 90.424046] [<ffffffff81273747>] ? fget_light+0x3a7/0x510
[ 90.449434] [<ffffffff81251d8a>] SyS_write+0x8a/0xf0
[ 90.472098] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
<snip>
[ 146.560012] general protection fault: 0000 [#1] SMP
[ 146.584420] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 146.985987] CPU: 2 PID: 11323 Comm: systemd-udevd Not tainted 3.10.0-495.el7.x86_64.debug #1
[ 147.024087] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 147.053829] task: ffff8800aa820000 ti: ffff88021f6d0000 task.ti: ffff88021f6d0000
[ 147.087476] RIP: 0010:[<ffffffff81228e2e>] [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 147.130727] RSP: 0018:ffff88021f6d3c20 EFLAGS: 00010246
[ 147.157297] RAX: 0000000000000000 RBX: 00000000000080d0 RCX: 0000000000000001
[ 147.193721] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff88021f6d3fd8
[ 147.226602] RBP: ffff88021f6d3c50 R08: 00000000001dac40 R09: ffff880187c03700
[ 147.258699] R10: 0000000000000001 R11: 0000000000000000 R12: 0200b81f02050000
[ 147.290809] R13: 00000000000080d0 R14: ffff880187c03700 R15: 00000000000000e0
[ 147.322871] FS: 00007f7ae3aa58c0(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 147.358849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 147.384801] CR2: 00007f7ae3ab7000 CR3: 000000022743d000 CR4: 00000000000407e0
[ 147.416844] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 147.449048] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 147.481236] Stack:
[ 147.490234] ffffffff812de9b8 ffff88022e4a4140 ffff880223644780 ffff8800b08228f0
[ 147.523595] ffffffff817e8660 ffff88022a6b8810 ffff88021f6d3c88 ffffffff812de9b8
[ 147.556838] ffff88022e4a4140 ffff8800b08228f0 ffff8800b7e86cc0 0000000000000000
[ 147.590069] Call Trace:
[ 147.601032] [<ffffffff812de9b8>] ? sysfs_open_file+0xb8/0x230
[ 147.627223] [<ffffffff812de9b8>] sysfs_open_file+0xb8/0x230
[ 147.652838] [<ffffffff8124eb27>] do_dentry_open+0x1b7/0x340
[ 147.682032] [<ffffffff8131141c>] ? security_inode_permission+0x1c/0x30
[ 147.713557] [<ffffffff812de900>] ? sysfs_schedule_callback+0x1e0/0x1e0
[ 147.743235] [<ffffffff8124ee02>] vfs_open+0x62/0xf0
[ 147.765539] [<ffffffff8125d1e8>] ? may_open+0x68/0x110
[ 147.788630] [<ffffffff8125fb12>] do_last+0x1f2/0x1290
[ 147.811716] [<ffffffff81262905>] path_openat+0xc5/0x4b0
[ 147.835546] [<ffffffff8126410b>] do_filp_open+0x4b/0xb0
[ 147.859405] [<ffffffff8175fd27>] ? _raw_spin_unlock+0x27/0x40
[ 147.885644] [<ffffffff812742ff>] ? __alloc_fd+0xaf/0x200
[ 147.909948] [<ffffffff81250363>] do_sys_open+0xf3/0x1f0
[ 147.933787] [<ffffffff8125047e>] SyS_open+0x1e/0x20
[ 147.956075] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 147.982768] Code: 8b 50 08 83 68 1c 01 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 e6 01 00 00 48 85 c0 0f 84 dd 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 84 02 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7
[ 148.068156] RIP [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 148.097873] RSP <ffff88021f6d3c20>
[ 148.113653] ---[ end trace 7160b024a0a6ed5b ]---
[ 148.138892] [Firmware Warn]: ERST: Firmware does not respond in time
[ 149.272021] general protection fault: 0000 [#2] SMP
[ 149.297338] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 149.693189] CPU: 2 PID: 1508 Comm: sendmail Tainted: G D ------------ 3.10.0-495.el7.x86_64.debug #1
[ 149.739689] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 149.769439] task: ffff880221a3c000 ti: ffff88022f3ec000 task.ti: ffff88022f3ec000
[ 149.806411] RIP: 0010:[<ffffffff81228e2e>] [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 149.848526] RSP: 0018:ffff88022f3efbb0 EFLAGS: 00010246
[ 149.872391] RAX: 0000000000000000 RBX: 00000000000000d0 RCX: 0000000000000001
[ 149.904459] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff88022f3effd8
[ 149.936551] RBP: ffff88022f3efbe0 R08: 00000000001dac40 R09: ffff880187c03700
[ 149.968730] R10: 0000000000000000 R11: 0000000000000000 R12: 0200b81f02050000
[ 150.000824] R13: 00000000000000d0 R14: ffff880187c03700 R15: 0000000000000100
[ 150.032916] FS: 00007ff8fc569840(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 150.069313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 150.095122] CR2: 00007f7ae3ab7000 CR3: 000000021772d000 CR4: 00000000000407e0
[ 150.127358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 150.159101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 150.192018] Stack:
[ 150.201679] ffffffff8127a246 0000000000000000 ffff88022e4a45c0 ffff88022d314e80
[ 150.237502] 0000000000000000 ffffffff8127a3d0 ffff88022f3efc08 ffffffff8127a246
[ 150.270922] ffff88022d314e80 ffff88022e4a45c0 ffffffff812d76f0 ffff88022f3efc40
[ 150.305532] Call Trace:
[ 150.317810] [<ffffffff8127a246>] ? seq_open+0x116/0x180
[ 150.344818] [<ffffffff8127a3d0>] ? single_open+0xb0/0xb0
[ 150.369724] [<ffffffff8127a246>] seq_open+0x116/0x180
[ 150.392920] [<ffffffff812d76f0>] ? loadavg_proc_open+0x20/0x20
[ 150.419526] [<ffffffff8127a381>] single_open+0x61/0xb0
[ 150.444142] [<ffffffff81228e81>] ? kmem_cache_alloc_trace+0x111/0x360
[ 150.474731] [<ffffffff812d76ea>] loadavg_proc_open+0x1a/0x20
[ 150.501091] [<ffffffff812cb87f>] proc_reg_open+0x6f/0x100
[ 150.527102] [<ffffffff8124eb27>] do_dentry_open+0x1b7/0x340
[ 150.552700] [<ffffffff8131141c>] ? security_inode_permission+0x1c/0x30
[ 150.583630] [<ffffffff812cb810>] ? proc_reg_release+0x70/0x70
[ 150.610386] [<ffffffff8124ee02>] vfs_open+0x62/0xf0
[ 150.632684] [<ffffffff8125d1e8>] ? may_open+0x68/0x110
[ 150.656134] [<ffffffff8125fb12>] do_last+0x1f2/0x1290
[ 150.678921] [<ffffffff81262905>] path_openat+0xc5/0x4b0
[ 150.703185] [<ffffffff8126410b>] do_filp_open+0x4b/0xb0
[ 150.727447] [<ffffffff8175fd27>] ? _raw_spin_unlock+0x27/0x40
[ 150.753748] [<ffffffff812742ff>] ? __alloc_fd+0xaf/0x200
[ 150.778005] [<ffffffff81250363>] do_sys_open+0xf3/0x1f0
[ 150.801862] [<ffffffff8125047e>] SyS_open+0x1e/0x20
[ 150.826304] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 150.857312] Code: 8b 50 08 83 68 1c 01 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 e6 01 00 00 48 85 c0 0f 84 dd 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 84 02 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7
[ 150.942632] RIP [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 150.972402] RSP <ffff88022f3efbb0>
[ 150.988127] ---[ end trace 7160b024a0a6ed5c ]---
[ 151.015799] general protection fault: 0000 [#3] SMP
[ 151.038194] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 151.441663] CPU: 2 PID: 1507 Comm: rs:main Q:Reg Tainted: G D ------------ 3.10.0-495.el7.x86_64.debug #1
[ 151.489829] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 151.519617] task: ffff880036264000 ti: ffff880228998000 task.ti: ffff880228998000
[ 151.553229] RIP: 0010:[<ffffffff81229bd7>] [<ffffffff81229bd7>] kmem_cache_alloc+0xb7/0x360
[ 151.591134] RSP: 0018:ffff88022899bb20 EFLAGS: 00010246
[ 151.615045] RAX: 0000000000000000 RBX: 00000000000082d0 RCX: 0000000000000001
[ 151.647240] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff88022899bfd8
[ 151.679433] RBP: ffff88022899bb50 R08: 00000000001dac40 R09: ffff880187c03700
[ 151.711504] R10: 0000000000000006 R11: 0000000000000000 R12: 0200b81f02050000
[ 151.743535] R13: 00000000000082d0 R14: ffff880187c03700 R15: ffffffffa0501b97
[ 151.777312] FS: 00007fa2b3f52700(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 151.814148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 151.841341] CR2: 00007f7ae3ab7000 CR3: 0000000224b13000 CR4: 00000000000407e0
[ 151.877197] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 151.909347] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 151.941366] Stack:
[ 151.950369] ffff880035ae6000 0000000000000000 00000000000082d0 ffff880187c03700
[ 151.983638] 0000000000000000 0000000051eb851f ffff88022899bb98 ffffffffa0501b97
[ 152.016948] ffff8800362646c0 ffff880036264000 ffff880035ae7000 0000000000000023
[ 152.050216] Call Trace:
[ 152.061182] [<ffffffffa0501b97>] kmem_zone_alloc+0x97/0x130 [xfs]
[ 152.088980] [<ffffffffa04ffe8a>] _xfs_trans_alloc+0x3a/0xa0 [xfs]
[ 152.116481] [<ffffffffa04fff24>] xfs_trans_alloc+0x34/0x40 [xfs]
[ 152.143871] [<ffffffffa04ed8d0>] xfs_vn_update_time+0x40/0x270 [xfs]
[ 152.172867] [<ffffffff8126fab5>] update_time+0x25/0xd0
[ 152.196348] [<ffffffff8126fd60>] file_update_time+0xa0/0xf0
[ 152.221790] [<ffffffffa04e1060>] xfs_file_aio_write_checks+0x190/0x200 [xfs]
[ 152.254477] [<ffffffffa04e1568>] xfs_file_buffered_aio_write+0x98/0x350 [xfs]
[ 152.289251] [<ffffffff81126293>] ? futex_wait+0x193/0x280
[ 152.314992] [<ffffffff812289fc>] ? kfree+0xdc/0x2d0
[ 152.337325] [<ffffffffa04e19ad>] xfs_file_aio_write+0x18d/0x1a0 [xfs]
[ 152.371053] [<ffffffff8125095d>] do_sync_write+0x8d/0xd0
[ 152.396454] [<ffffffff81251200>] vfs_write+0xc0/0x1f0
[ 152.419526] [<ffffffff81273499>] ? fget_light+0xf9/0x510
[ 152.443821] [<ffffffff81251d8a>] SyS_write+0x8a/0xf0
[ 152.466186] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 152.493267] Code: 8b 50 08 83 68 1c 01 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 ed 01 00 00 48 85 c0 0f 84 e4 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 8b 02 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7
[ 152.580798] RIP [<ffffffff81229bd7>] kmem_cache_alloc+0xb7/0x360
[ 152.608714] RSP <ffff88022899bb20>
[ 152.624361] ---[ end trace 7160b024a0a6ed5d ]---
[ 152.654492] general protection fault: 0000 [#4] SMP
[ 152.677246] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 153.091258] CPU: 2 PID: 13204 Comm: sosreport Tainted: G D ------------ 3.10.0-495.el7.x86_64.debug #1
[ 153.141585] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 153.173130] task: ffff88022ef20000 ti: ffff880223f10000 task.ti: ffff880223f10000
[ 153.208591] RIP: 0010:[<ffffffff81229bd7>] [<ffffffff81229bd7>] kmem_cache_alloc+0xb7/0x360
[ 153.248564] RSP: 0018:ffff880223f13b20 EFLAGS: 00010246
[ 153.273592] RAX: 0000000000000000 RBX: 00000000000082d0 RCX: 0000000000000001
[ 153.308117] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff880223f13fd8
[ 153.342470] RBP: ffff880223f13b50 R08: 00000000001dac40 R09: ffff880187c03700
[ 153.376484] R10: 0000000000000006 R11: 0000000000000000 R12: 0200b81f02050000
[ 153.412008] R13: 00000000000082d0 R14: ffff880187c03700 R15: ffffffffa0501b97
[ 153.447165] FS: 00007f7200dca740(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 153.485539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 153.512079] CR2: 00007f7200dd7000 CR3: 000000022a7d2000 CR4: 00000000000407e0
[ 153.545575] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 153.579320] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 153.613078] Stack:
[ 153.622510] ffff880035ae6000 0000000000000000 00000000000082d0 ffff880187c03700
[ 153.657431] 0000000000000000 0000000051eb851f ffff880223f13b98 ffffffffa0501b97
[ 153.692408] ffff88022ef206c0 ffff88022ef20000 ffff880035ae7000 0000000000000023
[ 153.727102] Call Trace:
[ 153.738573] [<ffffffffa0501b97>] kmem_zone_alloc+0x97/0x130 [xfs]
[ 153.767392] [<ffffffffa04ffe8a>] _xfs_trans_alloc+0x3a/0xa0 [xfs]
[ 153.796953] [<ffffffffa04fff24>] xfs_trans_alloc+0x34/0x40 [xfs]
[ 153.825940] [<ffffffffa04ed8d0>] xfs_vn_update_time+0x40/0x270 [xfs]
[ 153.856388] [<ffffffff8126fab5>] update_time+0x25/0xd0
[ 153.881080] [<ffffffff8126fd60>] file_update_time+0xa0/0xf0
[ 153.909068] [<ffffffffa04e1060>] xfs_file_aio_write_checks+0x190/0x200 [xfs]
[ 153.943603] [<ffffffffa04e1568>] xfs_file_buffered_aio_write+0x98/0x350 [xfs]
[ 153.979061] [<ffffffffa04e19ad>] xfs_file_aio_write+0x18d/0x1a0 [xfs]
[ 154.009840] [<ffffffff8125095d>] do_sync_write+0x8d/0xd0
[ 154.035432] [<ffffffff81251200>] vfs_write+0xc0/0x1f0
[ 154.059897] [<ffffffff8127362c>] ? fget_light+0x28c/0x510
[ 154.085948] [<ffffffff81251d8a>] SyS_write+0x8a/0xf0
[ 154.109950] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 154.138266] Code: 8b 50 08 83 68 1c 01 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 ed 01 00 00 48 85 c0 0f 84 e4 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 8b 02 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7
[ 154.228435] RIP [<ffffffff81229bd7>] kmem_cache_alloc+0xb7/0x360
[ 154.257513] RSP <ffff880223f13b20>
[ 154.274085] ---[ end trace 7160b024a0a6ed5e ]---
[ 154.304537] general protection fault: 0000 [#5] SMP
[ 154.328152] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 154.746971] CPU: 2 PID: 13243 Comm: rhsmcertd-worke Tainted: G D ------------ 3.10.0-495.el7.x86_64.debug #1
[ 154.798828] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 154.829979] task: ffff8801feff8000 ti: ffff880207a28000 task.ti: ffff880207a28000
[ 154.865400] RIP: 0010:[<ffffffff81228e2e>] [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 154.907618] RSP: 0018:ffff880207a2bb98 EFLAGS: 00010246
[ 154.933097] RAX: 0000000000000000 RBX: 0000000000000050 RCX: 0000000000000001
[ 154.968418] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff880207a2bfd8
[ 155.003522] RBP: ffff880207a2bbc8 R08: 00000000001dac40 R09: ffff880187c03700
[ 155.037052] R10: 0000000000000000 R11: 0000000000000000 R12: 0200b81f02050000
[ 155.070004] R13: 0000000000000050 R14: ffff880187c03700 R15: 0000000000000100
[ 155.102910] FS: 00007f3019c64740(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 155.141297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 155.168193] CR2: 0000000001228024 CR3: 0000000218fb6000 CR4: 00000000000407e0
[ 155.201711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 155.235375] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 155.269013] Stack:
[ 155.278465] ffffffff8131d7db ffff880226a4cd00 ffff880202273bb0 0000000000000000
[ 155.312931] ffff880226d97ac8 ffff880036070c00 ffff880207a2bc20 ffffffff8131d7db
[ 155.348069] ffff880202273bb0 ffff880226a4cd28 0000000004000000 000000009cc4a1fd
[ 155.384244] Call Trace:
[ 155.396084] [<ffffffff8131d7db>] ? inode_doinit_with_dentry+0x13b/0x670
[ 155.427780] [<ffffffff8131d7db>] inode_doinit_with_dentry+0x13b/0x670
[ 155.459335] [<ffffffff8131e8fc>] selinux_d_instantiate+0x1c/0x20
[ 155.488638] [<ffffffff81310ccb>] security_d_instantiate+0x1b/0x30
[ 155.518150] [<ffffffff8126a360>] d_instantiate+0x50/0x70
[ 155.543979] [<ffffffff8126b054>] d_splice_alias+0x44/0x120
[ 155.570911] [<ffffffffa04ecf72>] xfs_vn_lookup+0x92/0xd0 [xfs]
[ 155.598434] [<ffffffff8125c05d>] lookup_real+0x1d/0x50
[ 155.623321] [<ffffffff812604c3>] do_last+0xba3/0x1290
[ 155.647725] [<ffffffff81262905>] path_openat+0xc5/0x4b0
[ 155.672807] [<ffffffff813aa60b>] ? debug_check_no_obj_freed+0xfb/0x260
[ 155.704007] [<ffffffff8126410b>] do_filp_open+0x4b/0xb0
[ 155.730191] [<ffffffff8175fd27>] ? _raw_spin_unlock+0x27/0x40
[ 155.757383] [<ffffffff812742ff>] ? __alloc_fd+0xaf/0x200
[ 155.783329] [<ffffffff81250363>] do_sys_open+0xf3/0x1f0
[ 155.808407] [<ffffffff8125047e>] SyS_open+0x1e/0x20
[ 155.831839] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 155.860243] Code: 8b 50 08 83 68 1c 01 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 e6 01 00 00 48 85 c0 0f 84 dd 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 84 02 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7
[ 155.949553] RIP [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 155.981546] RSP <ffff880207a2bb98>
[ 155.998420] ---[ end trace 7160b024a0a6ed5f ]---
[ 156.161240] general protection fault: 0000 [#6] SMP
[ 156.184609] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 156.605517] CPU: 2 PID: 13248 Comm: rhsmcertd-worke Tainted: G D ------------ 3.10.0-495.el7.x86_64.debug #1
[ 156.658660] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 156.690178] task: ffff8800b7f08000 ti: ffff88022742c000 task.ti: ffff88022742c000
[ 156.725614] RIP: 0010:[<ffffffff81228e2e>] [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 156.768805] RSP: 0018:ffff88022742fbb0 EFLAGS: 00010246
[ 156.794291] RAX: 0000000000000000 RBX: 00000000000000d0 RCX: 0000000000000001
[ 156.828515] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff88022742ffd8
[ 156.862528] RBP: ffff88022742fbe0 R08: 00000000001dac40 R09: ffff880187c03700
[ 156.895996] R10: 0000000000000000 R11: 0000000000000000 R12: 0200b81f02050000
[ 156.929474] R13: 00000000000000d0 R14: ffff880187c03700 R15: 0000000000000100
[ 156.962929] FS: 00007f9822b86740(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 157.001634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 157.030397] CR2: 00007f98116b2627 CR3: 000000022742b000 CR4: 00000000000407e0
[ 157.065776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 157.100032] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 157.134278] Stack:
[ 157.143837] ffffffff8127a246 0000000000000000 ffff8800b053e300 ffff88022d314d40
[ 157.179081] 0000000000000000 ffffffff8127a3d0 ffff88022742fc08 ffffffff8127a246
[ 157.214162] ffff88022d314d40 ffff8800b053e300 ffffffff8162d360 ffff88022742fc40
[ 157.249428] Call Trace:
[ 157.261029] [<ffffffff8127a246>] ? seq_open+0x116/0x180
[ 157.285960] [<ffffffff8127a3d0>] ? single_open+0xb0/0xb0
[ 157.311159] [<ffffffff8127a246>] seq_open+0x116/0x180
[ 157.335438] [<ffffffff8162d360>] ? psched_open+0x20/0x20
[ 157.360790] [<ffffffff8127a381>] single_open+0x61/0xb0
[ 157.385455] [<ffffffff81228e81>] ? kmem_cache_alloc_trace+0x111/0x360
[ 157.416383] [<ffffffff8162d35a>] psched_open+0x1a/0x20
[ 157.441455] [<ffffffff812cb87f>] proc_reg_open+0x6f/0x100
[ 157.468409] [<ffffffff8124eb27>] do_dentry_open+0x1b7/0x340
[ 157.495017] [<ffffffff8131141c>] ? security_inode_permission+0x1c/0x30
[ 157.526479] [<ffffffff812cb810>] ? proc_reg_release+0x70/0x70
[ 157.555170] [<ffffffff8124ee02>] vfs_open+0x62/0xf0
[ 157.579565] [<ffffffff8125d1e8>] ? may_open+0x68/0x110
[ 157.605105] [<ffffffff8125fb12>] do_last+0x1f2/0x1290
[ 157.631037] [<ffffffff81262905>] path_openat+0xc5/0x4b0
[ 157.656706] [<ffffffff811f3b89>] ? do_read_fault.isra.38+0xe9/0x130
[ 157.686993] [<ffffffff8126410b>] do_filp_open+0x4b/0xb0
[ 157.712175] [<ffffffff812742ff>] ? __alloc_fd+0xaf/0x200
[ 157.737462] [<ffffffff81250363>] do_sys_open+0xf3/0x1f0
[ 157.762409] [<ffffffff8125047e>] SyS_open+0x1e/0x20
[ 157.785623] [<ffffffff8176a309>] system_call_fastpath+0x16/0x1b
[ 157.814145] Code: 8b 50 08 83 68 1c 01 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 e6 01 00 00 48 85 c0 0f 84 dd 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 84 02 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7
[ 157.900682] RIP [<ffffffff81228e2e>] kmem_cache_alloc_trace+0xbe/0x360
[ 157.931687] RSP <ffff88022742fbb0>
[ 157.948282] ---[ end trace 7160b024a0a6ed60 ]---
[ 194.061552] general protection fault: 0000 [#7] SMP
[ 194.087447] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf iTCO_wdt iTCO_vendor_support ipmi_ssif hpwdt sg sb_edac hpilo ioatdma lpc_ich edac_core pcspkr ipmi_si wmi ipmi_msghandler dca acpi_power_meter ntb shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata i2c_core qla2xxx tg3 ptp hpsa pps_core scsi_transport_sas scsi_transport_fc qlcnic scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod
[ 194.492883] CPU: 2 PID: 608 Comm: xfsaild/dm-2 Tainted: G D ------------ 3.10.0-495.el7.x86_64.debug #1
[ 194.540717] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[ 194.570547] task: ffff880035b5c000 ti: ffff880035b60000 task.ti: ffff880035b60000
[ 194.608157] RIP: 0010:[<ffffffff8122a2bb>] [<ffffffff8122a2bb>] __kmalloc+0xdb/0x3a0
[ 194.644830] RSP: 0018:ffff880035b63ba8 EFLAGS: 00010246
[ 194.668634] RAX: 0000000000000000 RBX: 0000000000000250 RCX: 0000000000000001
[ 194.700684] RDX: 0000000000008cc4 RSI: 0000000000000010 RDI: ffff880035b63fd8
[ 194.732735] RBP: ffff880035b63bd8 R08: 00000000001dac40 R09: ffff880187c03700
[ 194.765075] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000250
[ 194.797288] R13: 0000000000000100 R14: ffff880187c03700 R15: 0200b81f02050000
[ 194.829374] FS: 0000000000000000(0000) GS:ffff88022f800000(0000) knlGS:0000000000000000
[ 194.865734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 194.891588] CR2: 00007f98116b2627 CR3: 0000000001ab6000 CR4: 00000000000407e0
[ 194.923320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 194.955394] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 194.987484] Stack:
[ 194.996501] ffffffffa05018e7 0000000000000000 0000000000000250 0000000000000100
[ 195.029857] 0000000000000008 0000000051eb851f ffff880035b63c28 ffffffffa05018e7
[ 195.063113] ffffffffa04ca4f4 ffff880035b5c6c0 ffff880035b5c000 ffff8802252a7000
[ 195.097633] Call Trace:
[ 195.109902] [<ffffffffa05018e7>] ? kmem_alloc+0x97/0x130 [xfs]
[ 195.139363] [<ffffffffa05018e7>] kmem_alloc+0x97/0x130 [xfs]
[ 195.166121] [<ffffffffa04ca4f4>] ? xfs_perag_get+0xe4/0x290 [xfs]
[ 195.193900] [<ffffffffa04f09c8>] xfs_iflush_cluster+0x68/0x570 [xfs]
[ 195.222892] [<ffffffffa04f630b>] xfs_iflush+0x22b/0x330 [xfs]
[ 195.249512] [<ffffffffa050c672>] xfs_inode_item_push+0x152/0x1b0 [xfs]
[ 195.281638] [<ffffffffa05167f2>] xfsaild_push+0x1e2/0x970 [xfs]
[ 195.308519] [<ffffffffa0516f01>] ? xfsaild_push+0x8f1/0x970 [xfs]
[ 195.336345] [<ffffffffa0516f01>] ? xfsaild_push+0x8f1/0x970 [xfs]
[ 195.366222] [<ffffffffa0517054>] xfsaild+0xd4/0x190 [xfs]
[ 195.392018] [<ffffffffa0516f80>] ? xfsaild_push+0x970/0x970 [xfs]
[ 195.421437] [<ffffffff810c9fdd>] kthread+0xed/0x100
[ 195.444959] [<ffffffff810c9ef0>] ? insert_kthread_work+0x80/0x80
[ 195.474578] [<ffffffff8176a258>] ret_from_fork+0x58/0x90
[ 195.500203] [<ffffffff810c9ef0>] ? insert_kthread_work+0x80/0x80
[ 195.529212] Code: 8b 50 08 83 68 1c 01 4d 8b 38 49 8b 40 10 4d 85 ff 0f 84 d9 01 00 00 48 85 c0 0f 84 d0 01 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 07 0f 85 a4 02 00 00 48 8d 4a 01 4c 89 f8 65 49 0f c7
[ 195.619767] RIP [<ffffffff8122a2bb>] __kmalloc+0xdb/0x3a0
[ 195.646371] RSP <ffff880035b63ba8>
[ 195.663881] ---[ end trace 7160b024a0a6ed61 ]---
If on comment#0 I use sleep for 5 seconds instead of 1 second I could run the loop for over 30mins without any issue. Crash also happens with regular kernel [ 86.586116] general protection fault: 0000 [#1] SMP [ 86.610706] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd hpilo ipmi_devintf iTCO_wdt hpwdt iTCO_vendor_support acpi_power_meter ioatdma ipmi_ssif ntb pcspkr sb_edac edac_core ipmi_si lpc_ich shpchp dca ipmi_msghandler wmi sg pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata qla2xxx tg3 serio_raw qlcnic i2c_core hpsa ptp scsi_transport_fc pps_core scsi_transport_sas dm_multipath scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod [ 86.977842] CPU: 1 PID: 120 Comm: kworker/1:1 Not tainted 3.10.0-495.el7.x86_64 #1 [ 87.012985] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 87.043652] Workqueue: fc_wq_10 fc_rport_final_delete [scsi_transport_fc] [ 87.075085] task: ffff88023367de20 ti: ffff88023373c000 task.ti: ffff88023373c000 [ 87.113193] RIP: 0010:[<ffffffff811dcd43>] [<ffffffff811dcd43>] kmem_cache_alloc_node+0xd3/0x200 [ 87.157514] RSP: 0018:ffff88023373fc50 EFLAGS: 00010246 [ 87.183473] RAX: 0000000000000000 RBX: 000000000000005a RCX: 0000000000005ee9 [ 87.218040] RDX: 0000000000005ee8 RSI: 00000000000000d0 RDI: 0000000000019b00 [ 87.253094] RBP: ffff88023373fc90 R08: ffff880236459b00 R09: ffff880187c03700 [ 87.287053] R10: ffffffff8155c73d R11: 0000000000000000 R12: 0200b81f02050000 [ 87.321196] R13: 00000000000000d0 R14: 00000000ffffffff R15: ffff880187c03700 [ 87.356112] FS: 0000000000000000(0000) GS:ffff880236440000(0000) knlGS:0000000000000000 [ 87.394522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 87.421928] CR2: 00007fe06d6b08ca CR3: 00000000019b6000 CR4: 00000000000407e0 [ 87.456243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 87.490235] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 87.524069] Stack: [ 87.533561] 0000000000000800 ffff880218780010 ffffffff8155c73d 000000000000005a [ 87.568219] 00000000000000d0 00000000ffffffff ffff880187c03700 0000000000000108 [ 87.603293] ffff88023373fce0 ffffffff8155c73d 000000008476d120 0000000000000000 [ 87.638756] Call Trace: [ 87.650559] [<ffffffff8155c73d>] ? __alloc_skb+0x5d/0x2d0 [ 87.676975] [<ffffffff8155c73d>] __alloc_skb+0x5d/0x2d0 [ 87.702696] [<ffffffff8131b729>] kobject_uevent_env+0x3d9/0x620 [ 87.731708] [<ffffffff814384b3>] ? __pm_runtime_disable+0x23/0x150 [ 87.761465] [<ffffffff8131b97b>] kobject_uevent+0xb/0x10 [ 87.786637] [<ffffffff81427d1b>] device_del+0x18b/0x200 [ 87.811514] [<ffffffff81427dae>] device_unregister+0x1e/0x60 [ 87.838693] [<ffffffff813066be>] bsg_unregister_queue+0x5e/0xa0 [ 87.867552] [<ffffffffa0079f5e>] fc_rport_final_delete+0x11e/0x200 [scsi_transport_fc] [ 87.905742] [<ffffffff810a7e0b>] process_one_work+0x17b/0x470 [ 87.933459] [<ffffffff810a8c46>] worker_thread+0x126/0x410 [ 87.959890] [<ffffffff810a8b20>] ? rescuer_thread+0x460/0x460 [ 87.987764] [<ffffffff810b03ff>] kthread+0xcf/0xe0 [ 88.010712] [<ffffffff810b0330>] ? kthread_create_on_node+0x140/0x140 [ 88.041459] [<ffffffff81693798>] ret_from_fork+0x58/0x90 [ 88.066784] [<ffffffff810b0330>] ? kthread_create_on_node+0x140/0x140 [ 88.096968] Code: 48 8b 5d 08 66 66 66 66 90 48 83 c4 18 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 49 63 41 20 48 8d 4a 01 49 8b 39 <49> 8b 1c 04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 5a ff [ 88.187302] RIP [<ffffffff811dcd43>] kmem_cache_alloc_node+0xd3/0x200 [ 88.219174] RSP <ffff88023373fc50> This is not a regression as it is also reproducible on RHEL-7.2 kernel (-327) [ 166.598499] general protection fault: 0000 [#1] SMP [ 166.623886] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support lpc_ich hpwdt ipmi_ssif wmi hpilo sb_edac pcspkr sg edac_core ipmi_si acpi_power_meter mfd_core ntb ipmi_msghandler shpchp ioatdma pcc_cpufreq dca nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit sd_mod drm_kms_helper crc_t10dif crct10dif_generic ttm drm crct10dif_pclmul ata_piix crct10dif_common serio_raw qla2xxx libata qlcnic tg3 i2c_core ptp dm_multipath crc32c_intel scsi_transport_fc hpsa pps_core scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [ 166.950174] CPU: 2 PID: 1 Comm: systemd Not tainted 3.10.0-327.el7.x86_64 #1 [ 166.981857] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 167.011585] task: ffff8802340d8000 ti: ffff880234140000 task.ti: ffff880234140000 [ 167.045270] RIP: 0010:[<ffffffff811c0ea5>] [<ffffffff811c0ea5>] kmem_cache_alloc+0x75/0x1d0 [ 167.083216] RSP: 0018:ffff880234143d50 EFLAGS: 00010282 [ 167.107061] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000003555 [ 167.142160] RDX: 0000000000003554 RSI: 00000000000080d0 RDI: ffff880237003700 [ 167.176471] RBP: ffff880234143d80 R08: 00000000000175a0 R09: ffffffff811e067c [ 167.208678] R10: 8080808080808080 R11: 0000000000000000 R12: 0200b81f02050000 [ 167.241238] R13: 00000000000080d0 R14: ffff880237003700 R15: ffff880237003700 [ 167.273488] FS: 00007fe70a130940(0000) GS:ffff88023f440000(0000) knlGS:0000000000000000 [ 167.309852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 167.335731] CR2: 00007fae4915c010 CR3: 000000022d60c000 CR4: 00000000000407e0 [ 167.368548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 167.400242] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 167.432367] Stack: [ 167.441372] ffffffff811e067c 0000000000000000 ffff880233630f00 ffff8800b84c0000 [ 167.476574] ffff880234143f28 00000000ffffff9c ffff880234143da8 ffffffff811e067c [ 167.510491] ffff880234143da8 ffff880234143e50 ffff8800b84c0000 ffff880234143e40 [ 167.543828] Call Trace: [ 167.554859] [<ffffffff811e067c>] ? get_empty_filp+0x5c/0x1a0 [ 167.580702] [<ffffffff811e067c>] get_empty_filp+0x5c/0x1a0 [ 167.606102] [<ffffffff811ee5fd>] path_openat+0x4d/0x490 [ 167.630477] [<ffffffff81227b94>] ? ep_send_events_proc+0x174/0x1d0 [ 167.662885] [<ffffffff81229cdc>] ? signalfd_copyinfo+0x1c/0x250 [ 167.690848] [<ffffffff8122a0d4>] ? signalfd_read+0xd4/0x370 [ 167.716313] [<ffffffff811efe3b>] do_filp_open+0x4b/0xb0 [ 167.740194] [<ffffffff811c0e65>] ? kmem_cache_alloc+0x35/0x1d0 [ 167.767014] [<ffffffff811eec0f>] ? getname_flags+0x4f/0x1a0 [ 167.792583] [<ffffffff811fc9c7>] ? __alloc_fd+0xa7/0x130 [ 167.817297] [<ffffffff811dd7e3>] do_sys_open+0xf3/0x1f0 [ 167.841611] [<ffffffff811dd8fe>] SyS_open+0x1e/0x20 [ 167.863850] [<ffffffff81645909>] system_call_fastpath+0x16/0x1b [ 167.892146] Code: ce 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 1f 01 00 00 48 85 c0 0f 84 16 01 00 00 49 63 46 20 48 8d 4a 01 4d 8b 06 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 167.979255] RIP [<ffffffff811c0ea5>] kmem_cache_alloc+0x75/0x1d0 [ 168.006769] RSP <ffff880234143d50> I'm not sure about the lockdep warning from the debug kernel, it entirely possible that this condition cannot actually occur, and your system did not actually deadlock. The stack trace for the RHEL7.2 crash looks completely different, but the crashes in the kmalloc code seem to indicate possible kernel memory corruption. I suspect this may be due to a freed rport structure that is still being referenced, either on a timer list (we have another BZ showing this) or some other object holds a reference. Can you provide a crash dump? See also bug 1368174. There are some upstream fixes to the libfc code that fix the kref handling -- since the problem occurred on FCoE but could not be reproduced on lpfc I suspect the problem is in that area. kernel 495_test did not fix the problem. [ 90.134368] general protection fault: 0000 [#1] SMP [ 90.160040] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_service_time ipmi_ssif sb_edac edac_core iTCO_wdt sg hpilo hpwdt iTCO_vendor_support wmi ipmi_devintf acpi_power_meter pcspkr lpc_ich ntb ipmi_si ipmi_msghandler ioatdma shpchp dca pcc_cpufreq nfsd auth_rpcgss dm_multipath nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common qla2xxx tg3 serio_raw [ 90.482695] crc32c_intel i2c_core hpsa qlcnic scsi_transport_fc ptp scsi_transport_sas pps_core scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod [ 90.537239] CPU: 4 PID: 1431 Comm: kworker/4:2 Not tainted 3.10.0-495.el7.test.x86_64 #1 [ 90.573745] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 90.603448] Workqueue: fc_wq_5 fc_rport_final_delete [scsi_transport_fc] [ 90.633687] task: ffff8800360dde20 ti: ffff8800356d0000 task.ti: ffff8800356d0000 [ 90.668728] RIP: 0010:[<ffffffff811dcd43>] [<ffffffff811dcd43>] kmem_cache_alloc_node+0xd3/0x200 [ 90.712407] RSP: 0018:ffff8800356d3c50 EFLAGS: 00010246 [ 90.736246] RAX: 0000000000000000 RBX: 0000000000000057 RCX: 000000000000543d [ 90.768317] RDX: 000000000000543c RSI: 00000000000000d0 RDI: 0000000000019b00 [ 90.800426] RBP: ffff8800356d3c90 R08: ffff880236519b00 R09: ffff880187c03700 [ 90.832472] R10: ffffffff8155c73d R11: 0000000000000000 R12: 0200b81f02050000 [ 90.864606] R13: 00000000000000d0 R14: 00000000ffffffff R15: ffff880187c03700 [ 90.898710] FS: 0000000000000000(0000) GS:ffff880236500000(0000) knlGS:0000000000000000 [ 90.936265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 90.962116] CR2: 00007fa1aa5655e0 CR3: 00000000019b6000 CR4: 00000000000407e0 [ 90.994416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 91.026471] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 91.058522] Stack: [ 91.067528] 0000000000000800 ffff8802166a3410 ffffffff8155c73d 0000000000000057 [ 91.100795] 00000000000000d0 00000000ffffffff ffff880187c03700 0000000000000101 [ 91.136345] ffff8800356d3ce0 ffffffff8155c73d 000000008476d120 0000000000000000 [ 91.170953] Call Trace: [ 91.183122] [<ffffffff8155c73d>] ? __alloc_skb+0x5d/0x2d0 [ 91.210185] [<ffffffff8155c73d>] __alloc_skb+0x5d/0x2d0 [ 91.236261] [<ffffffff8131b729>] kobject_uevent_env+0x3d9/0x620 [ 91.265596] [<ffffffff814384b3>] ? __pm_runtime_disable+0x23/0x150 [ 91.295218] [<ffffffff8131b97b>] kobject_uevent+0xb/0x10 [ 91.320861] [<ffffffff81427d1b>] device_del+0x18b/0x200 [ 91.345970] [<ffffffff81427dae>] device_unregister+0x1e/0x60 [ 91.373230] [<ffffffff813066be>] bsg_unregister_queue+0x5e/0xa0 [ 91.401849] [<ffffffffa0067f5e>] fc_rport_final_delete+0x11e/0x200 [scsi_transport_fc] [ 91.440032] [<ffffffff810a7e0b>] process_one_work+0x17b/0x470 [ 91.467780] [<ffffffff810a8c46>] worker_thread+0x126/0x410 [ 91.494126] [<ffffffff810a8b20>] ? rescuer_thread+0x460/0x460 [ 91.521636] [<ffffffff810b03ff>] kthread+0xcf/0xe0 [ 91.544623] [<ffffffff810b0330>] ? kthread_create_on_node+0x140/0x140 [ 91.575658] [<ffffffff81693798>] ret_from_fork+0x58/0x90 [ 91.601167] [<ffffffff810b0330>] ? kthread_create_on_node+0x140/0x140 [ 91.632017] Code: 48 8b 5d 08 66 66 66 66 90 48 83 c4 18 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 49 63 41 20 48 8d 4a 01 49 8b 39 <49> 8b 1c 04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 5a ff [ 91.723172] RIP [<ffffffff811dcd43>] kmem_cache_alloc_node+0xd3/0x200 [ 91.755247] RSP <ffff8800356d3c50> (In reply to Bruno Goncalves from comment #10) > kernel 495_test did not fix the problem. > OK, thank you for testing, will continue to investigate. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |
Description of problem: When using sysfs to remove and add back and HBA port the server crashes. Version-Release number of selected component (if applicable): kernel 3.10.0-481.el7.x86_64.debug How reproducible: 100% Steps to Reproduce: 1.run a loop to remove and add the device back. (lspci -vt is useful to find out the PCI address of the device) # while true; do echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:04:00.6/remove; sleep 1; echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan; sleep 1; done Actual results: [ 116.387625] ====================================================== [ 116.419875] [ INFO: possible circular locking dependency detected ] [ 116.448670] 3.10.0-481.el7.x86_64.debug #1 Not tainted [ 116.471831] ------------------------------------------------------- [ 116.500816] bash/12948 is trying to acquire lock: [ 116.521935] (pci_rescan_remove_lock){+.+.+.}, at: [<ffffffff813c3d77>] pci_lock_rescan_remove+0x17/0x20 [ 116.564669] [ 116.564669] but task is already holding lock: [ 116.591051] (s_active#419){++++.+}, at: [<ffffffff812d7df3>] sysfs_write_file+0xc3/0x150 [ 116.627839] [ 116.627839] which lock already depends on the new lock. [ 116.627839] [ 116.664806] [ 116.664806] the existing dependency chain (in reverse order) is: [ 116.698616] -> #1 (s_active#419){++++.+}: [ 116.717512] [<ffffffff811203c6>] __lock_acquire+0xd06/0x16c0 [ 116.745820] [<ffffffff81121789>] lock_acquire+0x99/0x1e0 [ 116.772424] [<ffffffff812d8abe>] sysfs_deactivate+0x13e/0x1a0 [ 116.801005] [<ffffffff812d972b>] sysfs_addrm_finish+0x3b/0x70 [ 116.830186] [<ffffffff812d7823>] sysfs_hash_and_remove+0x63/0xc0 [ 116.861538] [<ffffffff812db4b1>] remove_files.isra.2+0x41/0x80 [ 116.891008] [<ffffffff812db607>] sysfs_remove_group+0x47/0xd0 [ 116.921973] [<ffffffff812db6c3>] sysfs_remove_groups+0x33/0x50 [ 116.955032] [<ffffffff8149dd8c>] device_remove_attrs+0x5c/0xa0 [ 116.984029] [<ffffffff8149ea3a>] device_del+0x13a/0x210 [ 117.010303] [<ffffffff813c70d0>] pci_remove_bus_device+0x70/0xd0 [ 117.040004] [<ffffffff813c7172>] pci_stop_and_remove_bus_device_locked+0x22/0x30 [ 117.076392] [<ffffffff813d0155>] remove_callback+0x15/0x20 [ 117.103801] [<ffffffff812d7a24>] sysfs_schedule_callback_work+0x14/0x60 [ 117.136258] [<ffffffff810c0116>] process_one_work+0x226/0x720 [ 117.164896] [<ffffffff810c0736>] worker_thread+0x126/0x3b0 [ 117.192397] [<ffffffff810c963d>] kthread+0xed/0x100 [ 117.217086] [<ffffffff8176bb18>] ret_from_fork+0x58/0x90 [ 117.243881] -> #0 (pci_rescan_remove_lock){+.+.+.}: [ 117.266601] [<ffffffff8111ebe7>] check_prevs_add+0x987/0x990 [ 117.294468] [<ffffffff811203c6>] __lock_acquire+0xd06/0x16c0 [ 117.322378] [<ffffffff81121789>] lock_acquire+0x99/0x1e0 [ 117.348996] [<ffffffff8175d339>] mutex_lock_nested+0x99/0x530 [ 117.378113] [<ffffffff813c3d77>] pci_lock_rescan_remove+0x17/0x20 [ 117.408245] [<ffffffff813cef35>] dev_rescan_store+0x65/0x90 [ 117.438771] [<ffffffff8149d8b8>] dev_attr_store+0x18/0x30 [ 117.468133] [<ffffffff812d7e0b>] sysfs_write_file+0xdb/0x150 [ 117.496610] [<ffffffff8124b190>] vfs_write+0xc0/0x1f0 [ 117.522699] [<ffffffff8124bd1a>] SyS_write+0x8a/0xf0 [ 117.547856] [<ffffffff8176bbc9>] system_call_fastpath+0x16/0x1b [ 117.577266] [ 117.577266] other info that might help us debug this: [ 117.577266] [ 117.613422] Possible unsafe locking scenario: [ 117.613422] [ 117.640111] CPU0 CPU1 [ 117.660410] ---- ---- [ 117.680797] lock(s_active#419); [ 117.695711] lock(pci_rescan_remove_lock); [ 117.725872] lock(s_active#419); [ 117.752193] lock(pci_rescan_remove_lock); [ 117.770979] [ 117.770979] *** DEADLOCK *** [ 117.770979] [ 117.797606] 3 locks held by bash/12948: [ 117.814798] #0: (sb_writers#3){.+.+.+}, at: [<ffffffff8124b28b>] vfs_write+0x1bb/0x1f0 [ 117.851390] #1: (&buffer->mutex){+.+.+.}, at: [<ffffffff812d7d6c>] sysfs_write_file+0x3c/0x150 [ 117.890675] #2: (s_active#419){++++.+}, at: [<ffffffff812d7df3>] sysfs_write_file+0xc3/0x150 [ 117.929706] [ 117.929706] stack backtrace: [ 117.951962] CPU: 1 PID: 12948 Comm: bash Not tainted 3.10.0-481.el7.x86_64.debug #1 [ 117.988531] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 118.018289] ffffffff82c0c1e0 0000000081657ae4 ffff880201fd7bf8 ffffffff81757b20 [ 118.051722] ffff880201fd7c38 ffffffff81751425 ffff880201fd7c60 0000000000000002 [ 118.085477] ffff8800aebf4000 ffff8800aebf49f8 ffff8800aebf49f8 0000000000000003 [ 118.118429] Call Trace: [ 118.129380] [<ffffffff81757b20>] dump_stack+0x19/0x1b [ 118.152511] [<ffffffff81751425>] print_circular_bug+0x1f9/0x207 [ 118.179487] [<ffffffff8111ebe7>] check_prevs_add+0x987/0x990 [ 118.205344] [<ffffffff8111f1cd>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [ 118.235207] [<ffffffff811203c6>] __lock_acquire+0xd06/0x16c0 [ 118.261148] [<ffffffff81121789>] lock_acquire+0x99/0x1e0 [ 118.285457] [<ffffffff813c3d77>] ? pci_lock_rescan_remove+0x17/0x20 [ 118.314138] [<ffffffff8175d339>] mutex_lock_nested+0x99/0x530 [ 118.340364] [<ffffffff813c3d77>] ? pci_lock_rescan_remove+0x17/0x20 [ 118.368992] [<ffffffff813c3d77>] ? pci_lock_rescan_remove+0x17/0x20 [ 118.397601] [<ffffffff813c3d77>] pci_lock_rescan_remove+0x17/0x20 [ 118.425379] [<ffffffff813cef35>] dev_rescan_store+0x65/0x90 [ 118.452719] [<ffffffff8149d8b8>] dev_attr_store+0x18/0x30 [ 118.481483] [<ffffffff812d7e0b>] sysfs_write_file+0xdb/0x150 [ 118.507423] [<ffffffff8124b190>] vfs_write+0xc0/0x1f0 [ 118.531356] [<ffffffff8126d627>] ? fget_light+0x3a7/0x510 [ 118.556129] [<ffffffff8124bd1a>] SyS_write+0x8a/0xf0 [ 118.578814] [<ffffffff8176bbc9>] system_call_fastpath+0x16/0x1b <snip> [ 2027.218801] general protection fault: 0000 [#1] SMP [ 2027.243359] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt ipmi_devintf iTCO_vendor_support ipmi_ssif hpwdt pcspkr sb_edac edac_core ioatdma hpilo sg lpc_ich ntb dca ipmi_si ipmi_msghandler wmi acpi_power_meter shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit dm_multipath drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm ata_piix serio_raw drm libata qla2xxx i2c_core tg3 hpsa ptp scsi_transport_sas scsi_transport_fc qlcnic pps_core scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod [ 2027.659607] CPU: 10 PID: 79 Comm: kworker/10:0 Not tainted 3.10.0-481.el7.x86_64.debug #1 [ 2027.697894] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 2027.729205] Workqueue: fc_wq_178 fc_rport_final_delete [scsi_transport_fc] [ 2027.762247] task: ffff88018430c000 ti: ffff880184318000 task.ti: ffff880184318000 [ 2027.798377] RIP: 0010:[<ffffffff81226103>] [<ffffffff81226103>] kmem_cache_alloc_node+0x1c3/0x380 [ 2027.842879] RSP: 0018:ffff88018431bb90 EFLAGS: 00010246 [ 2027.867870] RAX: 0000000000000000 RBX: 00000000000000d0 RCX: 0000000000000001 [ 2027.901571] RDX: 000000000000a882 RSI: 0000000000000010 RDI: ffff88018431bfd8 [ 2027.935525] RBP: ffff88018431bbd0 R08: 00000000001dac40 R09: ffff880187c03700 [ 2027.969370] R10: 0000000000000000 R11: 0000000000000000 R12: 0200b81f02050000 [ 2028.002897] R13: 00000000000000d0 R14: 00000000ffffffff R15: ffff880187c03700 [ 2028.036122] FS: 0000000000000000(0000) GS:ffff880230800000(0000) knlGS:0000000000000000 [ 2028.074496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2028.101660] CR2: 00007fb9691e9400 CR3: 0000000001aba000 CR4: 00000000000407e0 [ 2028.135239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2028.168863] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2028.202636] Stack: [ 2028.212131] 0000000000000246 0000000000000000 ffffffff815ebcfd 000000000000005d [ 2028.247292] 00000000000000d0 00000000ffffffff ffff880187c03700 0000000000000110 [ 2028.282477] ffff88018431bc20 ffffffff815ebcfd 00000000842483c0 ffff8801843a89c0 [ 2028.319043] Call Trace: [ 2028.331019] [<ffffffff815ebcfd>] ? __alloc_skb+0x5d/0x2d0 [ 2028.358088] [<ffffffff815ebcfd>] __alloc_skb+0x5d/0x2d0 [ 2028.383148] [<ffffffff813875d9>] kobject_uevent_env+0x3d9/0x620 [ 2028.411306] [<ffffffff8138782b>] kobject_uevent+0xb/0x10 [ 2028.436722] [<ffffffff8149ea98>] device_del+0x198/0x210 [ 2028.461313] [<ffffffff8149eb2e>] device_unregister+0x1e/0x60 [ 2028.487939] [<ffffffff81370ad0>] bsg_unregister_queue+0x60/0xa0 [ 2028.516010] [<ffffffffa00adff1>] fc_rport_final_delete+0x131/0x210 [scsi_transport_fc] [ 2028.553390] [<ffffffff810c0116>] process_one_work+0x226/0x720 [ 2028.580494] [<ffffffff810c00aa>] ? process_one_work+0x1ba/0x720 [ 2028.608741] [<ffffffff810c0736>] worker_thread+0x126/0x3b0 [ 2028.635145] [<ffffffff810c0610>] ? process_one_work+0x720/0x720 [ 2028.663484] [<ffffffff810c963d>] kthread+0xed/0x100 [ 2028.687003] [<ffffffff810c9550>] ? insert_kthread_work+0x80/0x80 [ 2028.715878] [<ffffffff8176bb18>] ret_from_fork+0x58/0x90 [ 2028.741716] [<ffffffff810c9550>] ? insert_kthread_work+0x80/0x80 [ 2028.770666] Code: 0f 85 c5 01 00 00 44 89 ee 4c 89 ff e8 57 10 01 00 49 89 c1 e9 c1 fe ff ff 0f 1f 80 00 00 00 00 49 63 41 20 4d 8b 01 41 f6 c0 0f <49> 8b 1c 04 0f 85 94 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7 [ 2028.860491] RIP [<ffffffff81226103>] kmem_cache_alloc_node+0x1c3/0x380 [ 2028.892024] RSP <ffff88018431bb90> [ 2028.913040] ---[ end trace 039c18a1a3b78227 ]--- [ 2028.947297] BUG: unable to handle kernel paging request at ffffffffffffff98 [ 2028.980145] IP: [<ffffffff810c9fc0>] kthread_data+0x10/0x20 [ 2029.006228] PGD 1abd067 PUD 1abf067 PMD 0 [ 2029.025664] Oops: 0000 [#2] SMP [ 2029.040890] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt ipmi_devintf iTCO_vendor_support ipmi_ssif hpwdt pcspkr sb_edac edac_core ioatdma hpilo sg lpc_ich ntb dca ipmi_si ipmi_msghandler wmi acpi_power_meter shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit dm_multipath drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm ata_piix serio_raw drm libata qla2xxx i2c_core tg3 hpsa ptp scsi_transport_sas scsi_transport_fc qlcnic pps_core scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod [ 2029.459484] CPU: 10 PID: 79 Comm: kworker/10:0 Tainted: G D ------------ 3.10.0-481.el7.x86_64.debug #1 [ 2029.509674] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 2029.540878] task: ffff88018430c000 ti: ffff880184318000 task.ti: ffff880184318000 [ 2029.576068] RIP: 0010:[<ffffffff810c9fc0>] [<ffffffff810c9fc0>] kthread_data+0x10/0x20 [ 2029.614140] RSP: 0018:ffff88018431b930 EFLAGS: 00010002 [ 2029.639313] RAX: 0000000000000000 RBX: 000000000000000a RCX: ffffffff8248d840 [ 2029.673230] RDX: 000000000000b854 RSI: 000000000000000a RDI: ffff88018430c000 [ 2029.706957] RBP: ffff88018431b930 R08: 000000000000bc00 R09: 0000000000000001 [ 2029.740505] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8802309d7bc0 [ 2029.774273] R13: 000000000000000a R14: ffff88018431b538 R15: ffff880187c03700 [ 2029.807612] FS: 0000000000000000(0000) GS:ffff880230800000(0000) knlGS:0000000000000000 [ 2029.845517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2029.872842] CR2: 0000000000000028 CR3: 0000000001aba000 CR4: 00000000000407e0 [ 2029.908095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2029.942844] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2029.975965] Stack: [ 2029.985553] ffff88018431b948 ffffffff810c0dd1 ffff88018430c000 ffff88018431b9b0 [ 2030.021383] ffffffff8175ee79 ffff88018431bfd8 ffff88018431bfd8 ffff88018431bfd8 [ 2030.057115] ffff88018431bfd8 ffffffff810d1bf2 ffff88018430c000 ffff88018430c848 [ 2030.092115] Call Trace: [ 2030.103558] [<ffffffff810c0dd1>] wq_worker_sleeping+0x11/0x90 [ 2030.132100] [<ffffffff8175ee79>] __schedule+0x8d9/0xa10 [ 2030.157315] [<ffffffff810d1bf2>] ? creds_are_invalid.part.1+0x12/0x50 [ 2030.189157] [<ffffffff8175efd9>] schedule+0x29/0x70 [ 2030.212745] [<ffffffff8109c528>] do_exit+0x7c8/0xc50 [ 2030.236720] [<ffffffff817635d7>] oops_end+0xa7/0x150 [ 2030.260685] [<ffffffff8103727b>] die+0x4b/0x70 [ 2030.282628] [<ffffffff81762f0e>] do_general_protection+0x13e/0x1d0 [ 2030.312524] [<ffffffff81762708>] general_protection+0x28/0x30 [ 2030.340353] [<ffffffff81226103>] ? kmem_cache_alloc_node+0x1c3/0x380 [ 2030.371047] [<ffffffff815ebcfd>] ? __alloc_skb+0x5d/0x2d0 [ 2030.398102] [<ffffffff815ebcfd>] __alloc_skb+0x5d/0x2d0 [ 2030.424144] [<ffffffff813875d9>] kobject_uevent_env+0x3d9/0x620 [ 2030.453554] [<ffffffff8138782b>] kobject_uevent+0xb/0x10 [ 2030.479460] [<ffffffff8149ea98>] device_del+0x198/0x210 [ 2030.504201] [<ffffffff8149eb2e>] device_unregister+0x1e/0x60 [ 2030.531508] [<ffffffff81370ad0>] bsg_unregister_queue+0x60/0xa0 [ 2030.560099] [<ffffffffa00adff1>] fc_rport_final_delete+0x131/0x210 [scsi_transport_fc] [ 2030.597808] [<ffffffff810c0116>] process_one_work+0x226/0x720 [ 2030.625380] [<ffffffff810c00aa>] ? process_one_work+0x1ba/0x720 [ 2030.653763] [<ffffffff810c0736>] worker_thread+0x126/0x3b0 [ 2030.680115] [<ffffffff810c0610>] ? process_one_work+0x720/0x720 [ 2030.708711] [<ffffffff810c963d>] kthread+0xed/0x100 [ 2030.732266] [<ffffffff810c9550>] ? insert_kthread_work+0x80/0x80 [ 2030.760956] [<ffffffff8176bb18>] ret_from_fork+0x58/0x90 [ 2030.786362] [<ffffffff810c9550>] ? insert_kthread_work+0x80/0x80 [ 2030.815275] Code: 00 48 89 e5 5d 48 8b 40 88 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 87 78 05 00 00 48 89 e5 <48> 8b 40 98 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 [ 2030.905140] RIP [<ffffffff810c9fc0>] kthread_data+0x10/0x20 [ 2030.933043] RSP <ffff88018431b930> [ 2030.949901] CR2: ffffffffffffff98 [ 2030.966368] ---[ end trace 039c18a1a3b78228 ]--- [ 2031.004152] Fixing recursive fault but reboot is needed! [ 2046.218928] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0 [ 2046.249344] Modules linked in: ip6table_filter ip6_tables iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt ipmi_devintf iTCO_vendor_support ipmi_ssif hpwdt pcspkr sb_edac edac_core ioatdma hpilo sg lpc_ich ntb dca ipmi_si ipmi_msghandler wmi acpi_power_meter shpchp pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c dm_service_time ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit dm_multipath drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm ata_piix serio_raw drm libata qla2xxx i2c_core tg3 hpsa ptp scsi_transport_sas scsi_transport_fc qlcnic pps_core scsi_tgt fjes dm_mirror dm_region_hash dm_log dm_mod [ 2046.647507] irq event stamp: 1609904 [ 2046.663573] hardirqs last enabled at (1609903): [<ffffffff81118ce7>] tick_nohz_idle_enter+0x47/0x80 [ 2046.704830] hardirqs last disabled at (1609904): [<ffffffff8110bc02>] cpu_startup_entry+0xf2/0x490 [ 2046.747042] softirqs last enabled at (1609880): [<ffffffff810a0545>] irq_enter+0xa5/0xb0 [ 2046.788335] softirqs last disabled at (1609879): [<ffffffff8176e3c5>] smp_apic_timer_interrupt+0x35/0x60 [ 2046.830993] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D ------------ 3.10.0-481.el7.x86_64.debug #1 [ 2046.877353] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 2046.907009] task: ffffffff81ac14c0 ti: ffffffff81aac000 task.ti: ffffffff81aac000 [ 2046.940659] RIP: 0010:[<ffffffff8159dfd4>] [<ffffffff8159dfd4>] cpuidle_enter_state+0x54/0xd0 [ 2046.979371] RSP: 0018:ffffffff81aafe48 EFLAGS: 00000246 [ 2047.003522] RAX: 000001d857b14da1 RBX: ffffffff81762473 RCX: 0000000000000017 [ 2047.035762] RDX: 0000000225c17d03 RSI: ffffffff81aaffd8 RDI: ffffffff8159dfd0 [ 2047.067839] RBP: ffffffff81aafe70 R08: 00000000000d7c4f R09: 0000000000000018 [ 2047.099950] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff811717d7 [ 2047.132033] R13: ffffffff81aafdb8 R14: ffffffff81ac14c0 R15: ffffffff81aac000 [ 2047.164164] FS: 0000000000000000(0000) GS:ffff88022f400000(0000) knlGS:0000000000000000 [ 2047.200591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2047.226564] CR2: 00007f8137d4c000 CR3: 0000000001aba000 CR4: 00000000000407f0 [ 2047.261352] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2047.297016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2047.329132] Stack: [ 2047.338124] ffffe8fff7e01000 0000000000000005 0000000000000005 ffffffff81bb22c0 [ 2047.371805] ffffffff81aaffa8 ffffffff81aafeb0 ffffffff8159e12b 0000000000000000 [ 2047.405354] 00000000ffffffff ffffffff81c2e340 ffffffff81aac000 ffffffff81aac000 [ 2047.438690] Call Trace: [ 2047.449696] [<ffffffff8159e12b>] cpuidle_idle_call+0xdb/0x3c0 [ 2047.475898] [<ffffffff8103e0de>] arch_cpu_idle+0xe/0x40 [ 2047.499779] [<ffffffff8110bdc5>] cpu_startup_entry+0x2b5/0x490 [ 2047.526367] [<ffffffff8174442b>] rest_init+0x14b/0x150 [ 2047.549848] [<ffffffff817442e5>] ? rest_init+0x5/0x150 [ 2047.573370] [<ffffffff81e18091>] start_kernel+0x463/0x484 [ 2047.598638] [<ffffffff81e17a37>] ? repair_env_string+0x5c/0x5c [ 2047.627008] [<ffffffff81e17120>] ? early_idt_handlers+0x120/0x120 [ 2047.655380] [<ffffffff81e175ee>] x86_64_start_reservations+0x2a/0x2c [ 2047.684761] [<ffffffff81e17742>] x86_64_start_kernel+0x152/0x175 [ 2047.712135] Code: 89 fb e8 00 0e b7 ff 44 89 ea 4c 89 e6 48 89 df 49 89 c6 41 ff 57 48 41 89 c4 e8 e8 0d b7 ff 49 89 c5 e8 c0 12 b8 ff fb 66 66 90 <66> 66 90 4c 89 ef 4c 29 f7 e8 7e ff af ff 48 69 c0 40 42 0f 00 [ 2047.803265] Kernel panic - not syncing: Hard LOCKUP