Bug 614206
| Summary: | WARNING: at fs/buffer.c:1159 mark_buffer_dirty+0x82/0xa0() after device removal | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Babu Moger <babu.moger> | ||||||
| Component: | kernel | Assignee: | Red Hat Kernel Manager <kernel-mgr> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Red Hat Kernel QE team <kernel-qe> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | low | ||||||||
| Version: | 6.0 | CC: | coughlan, dl-iop-bugzilla, esandeen, rwheeler, samuel.miller, Sean.Stewart | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2010-07-15 16:54:33 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Babu Moger
2010-07-13 20:58:10 UTC
Created attachment 431593 [details]
Messages file showing the OOps(time stamp Jul 13 17:18:12)
Created attachment 431594 [details]
Kernel abrt files from /var/spool/abrt/
So, this was down an ext3 error (__ext3_std_error) path, coming from ext3_reserve_inode_write().
We got these errors just prior:
Jul 13 17:18:12 kswc-kenpachi kernel: sd 5:0:0:5: [sdg] Unhandled error code
Jul 13 17:18:12 kswc-kenpachi kernel: sd 5:0:0:5: [sdg] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 13 17:18:12 kswc-kenpachi kernel: sd 5:0:0:5: [sdg] CDB: Write(10): 2a 00 00 0c e4 e8 00 00 80 00
Jul 13 17:18:12 kswc-kenpachi kernel: end_request: I/O error, dev sdg, sector 845032
...
Jul 13 17:18:12 kswc-kenpachi kernel: end_request: I/O error, dev sdg, sector 1067160
...
...
Jul 13 17:18:12 kswc-kenpachi kernel: Aborting journal on device sdg.
Jul 13 17:18:12 kswc-kenpachi kernel: EXT3-fs error (device sdg) in ext3_reserve_inode_write: Journal has aborted
due to the:
> 5. When i/o is running powercycle the switch causing devices to go away.
The warning was in mark_buffer_dirty, warning that the buffer was not marked uptodate: WARN_ON_ONCE(!buffer_uptodate(bh));
going down the ext3_commit_super() path from ext3_handle_error.
So in the end we have a not-uptodate superblock buffer which gets marked dirty on the way to commit, and mark_buffer_dirty() warns us that it is not uptodate.
It's likely not uptodate because of the IO errors ... note that this warning isn't an oops or a BUG(), and should not lead directly to any system instability.
Just because abrt called this an "oops" doesn't make it so, it tends to call any backtrace an "oops" sadly; the noisiness in the logs is not a huge problem per se.
You mentioned that the system becomes unreliable after this happens. In what way does it become unreliable? That seems to be the actual bug here, but it's not well explained.
After this problem system locks up with NMI. This happens when the devices come back when the FC switch is back online(from powercycle). NMI lockup happens when the system starts scanning the scsi devices. Here is the console output.. EXT3-fs error (device sdg) in ext3_reserve_inode_write: Journal has aborted EXT3-fs error (device sdg) in ext3_orphan_del: Journal has aborted EXT3-fs error (device sdg) in ext3_truncate: Journal has aborted journal commit I/O error ext3_abort called. EXT3-fs error (device sdg): ext3_journal_start_sb: Remounting filesystem read-only EXT3-fs error (device sdg): ext3_get_inode_loc: unable to read inode block - inode=24622, block=98373 EXT3-fs error (device sdg): ext3_get_inode_loc: unable to read inode block - inode=24622, block=98373 EXT3-fs error (device sdg): ext3_get_inode_loc: unable to read inode block - inode=24622, block=98373 BUG: NMI Watchdog detected LOCKUP on CPU1, ip ffffffff814da62f, registers: CPU 1 Modules linked in: ext3 jbd fuse nfs lockd fscache nfs_acl auth_rpcgss autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables be2iscsi bnx2i cnic uio cxgb3i iw_cxgb3 cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_mirror dm_region_hash dm_log uinput dcdbas serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core bnx2 ext4 mbcache jbd2 mppVhba(U) sr_mod cdrom ata_generic pata_acpi ata_piix qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas mppUpper(U) sg sd_mod crc_t10dif dm_mod [last unloaded: microcode] Modules linked in: ext3 jbd fuse nfs lockd fscache nfs_acl auth_rpcgss autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables be2iscsi bnx2i cnic uio cxgb3i iw_cxgb3 cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_mirror dm_region_hash dm_log uinput dcdbas serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core bnx2 ext4 mbcache jbd2 mppVhba(U) sr_mod cdrom ata_generic pata_acpi ata_piix qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas mppUpper(U) sg sd_mod crc_t10dif dm_mod [last unloaded: microcode] Pid: 384, comm: mpp_dcr/1 Tainted: G W 2.6.32-44.el6.x86_64 #1 PowerEdge R710 RIP: 0010:[<ffffffff814da62f>] [<ffffffff814da62f>] _spin_lock_irqsave+0x2f/0x40 RSP: 0018:ffff88041712fce0 EFLAGS: 00000097 RAX: 0000000000000a5d RBX: ffff8803e90b0000 RCX: 0000000000000a5c RDX: 0000000000000286 RSI: 000000000000001e RDI: ffff880417764f00 RBP: ffff88041712fce0 R08: ffff8803f8c20358 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8803e90b0000 R13: 000000000000001e R14: ffff88041712fd8f R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff880036820000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000000022e22f8 CR3: 00000003f68c9000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process mpp_dcr/1 (pid: 384, threadinfo ffff88041712e000, task ffff880417234a60) Stack: ffff88041712fd30 ffffffffa004762f ffff88041712fd10 ffff88041712fd8e <0> ffff88041712fd60 ffff8803f8c20358 ffff8803e90b0000 ffff88041712fde8 <0> ffff88041712fdf0 ffff8803e057b9c0 ffff88041712fdc0 ffffffffa01443d4 Call Trace: [<ffffffffa004762f>] mppCmn_NewSelectPath+0x3f/0xc0 [mppUpper] [<ffffffffa01443d4>] mppLnx_do_queuecommand+0xa4/0xb60 [mppVhba] [<ffffffffa0138438>] mppLnx_process_ios+0x118/0x170 [mppVhba] [<ffffffffa0138320>] ? mppLnx_process_ios+0x0/0x170 [mppVhba] [<ffffffff8108b6c0>] worker_thread+0x170/0x2a0 [<ffffffff81090d50>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8108b550>] ? worker_thread+0x0/0x2a0 [<ffffffff810909e6>] kthread+0x96/0xa0 [<ffffffff810141ca>] child_rip+0xa/0x20 [<ffffffff81090950>] ? kthread+0x0/0xa0 [<ffffffff810141c0>] ? child_rip+0x0/0x20 This lockup is in the mppUpper (RDAC?) module, which is not something we can debug or support, I think. Perhaps we have an issue with the WARNING() and the superblock buffer, I need to think about that little more, but this looks like a different bug, in a 3rd party kernel module. -Eric It would be interesting to try this with RHEL native multipathing - dm multipath. Ok.. We will try with native multipathing and let you know the results.. Great - let us know if you have questions about multipathing... This issue has been proposed when we are only considering blocker issues in the current Red Hat Enterprise Linux release. It has been denied for the current Red Hat Enterprise Linux release. ** If you would still like this issue considered for the current release, ask your support representative to file as a blocker on your behalf. Otherwise ask that it be considered for the next Red Hat Enterprise Linux release. ** I'm going to dup this one to bug 591466 for the mark_buffer_dirty issue. If there are other problems that arise in testing, please open new bugs. Thanks! -Eric *** This bug has been marked as a duplicate of bug 591466 *** |