Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

For bugs related to Red Hat Enterprise Linux 4 product line. The current stable release is 4.9. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 664376

Summary:	temporary loss of path to SAN results in corrupted filesystem
Product:	Red Hat Enterprise Linux 4	Reporter:	Lachlan McIlroy <lmcilroy>
Component:	kernel	Assignee:	Red Hat Kernel Manager <kernel-mgr>
Status:	CLOSED WONTFIX	QA Contact:	Gris Ge <fge>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	4.6	CC:	fge, jwest, rwheeler, vgaikwad
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-06-14 18:45:00 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lachlan McIlroy 2010-12-20 08:22:01 UTC

Description of problem:

This is following on from BZ645633 (that bug has been fixed):
https://bugzilla.redhat.com/show_bug.cgi?id=645633

The customer is now reporting filesystem corruption after running their tests. The test involves temporarily dropping the path to the SAN while a program is performing I/O to an ext2 filesystem. The test does not report user data corruption but occasionally the filesystem reports errors about being unable to read inodes/blocks such as:

EXT2-fs error (device sdr1): read_block_bitmap: Cannot read block bitmap - block_group = 0, block_bitmap = 259
EXT2-fs error (device sdr2): read_block_bitmap: Cannot read block bitmap - block_group = 2, block_bitmap = 16385
EXT2-fs error (device sdr3): read_block_bitmap: Cannot read block bitmap - block_group = 1, block_bitmap = 8451

Which looks like it's trying to read metadata while the path to the SAN is unavailable. After the filesystem is unmounted e2fsck reports corruption in the filesystem. Lost metadata writes while the path is unavailable is probably the reason for the corrupted filesystem.

The same test also reveals filesystem corruption on ext3 but barriers might help in that case.

Version-Release number of selected component (if applicable):
kernel-2.6.9-89.EL

How reproducible:
Fairly regularly.

Steps to Reproduce:
1. Create and mount an ext2 filesystem
2. Add a metadata intensive load on the filesystem, ie
for i in `seq 1 1000000`; do touch $i; done
3. Wait a while and then unplug and reconnect cable to SAN or simulate with:
# cd /sys/class/scsi_host/host0/device/target0:0:0/0:0:0:0/
# echo offline > state ; sleep 1 ; echo running > state
4. Stop load and unmount filesystem
5. Run e2fsck over filesystem

Comment 1 Lachlan McIlroy 2011-01-14 05:16:57 UTC

While running the test for this bug I got these errors:

EXT2-fs error (device dm-10): read_inode_bitmap: Cannot read inode bitmap - block_group = 6, inode_bitmap = 196609
EXT2-fs error (device dm-10): read_inode_bitmap: Cannot read inode bitmap - block_group = 6, inode_bitmap = 196609
EXT2-fs error (device dm-10): read_inode_bitmap: Cannot read inode bitmap - block_group = 6, inode_bitmap = 196609
...

And when I tried to remove the files generated by the test I got these errors:

EXT2-fs error (device dm-10): ext2_free_inode: bit already cleared for inode 96002
EXT2-fs error (device dm-10): ext2_free_inode: bit already cleared for inode 96011
EXT2-fs error (device dm-10): ext2_free_inode: bit already cleared for inode 96101
...

We've seen many cases of the "bit already cleared" errors on ext3 so maybe now we have an explanation - failed metadata updates.

To workaround this bug I use dm-multipath (even for just one path) with no_path_retry set to queue.  This causes all failed I/Os to be queued and retried indefinitely so no metadata updates are lost.  With this setup I cannot reproduce the bug.