Red Hat Bugzilla – Bug 549397
I/O errors while accessing loop devices or file-based Xen images from GFS volume after Update from RHEL 5.3 to 5.4
Last modified: 2010-10-23 09:53:09 EDT
Description of problem:
An update on a Xen virtualization cluster has been made: yum update, RHEL 5.3 to RHEL 5.4
The cluster has a gfs volume where image files for VMs are stored.
Afterwards, the machines fail to boot with fs errors, console output: (console.log)
Create a loop device from a file which is stored on GFS volume and try to mount or fsck it
Steps to Reproduce:
1. Create an image file (ex. using dd) on a local ext3 volume
2. Create a loop device from the file (kpartx)
3. Create a partition table and a partition on the loop device (fdisk)
4. Create an ext3 filesystem on the partition (mkfs -t ext3)
5. Mount the filesystem, check dmesg output
6. Unmount fs, destroy the loop device
7. Copy the image file to an GFS volume
8. Create a loop device from the file on a GFS volume
9. Mount the filesystem, check dmesg output
The filesystem from ext3 volume can be mounted, the one from gfs volume cannot.
Both filesystems should be mountable, as they are exact clones
It did not happen to all of the virtual machines, but ca. 30% of them. Newly created image files are affected. xend.log reports: (see xend.log)
If one tries to mount the image files as loop devices, FS-errors on all partitions in the file are reported, read-only access is possible, no rw access.
Once copied and repaired (or newly created) on a local ext3 filesystem, the image files (or files acting as loop devices generally) are healthy and usable. If they are copied back onto the GFS volume, same errors occur.
Error after trying to mount a loop device: (mount.log)
dmesg reports: (dmesg.log)
Failback to the .128.el5xen kernel (with updated packages) did not solve the issue. The issue applies both to 164.6.1.el5xen and 164.9.1.el5xen kernels.
RPM list: (rpm.list)
We were able to reproduce the problem on a different, fresh installation of a single-node cluster - RHEL5.4 Clone; 2.6.18-164.6.1.el5 kernel
Similar bug has been known: https://bugzilla.redhat.com/show_bug.cgi?id=164499
Created attachment 379634 [details]
Xen console output after starting a VM with disk on a gfs volume
Created attachment 379635 [details]
dmesg output after trying to mount a loop device from gfs volume
Created attachment 379636 [details]
console output after trying to mount a loof device from a gfs volume
Created attachment 379637 [details]
xen log after starting a VM with disk on a gfs volume
Created attachment 379638 [details]
List of rpms istalled on the system
I can see whats going on there. The loop device is trying to call (not unreasonably) gfs_prepare_write as gfs is trying to write to a block. The issue is that this is not allowed unless the caller has already locked the glock.
Unfortunately it is a consequence of the level at which gfs does its locking. GFS2 locks at the page cache level so that it will not have this issue. Also it will only affect writes, so that read only mounts via loop should be unaffected.
The problem seems to be due to this bit of the new aops patches:
@@ -791,7 +770,7 @@ static int loop_set_fd(struct loop_device *lo, struct file *
- if (aops->prepare_write && aops->commit_write)
+ if (aops->prepare_write || aops->write_begin)
lo_flags |= LO_FLAGS_USE_AOPS;
if (!(lo_flags & LO_FLAGS_USE_AOPS) && !file->f_op->write)
lo_flags |= LO_FLAGS_READ_ONLY;
We were relying on the lack of a commit_write aop to not set the LO_FLAGS_USE_AOPS flag.
Before I forget where it is, here is the gfs end of this issue:
That patch was added to fix this issue the first time it cropped up.
Created attachment 380060 [details]
Please try this patch and verify it fixes your problem. I've built it and tested it to make sure it doesn't blow up, but I didn't test it with gfs.
Changing bug from RHEL4.x to 5.x.
In my opinion, this needs to get fixed in 5.5. Setting and/or
requesting flags appropriately.
I currently have some problems building a kernel RPM from SRPM. As soon as that works, I will check out the patch and give you feedback on this matter.
Happy New Year!
I've built new kernel rpm, installed it, tested. It seems to work fine now.
When can I expect an official bugfix, so I can go productive with it, with a supported kernel?
Trying to join cluster "lock_nolock", "clurhel5:root"
Joined cluster. Now mounting FS...
GFS: fsid=clurhel5:root.0: jid=0: Trying to acquire journal lock...
GFS: fsid=clurhel5:root.0: jid=0: Looking at journal...
GFS: fsid=clurhel5:root.0: jid=3: Done
GFS: fsid=clurhel5:root.0: Scanning for log elements...
GFS: fsid=clurhel5:root.0: Found 0 unlinked inodes
GFS: fsid=clurhel5:root.0: Found quota changes for 0 IDs
GFS: fsid=clurhel5:root.0: Done
[create loop device from a file on the gfs volume]
loop: loaded (max 8 devices)
[create ext3 filesystem on loop device, mount it]
kjournald starting. Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
[no gfs errors! good job!]
I'm reassigning this to Josef, since he wrote the patch.
We're investigating whether we can still push this into 5.5 or not.
It's very late in the process but we might be able to go through
the exception process. I'll bump the priority toward that end,
although we might need to bump it higher.
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release. Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products. This request is not yet committed for inclusion in an Update
RHEL 5.5 for this problem is not an option. I have a productive system with failed RHEL 5.4 update, starting the productive virtual machines may lead to data corruption.
I need a supported kernel patch ASAP.
This bug has to go into 5.5 first. If it is also required in 5.4.z then this bug needs to be cloned in order for that to happen. That is normally done via GSS so if you have a contact there, please ask them to do that. If not let us know and I'll try and kick off the process directly.
*** Bug 566184 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.