RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 617165 - mount operation failed and hung on some images which running in read-only mode
Summary: mount operation failed and hung on some images which running in read-only mode
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libguestfs
Version: 6.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 612493 (view as bug list)
Depends On: 617200
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-22 11:23 UTC by Linqing Lu
Modified: 2013-04-30 22:46 UTC (History)
5 users (show)

Fixed In Version: libguestfs-1.2.7-1.23.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 617200 (view as bug list)
Environment:
Last Closed: 2010-11-10 21:03:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
output info of virt-ls command with an image which has this problem (28.76 KB, text/plain)
2010-07-22 11:25 UTC, Linqing Lu 		no flags Details
View All

Description Linqing Lu 2010-07-22 11:23:05 UTC 
Description of problem:
1- If running images in read-only mode, SOME of them will cause I/O errors and hang forever during mount ro operation. Such as using "mount ro" in guestfish, or virt-ls, virt-cat tools on these images. 
2- But if we run these images in rw mode, this issue will not appear. And this issue will never happen with the image after this successful mount rw operation. Such as using "mount rw" or virt-edit tool on these images.
3- When "mount" operation executing in those images which have this issue, it shows "INFO: recovery required on readonly filesystem." at first. If this "mount" operation is ro, it will show some I/O error and hang. Otherwise the "mount" is rw, is will succeed and show "recovery completed."

Version-Release number of selected component (if applicable):
libguestfs-1.2.7-1.17.el6.x86_64

How reproducible:
always, unless after a successful "mount"

Steps to Reproduce:
1. running guestfish or virt-tools on a image which is "recovery required" in read-only mode.
2. if in guestfish, use mount ro. if using virt-tools, use virt-cat/virt-df/virt-ls etc.

Actual results:
mount failed with I/O errors showed (if using virt-tools, need env LIBGUESTFS_DEBUG=1 to show the info), then hung there

Expected results:
mount successfully, and output correct info of the image

Additional info:
the error output info while using virt-ls as an example is attached
Comment 1 Linqing Lu 2010-07-22 11:25:02 UTC 
Created attachment 433671 [details]
output info of virt-ls command with an image which has this problem
Comment 2 Linqing Lu 2010-07-22 11:35:33 UTC 
this issue happened on a RHEL5.5_32 image and a RHEL5.4_64 image
Comment 4 RHEL Program Management 2010-07-22 11:57:51 UTC 
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 5 Qixiang Wan 2010-07-22 12:40:39 UTC 
after checked the images (boot it with qemu-kvm), there was some minor error in filesystem:
[...]
Mounting root filesystem.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access wilee be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
[...]

libguestfs should do some enhancement for such cases (quit after mount failed):

[host]# pstree -p 706
virt-ls(706)─┬─qemu-kvm(713)───{qemu-kvm}(715)
             └─virt-ls(714)
[host]# strace -p 706
Process 706 attached - interrupt to quit
select(7, [4 6], NULL, NULL, NULL^C <unfinished ...>
Process 706 detached
[host]# strace -p 714
Process 714 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff68c46ac0)       = 0
kill(713, SIG_0)                        = 0
kill(706, SIG_0)                        = 0
Comment 6 Richard W.M. Jones 2010-07-22 13:23:56 UTC 
Reproducer #1:

$ cat make-image.sh 
#!/bin/sh -
guestfish <<EOF
sparse /tmp/test.img 1G
run
part-disk /dev/sda mbr
mkfs ext3 /dev/sda1
mount /dev/sda1 /
touch /hello
sync
sleep 1
kill-subprocess
EOF

$ ./make-image.sh
[ignore the error from the above command]
$ guestfish --ro -a /tmp/test.img -m /dev/sda1 -v

The error you will see is:

  mount -o ro /dev/vda1 /sysroot/
  EXT3-fs: INFO: recovery required on readonly filesystem.
  EXT3-fs: write access unavailable, cannot proceed.

Reproducer #2:


$ cat make-image.sh 
#!/bin/sh -
guestfish <<EOF
sparse /tmp/test.img 1G
run
part-disk /dev/sda mbr
pvcreate /dev/sda1
vgcreate VG /dev/sda1
lvcreate LV VG 128
mkfs ext3 /dev/VG/LV
mount /dev/VG/LV /
touch /hello
sync
sleep 1
kill-subprocess
EOF

$ ./make-image.sh
[ignore the error from the above command]
$ guestfish --ro -a /tmp/test.img -m /dev/VG/LV -v

The error you will see is different from reproducer #1 but
similar to the errors shown in Linglu's comment 1:

  mount -o ro /dev/VG/LV /sysroot/
  EXT3-fs: INFO: recovery required on readonly filesystem.
  EXT3-fs: write access will be enabled during recovery.
  end_request: I/O error, dev vda, sector 389
  Buffer I/O error on device dm-0, logical block 2
  lost page write due to I/O error on dm-0
  [more of these errors or hangs]
Comment 7 Richard W.M. Jones 2010-07-22 13:25:52 UTC 
The solution to this is to remove the code which detects if the
readonly=yes|no option is supported by qemu.  Instead we should
just use snapshots as we do on Fedora.

http://git.annexia.org/?p=libguestfs.git;a=blob;f=src/guestfs.c;h=85a042a0a2eaf6b48d068f8e1d4ee0d854018e2f;hb=HEAD#l852

Since this is also an upstream bug, I will clone this and fix it
upstream first.
Comment 8 Richard W.M. Jones 2010-07-22 15:56:44 UTC 
Upstream patch here:
http://git.annexia.org/?p=libguestfs.git;a=commitdiff;h=799d52be4f08f6c70c0e8ba1aa7367ba4cdd78c4

I will try this on my private RHEL 6 VM first to see
if it resolves the issues in comment 6.
Comment 9 Richard W.M. Jones 2010-07-22 19:03:44 UTC 
Brew build:
https://brewweb.devel.redhat.com/taskinfo?taskID=2618737

I'm still testing this one.
Comment 10 Jinxin Zheng 2010-07-23 02:41:21 UTC 
(In reply to comment #8)
> Upstream patch here:
> http://git.annexia.org/?p=libguestfs.git;a=commitdiff;h=799d52be4f08f6c70c0e8ba1aa7367ba4cdd78c4
> 

Seems this fix will affect bug 612493 as well.
Comment 11 Richard W.M. Jones 2010-07-23 17:26:17 UTC 
Sorry, took quite a lot longer to test this than I anticipated,
because I had to download and install RHEL 6 beta 2 refresh.

Anyway, I can confirm that the build in comment 9 fixes
the problem.
Comment 12 Richard W.M. Jones 2010-07-23 17:35:35 UTC 
Non-scratch brew build:
https://brewweb.devel.redhat.com/taskinfo?taskID=2621237
Comment 14 Linqing Lu 2010-07-29 07:24:07 UTC 
1- Update packages:
guestfish-1.2.7-1.21.el6.x86_64.rpm
libguestfs-mount-1.2.7-1.21.el6.x86_64.rpm
perl-libguestfs-1.2.7-1.21.el6.x86_64.rpm
libguestfs-1.2.7-1.21.el6.x86_64.rpm
libguestfs-tools-1.2.7-1.21.el6.x86_64.rpm

2- Verified with the Reproducer in Comment 6:
Instead of the error message "...write access unavailable, cannot proceed...", the mount operation ran well with following message:
    mount -o ro /dev/vda1 /sysroot/
    [    2.955797] EXT3-fs: INFO: recovery required on readonly filesystem.
    [    2.957494] EXT3-fs: write access will be enabled during recovery.
    [    3.557200] kjournald starting.  Commit interval 5 seconds
    [    3.561454] EXT3-fs: recovery complete.
    [    3.564659] EXT3-fs: mounted filesystem with ordered data mode.
Test passed.

3- Verified with the images mention in Comment 2:
Same situation with PART 2.
Test passed.
Comment 15 Jinxin Zheng 2010-07-29 07:27:44 UTC 
Move to VERIFIED according to comment 14.
Comment 16 Richard W.M. Jones 2010-08-04 16:25:53 UTC 
I made a small modification to the patch -- to fix the documentation.
It probably doesn't affect anything, but I've put this back to MODIFIED
just in case.  The new build is libguestfs-1.2.7-1.23.el6 (snap 11).
Comment 18 Linqing Lu 2010-08-06 10:14:55 UTC 
Test passed with the same procedures in Comment 14.
Comment 19 Richard W.M. Jones 2010-08-23 09:03:47 UTC 
*** Bug 612493 has been marked as a duplicate of this bug. ***
Comment 20 releng-rhel@redhat.com 2010-11-10 21:03:13 UTC 
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.
Note You need to log in before you can comment on or make changes to this bug.