Bug 1567763 - guestfsd: error: /Windows/Drivers/VirtIO: Read-only file system
Summary: guestfsd: error: /Windows/Drivers/VirtIO: Read-only file system
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1551055
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-16 07:47 UTC by Richard W.M. Jones
Modified: 2018-10-30 07:46 UTC (History)
5 users (show)

Fixed In Version: libguestfs-1.38.1-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 07:45:35 UTC
Target Upstream Version:


Attachments (Terms of Use)
log.txt (163.18 KB, text/plain)
2018-04-16 07:48 UTC, Richard W.M. Jones
no flags Details
can-not-access-c-drive (33.50 KB, image/png)
2018-05-11 10:52 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:3021 0 None None None 2018-10-30 07:46:21 UTC

Description Richard W.M. Jones 2018-04-16 07:47:29 UTC
Description of problem:

Windows guest using Fast Startup triggers a fallback path in
ntfs-3g where the disk is mounted readonly (without actually
failing) and you end up with a confusing error message.  We
should detect if the fallback path was triggered and error out
earlier.

libguestfs: trace: v2v: mount "/dev/sda2" "/"
guestfsd: main_loop: proc 47 (umount_all) took 0.02 seconds
guestfsd: main_loop: new request, len 0x40
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: udevadm --debug settle -E /dev/sda2
calling: settle
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: mount -o  /dev/sda2 /sysroot/
The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
libguestfs: trace: v2v: mount = 0

Bug reported by Piotr Kliczewski.

Version-Release number of selected component (if applicable):

libguestfs 1.36.13fedora=26,release=1.fc26,libvirt (x86_64)

Steps to Reproduce:
1. Unknown.

Comment 2 Richard W.M. Jones 2018-04-16 07:48:08 UTC
Created attachment 1422380 [details]
log.txt

virt-v2v -v -x log.

Comment 3 Richard W.M. Jones 2018-04-16 09:48:49 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2018-April/msg00134.html

Comment 4 Pino Toscano 2018-05-02 11:28:32 UTC
Fixed upstream with
https://github.com/libguestfs/libguestfs/commit/201e6d1595c1cdae7f1fbce11626412768b33f99
which is in libguestfs >= 1.39.3.

Also, it was backported to 1.38.1.

Comment 6 mxie@redhat.com 2018-05-11 10:52:07 UTC
Hi rjones,
   
   I have no idea how to reproduce the bug, could you please give me some suggestion? And I have a question why v2v could convert the windows guest when even os's administrator can't access to c:\ drive

Packages:
virt-v2v-1.38.0-1.el7.x86_64
libguestfs-1.38.0-1.el7.x86_64

Steps:
1.Prepare a windows guest and remove all permission for all users in C:\ drive setting, then c:\ can't be accessed by anyone, pls refer to screenshot
2.Use virt-v2v to convert the guest
# virt-v2v win2k16 -o null
[   0.0] Opening the source -i libvirt win2k16
[   0.0] Creating an overlay to protect the source from being modified
[   0.5] Initializing the target -o null
[   0.5] Opening the overlay
[  78.5] Inspecting the overlay
[  82.2] Checking for sufficient free disk space in the guest
[  82.2] Estimating space required on target for each disk
[  82.2] Converting Windows Server 2016 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  93.2] Mapping filesystem data to avoid copying unused and blank areas
[  96.0] Closing the overlay
[  96.4] Checking if the guest needs BIOS or UEFI to boot
[  96.4] Assigning disks to buses
[  96.4] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[ 217.0] Creating output metadata
[ 217.0] Finishing off


Thanks

Comment 7 mxie@redhat.com 2018-05-11 10:52:49 UTC
Created attachment 1434863 [details]
can-not-access-c-drive

Comment 8 Richard W.M. Jones 2018-05-16 10:40:04 UTC
Can you try converting a Windows guest which is running, or one
which has been powered off abruptly (without proper shutdown of
the guest), or one which has hibernation enabled.  One of those
ought to reproduce the bug.

By the way this is actually caused by a change in ntfs-3g:

https://sourceforge.net/p/ntfs-3g/ntfs-3g/ci/8a1358fff439d2fe464ab67ba5d5394060880457

Comment 9 Piotr Kliczewski 2018-05-16 12:02:06 UTC
I noticed this behaviour when using stopped vm with hibernation enabled.

Comment 10 mxie@redhat.com 2018-05-21 08:56:17 UTC
I only can reproduce the bug with virt-v2v fedora builds:
virt-v2v-1.36.13-1.fc26.x86_64
libguestfs-1.36.13-1.fc26.x86_64

Reproduce steps:
1.Convert a win2012 guest by virt-v2v which is forced shutdown using external button, the conversion is failed with same error with bug no matter if hibernation is on or off in windows guest
# virt-v2v -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export -on fedora-win2012
libguestfs: trace: v2v: mount "/dev/sda2" "/"
guestfsd: main_loop: proc 47 (umount_all) took 0.73 seconds
guestfsd: main_loop: new request, len 0x40
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: udevadm --debug settle -E /dev/sda2
calling: settle
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: mount -o  /dev/sda2 /sysroot/
The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
libguestfs: trace: v2v: mount = 0
......
......
guestfsd: error: /Windows/Drivers/VirtIO: Read-only file system
libguestfs: trace: v2v: mkdir_p = -1 (error)
libguestfs: trace: v2v: hivex_close
guestfsd: main_loop: proc 33 (mkdir_p) took 0.09 seconds
guestfsd: main_loop: new request, len 0x28
hivex: hivex_close: hivex_close
libguestfs: trace: v2v: hivex_close = 0
virt-v2v: error: libguestfs error: mkdir_p: /Windows/Drivers/VirtIO: 
Read-only file system
.....
.....

2.If convert same guest with step1 with virt-v2v-1.38.2-1.el7.x86_64,get different error with bug but the error info is correct 

# virt-v2v -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   2.0] Creating an overlay to protect the source from being modified
[   3.1] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.5] Opening the overlay
[  19.9] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Hi rjones,

   I think this bug should be filed on fedora, it is not virt-v2v x86_64 bug

Comment 11 Richard W.M. Jones 2018-05-21 10:17:14 UTC
First we have to be clear what the fix does.  It does *not* allow the
guest to be converted, all it does it to improve the error message
that the user sees.

Before the fix:

  guestfsd: error: /Windows/Drivers/VirtIO: Read-only file system

After the fix:

  error: filesystem was mounted read-only, even though we asked for it to be mounted read-write.  This usually means that the filesystem was not cleanly unmounted.  Possible causes include trying to convert a guest which is running, or using Windows Hibernation or Fast Restart.

So comment 10, (1) has reproduced the original bug successfully.

However comment 10, (2) has not shown the fix.

This is likely because your version of ntfs-3g is too old.  It lacks
the undesirable fallback-to-readonly behaviour which was introduced
in ntfs-3g in 2016-04 (https://sourceforge.net/p/ntfs-3g/ntfs-3g/ci/8a1358fff439d2fe464ab67ba5d5394060880457).

So I'm interested in what version of ntfs-3g you are using.  It might
be either ntfs-3g-2017.3.23-4.fc26 (contains the undesirable behaviour
and should demonstrate the bug) or ntfs-3g-2016.2.22-4.fc26 (does not
have the undesirable behaviour and would not show the bug).

Comment 12 mxie@redhat.com 2018-06-06 12:55:20 UTC
Hi rjones,

   What I want to say in comment10 is I could reproduce the bug with virt-v2v-1.36.13-1.fc26.x86_64, but I can't reproduce the bug with v2v rhel7 builds (such as virt-v2v-1.36.10-6.el7.x86_64), so I think this bug is v2v fedora bug rather than v2v rhel7 bug

  Besides, I can find ntfs-3g in fedora26 v2v conversion server

    # rpm -q virt-v2v libguestfs ntfs-3g
    virt-v2v-1.36.13-1.fc26.x86_64
    libguestfs-1.36.13-1.fc26.x86_64
    ntfs-3g-2017.3.23-4.fc26.x86_64
   
  But I can't find ntfs-3g in rhel7 v2v conversion server
    
    # rpm -q virt-v2v libguestfs ntgs-3g
    virt-v2v-1.36.10-6.el7.x86_64
    libguestfs-1.36.10-6.el7.x86_64
    package ntgs-3g is not installed


    According to your comment11, this bug is also related to ntfs-3g, because v2v doesn't depend on ntgs-3g on rhel7, this bug should be opened on fedora product

Comment 14 mxie@redhat.com 2018-06-07 08:30:59 UTC
*Try to reproduce the bug:

Reproduce scenario1:
Try to reproduce the bug with v2v fedora26 builds:
virt-v2v-1.36.13-1.fc26.x86_64
libguestfs-1.36.13-1.fc26.x86_64
ntfs-3g-2017.3.23-4.fc26.x86_64

1.1 Convert a win2012 guest from vmware which is forced shut down using external button
#  virt-v2v -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   1.3] Creating an overlay to protect the source from being modified
[   2.0] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   2.4] Opening the overlay
[ 111.5] Inspecting the overlay
[ 218.5] Checking for sufficient free disk space in the guest
[ 218.5] Estimating space required on target for each disk
[ 218.5] Converting Windows Server 2012 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: /usr/share/virt-tools/rhev-apt.exe is missing, but the 
output hypervisor is oVirt or RHV.  Installing RHEV-APT in the guest would 
mean the guest is automatically updated with new drivers etc.  You may wish 
to install RHEV-APT manually after conversion.
virt-v2v: error: libguestfs error: mkdir_p: /Windows/Drivers/VirtIO: 
Read-only file system

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Reproduce result 1:
      Can reproduce the bug with v2v fedora26 builds
       

Reproduce scenario2:
Try to reproduce the bug with v2v rhel7.5 builds:
virt-v2v-1.36.10-6.el7.x86_64
libguestfs-1.36.10-6.el7.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64

2.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   2.3] Creating an overlay to protect the source from being modified
[   3.3] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.6] Opening the overlay
[  13.7] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: /dev/sda2 on / (options: ''): The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Reproduce result 2:
      Can't reproduce the bug with v2v rhel7.5 builds


Reproduce scenario3:
Try to reproduce the bug with v2v old rhel8 builds:
virt-v2v-1.38.0-1.el8+5.x86_64
libguestfs-1.38.0-1.el8+5.x86_64
libguestfs-winsupport-7.2-2.el8+5.x86_64

3.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -i vmx esx6.0-win2012-x86_64.vmx -o null
[   0.0] Opening the source -i vmx esx6.0-win2012-x86_64.vmx
[   0.0] Creating an overlay to protect the source from being modified
[   0.2] Initializing the target -o null
[   0.2] Opening the overlay
[   4.6] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Reproduce result 3:
      Can't reproduce the bug with v2v old rhel8 builds

_____________________________________________________________________________


*Try to verify the bug:

Verify scenario1:
Verify the bug with v2v fedora28 builds
virt-v2v-1.38.2-1.fc28.x86_64
libguestfs-1.38.2-1.fc28.x86_64
ntfs-3g-2017.3.23-6.fc28.x86_64

1.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -i vmx esx6.0-win2012-x86_64.vmx -o null
[   0.0] Opening the source -i vmx esx6.0-win2012-x86_64.vmx
[   0.2] Creating an overlay to protect the source from being modified
[   0.3] Initializing the target -o null
[   0.3] Opening the overlay
[  47.2] Inspecting the overlay
virt-v2v: error: filesystem was mounted read-only, even though we asked for 
it to be mounted read-write.  This usually means that the filesystem was 
not cleanly unmounted.  Possible causes include trying to convert a guest 
which is running, or using Windows Hibernation or Fast Restart.

Original error message: touch: open: /4n7d407y: Read-only file system

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Verify result 1:
       The virt-v2v error is same with comment11's expect result


Verify scenario2:
Verify the bug with v2v latest rhel7.6 builds
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64

2.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   1.9] Creating an overlay to protect the source from being modified
[   2.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.1] Opening the overlay
[  41.1] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shut down
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Verify result 2:
       The virt-v2v error shows correct with virt-v2v latest rhel7.6 builds although it is different with comment11


Verify scenario3 :
Verify the bug with v2v rhel8 new builds
virt-v2v-1.38.0-5.el8+587+3b2138fb.x86_64
libguestfs-1.38.0-5.el8+587+3b2138fb.x86_64
libguestfs-winsupport-7.2-2.el8+526+412dc3e0.x86_64

3.1 Convert a win2012 guest from vmware which is forced shut down using external button
#  virt-v2v -i vmx esx6.0-win2012-x86_64.vmx -o null
[   1.8] Opening the source -i vmx esx6.0-win2012-x86_64.vmx
[   1.8] Creating an overlay to protect the source from being modified
[   2.3] Initializing the target -o null
[   2.3] Opening the overlay
[  37.1] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Verify result 3:
       The virt-v2v error shows correct with virt-v2v rhel8 builds although it is different with comment11



Hi rjones,
 
    This bug could be reproduced on v2v fedora26 builds and has fixed in v2v fedora28 builds, I can't reproduce it on rhel7 or rhel8 (it's hard to reproduce on rhel7 or rhel8 as comment13 said), could I move the bug to verified?

Comment 16 mxie@redhat.com 2018-06-07 09:28:57 UTC
Move the bug from ON_QA to VERIFIED according to comment11 ~ comment 14

Comment 18 errata-xmlrpc 2018-10-30 07:45:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3021


Note You need to log in before you can comment on or make changes to this bug.