Bug 1567763

Created attachment 1422380 [details]
log.txt

virt-v2v -v -x log.

Patch posted:
https://www.redhat.com/archives/libguestfs/2018-April/msg00134.html

Fixed upstream with
https://github.com/libguestfs/libguestfs/commit/201e6d1595c1cdae7f1fbce11626412768b33f99
which is in libguestfs >= 1.39.3.

Also, it was backported to 1.38.1.

Hi rjones,
   
   I have no idea how to reproduce the bug, could you please give me some suggestion? And I have a question why v2v could convert the windows guest when even os's administrator can't access to c:\ drive

Packages:
virt-v2v-1.38.0-1.el7.x86_64
libguestfs-1.38.0-1.el7.x86_64

Steps:
1.Prepare a windows guest and remove all permission for all users in C:\ drive setting, then c:\ can't be accessed by anyone, pls refer to screenshot
2.Use virt-v2v to convert the guest
# virt-v2v win2k16 -o null
[   0.0] Opening the source -i libvirt win2k16
[   0.0] Creating an overlay to protect the source from being modified
[   0.5] Initializing the target -o null
[   0.5] Opening the overlay
[  78.5] Inspecting the overlay
[  82.2] Checking for sufficient free disk space in the guest
[  82.2] Estimating space required on target for each disk
[  82.2] Converting Windows Server 2016 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  93.2] Mapping filesystem data to avoid copying unused and blank areas
[  96.0] Closing the overlay
[  96.4] Checking if the guest needs BIOS or UEFI to boot
[  96.4] Assigning disks to buses
[  96.4] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[ 217.0] Creating output metadata
[ 217.0] Finishing off


Thanks

Created attachment 1434863 [details]
can-not-access-c-drive

Can you try converting a Windows guest which is running, or one
which has been powered off abruptly (without proper shutdown of
the guest), or one which has hibernation enabled.  One of those
ought to reproduce the bug.

By the way this is actually caused by a change in ntfs-3g:

https://sourceforge.net/p/ntfs-3g/ntfs-3g/ci/8a1358fff439d2fe464ab67ba5d5394060880457

I noticed this behaviour when using stopped vm with hibernation enabled.

I only can reproduce the bug with virt-v2v fedora builds:
virt-v2v-1.36.13-1.fc26.x86_64
libguestfs-1.36.13-1.fc26.x86_64

Reproduce steps:
1.Convert a win2012 guest by virt-v2v which is forced shutdown using external button, the conversion is failed with same error with bug no matter if hibernation is on or off in windows guest
# virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export -on fedora-win2012
libguestfs: trace: v2v: mount "/dev/sda2" "/"
guestfsd: main_loop: proc 47 (umount_all) took 0.73 seconds
guestfsd: main_loop: new request, len 0x40
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: udevadm --debug settle -E /dev/sda2
calling: settle
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: mount -o  /dev/sda2 /sysroot/
The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
libguestfs: trace: v2v: mount = 0
......
......
guestfsd: error: /Windows/Drivers/VirtIO: Read-only file system
libguestfs: trace: v2v: mkdir_p = -1 (error)
libguestfs: trace: v2v: hivex_close
guestfsd: main_loop: proc 33 (mkdir_p) took 0.09 seconds
guestfsd: main_loop: new request, len 0x28
hivex: hivex_close: hivex_close
libguestfs: trace: v2v: hivex_close = 0
virt-v2v: error: libguestfs error: mkdir_p: /Windows/Drivers/VirtIO: 
Read-only file system
.....
.....

2.If convert same guest with step1 with virt-v2v-1.38.2-1.el7.x86_64,get different error with bug but the error info is correct 

# virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   2.0] Creating an overlay to protect the source from being modified
[   3.1] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.5] Opening the overlay
[  19.9] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Hi rjones,

   I think this bug should be filed on fedora, it is not virt-v2v x86_64 bug

First we have to be clear what the fix does.  It does *not* allow the
guest to be converted, all it does it to improve the error message
that the user sees.

Before the fix:

  guestfsd: error: /Windows/Drivers/VirtIO: Read-only file system

After the fix:

  error: filesystem was mounted read-only, even though we asked for it to be mounted read-write.  This usually means that the filesystem was not cleanly unmounted.  Possible causes include trying to convert a guest which is running, or using Windows Hibernation or Fast Restart.

So comment 10, (1) has reproduced the original bug successfully.

However comment 10, (2) has not shown the fix.

This is likely because your version of ntfs-3g is too old.  It lacks
the undesirable fallback-to-readonly behaviour which was introduced
in ntfs-3g in 2016-04 (https://sourceforge.net/p/ntfs-3g/ntfs-3g/ci/8a1358fff439d2fe464ab67ba5d5394060880457).

So I'm interested in what version of ntfs-3g you are using.  It might
be either ntfs-3g-2017.3.23-4.fc26 (contains the undesirable behaviour
and should demonstrate the bug) or ntfs-3g-2016.2.22-4.fc26 (does not
have the undesirable behaviour and would not show the bug).

Hi rjones,

   What I want to say in comment10 is I could reproduce the bug with virt-v2v-1.36.13-1.fc26.x86_64, but I can't reproduce the bug with v2v rhel7 builds (such as virt-v2v-1.36.10-6.el7.x86_64), so I think this bug is v2v fedora bug rather than v2v rhel7 bug

  Besides, I can find ntfs-3g in fedora26 v2v conversion server

    # rpm -q virt-v2v libguestfs ntfs-3g
    virt-v2v-1.36.13-1.fc26.x86_64
    libguestfs-1.36.13-1.fc26.x86_64
    ntfs-3g-2017.3.23-4.fc26.x86_64
   
  But I can't find ntfs-3g in rhel7 v2v conversion server
    
    # rpm -q virt-v2v libguestfs ntgs-3g
    virt-v2v-1.36.10-6.el7.x86_64
    libguestfs-1.36.10-6.el7.x86_64
    package ntgs-3g is not installed


    According to your comment11, this bug is also related to ntfs-3g, because v2v doesn't depend on ntgs-3g on rhel7, this bug should be opened on fedora product

*Try to reproduce the bug:

Reproduce scenario1:
Try to reproduce the bug with v2v fedora26 builds:
virt-v2v-1.36.13-1.fc26.x86_64
libguestfs-1.36.13-1.fc26.x86_64
ntfs-3g-2017.3.23-4.fc26.x86_64

1.1 Convert a win2012 guest from vmware which is forced shut down using external button
#  virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   1.3] Creating an overlay to protect the source from being modified
[   2.0] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   2.4] Opening the overlay
[ 111.5] Inspecting the overlay
[ 218.5] Checking for sufficient free disk space in the guest
[ 218.5] Estimating space required on target for each disk
[ 218.5] Converting Windows Server 2012 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: /usr/share/virt-tools/rhev-apt.exe is missing, but the 
output hypervisor is oVirt or RHV.  Installing RHEV-APT in the guest would 
mean the guest is automatically updated with new drivers etc.  You may wish 
to install RHEV-APT manually after conversion.
virt-v2v: error: libguestfs error: mkdir_p: /Windows/Drivers/VirtIO: 
Read-only file system

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Reproduce result 1：
      Can reproduce the bug with v2v fedora26 builds
       

Reproduce scenario2:
Try to reproduce the bug with v2v rhel7.5 builds:
virt-v2v-1.36.10-6.el7.x86_64
libguestfs-1.36.10-6.el7.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64

2.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   2.3] Creating an overlay to protect the source from being modified
[   3.3] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.6] Opening the overlay
[  13.7] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: /dev/sda2 on / (options: ''): The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Reproduce result 2：
      Can't reproduce the bug with v2v rhel7.5 builds


Reproduce scenario3:
Try to reproduce the bug with v2v old rhel8 builds:
virt-v2v-1.38.0-1.el8+5.x86_64
libguestfs-1.38.0-1.el8+5.x86_64
libguestfs-winsupport-7.2-2.el8+5.x86_64

3.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -i vmx esx6.0-win2012-x86_64.vmx -o null
[   0.0] Opening the source -i vmx esx6.0-win2012-x86_64.vmx
[   0.0] Creating an overlay to protect the source from being modified
[   0.2] Initializing the target -o null
[   0.2] Opening the overlay
[   4.6] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Reproduce result 3：
      Can't reproduce the bug with v2v old rhel8 builds

_____________________________________________________________________________


*Try to verify the bug:

Verify scenario1:
Verify the bug with v2v fedora28 builds
virt-v2v-1.38.2-1.fc28.x86_64
libguestfs-1.38.2-1.fc28.x86_64
ntfs-3g-2017.3.23-6.fc28.x86_64

1.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -i vmx esx6.0-win2012-x86_64.vmx -o null
[   0.0] Opening the source -i vmx esx6.0-win2012-x86_64.vmx
[   0.2] Creating an overlay to protect the source from being modified
[   0.3] Initializing the target -o null
[   0.3] Opening the overlay
[  47.2] Inspecting the overlay
virt-v2v: error: filesystem was mounted read-only, even though we asked for 
it to be mounted read-write.  This usually means that the filesystem was 
not cleanly unmounted.  Possible causes include trying to convert a guest 
which is running, or using Windows Hibernation or Fast Restart.

Original error message: touch: open: /4n7d407y: Read-only file system

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Verify result 1:
       The virt-v2v error is same with comment11's expect result


Verify scenario2:
Verify the bug with v2v latest rhel7.6 builds
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64

2.1 Convert a win2012 guest from vmware which is forced shut down using external button
# virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64 --password-file /tmp/passwd -o rhv -os 10.66.144.40:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2012-x86_64
[   1.9] Creating an overlay to protect the source from being modified
[   2.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.1] Opening the overlay
[  41.1] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shut down
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Verify result 2:
       The virt-v2v error shows correct with virt-v2v latest rhel7.6 builds although it is different with comment11


Verify scenario3 :
Verify the bug with v2v rhel8 new builds
virt-v2v-1.38.0-5.el8+587+3b2138fb.x86_64
libguestfs-1.38.0-5.el8+587+3b2138fb.x86_64
libguestfs-winsupport-7.2-2.el8+526+412dc3e0.x86_64

3.1 Convert a win2012 guest from vmware which is forced shut down using external button
#  virt-v2v -i vmx esx6.0-win2012-x86_64.vmx -o null
[   1.8] Opening the source -i vmx esx6.0-win2012-x86_64.vmx
[   1.8] Creating an overlay to protect the source from being modified
[   2.3] Initializing the target -o null
[   2.3] Opening the overlay
[  37.1] Inspecting the overlay
virt-v2v: error: unable to mount the disk image for writing. This has 
probably happened because Windows Hibernation or Fast Restart is being used 
in this guest. You have to disable this (in the guest) in order to use 
virt-v2v.

Original error message: mount: mount exited with status 14: The disk 
contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Verify result 3:
       The virt-v2v error shows correct with virt-v2v rhel8 builds although it is different with comment11



Hi rjones,
 
    This bug could be reproduced on v2v fedora26 builds and has fixed in v2v fedora28 builds, I can't reproduce it on rhel7 or rhel8 (it's hard to reproduce on rhel7 or rhel8 as comment13 said), could I move the bug to verified?

Move the bug from ON_QA to VERIFIED according to comment11 ~ comment 14

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3021