Bug 1764569 - V2V conversion fails when a filesystem has no free inodes
Summary: V2V conversion fails when a filesystem has no free inodes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: virt-v2v
Version: 8.1
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.0
Assignee: Richard W.M. Jones
QA Contact: mxie@redhat.com
URL:
Whiteboard: V2V
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-23 10:59 UTC by mxie@redhat.com
Modified: 2021-11-16 07:57 UTC (History)
5 users (show)

Fixed In Version: virt-v2v-1.42.0-13.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 07:49:54 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
space-check,log (78.83 KB, text/plain)
2019-10-23 10:59 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:4684 0 None None None 2021-11-16 07:50:58 UTC

Description mxie@redhat.com 2019-10-23 10:59:14 UTC
Created attachment 1628393 [details]
space-check,log

Description:
V2V conversion is failed even if /boot of the guest still has 879M


Version:
virt-v2v-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64
libguestfs-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64
libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64
qemu-kvm-4.1.0-13.module+el8.1.0+4313+ef76ec61.x86_64
kernel-4.18.0-147.el8.x86_64

Steps:
1.Prepare a disk and use guestfish to fill some constant to disk (there is no info "No space left on device
" shown after filling)
# guestfish -a space-check.img -i fill-dir /etc 1600000

2.Convert the disk to null by virt-v2v(test only)
# virt-v2v -i disk space-check.img -o null
[   0.0] Opening the source -i disk space-check.img
[   0.0] Creating an overlay to protect the source from being modified
[   0.1] Opening the overlay
[   4.7] Inspecting the overlay
virt-v2v: error: could not write to the guest filesystem: touch: open: 
/g1kzdnzu: No space left on device

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


3.Launch the disk as a guest in virt-manager, then check disk space after powering on
# df -h
Filesystem                         Size  Used Avail Use% Mounted on
devtmpfs                           484M     0  484M   0% /dev
tmpfs                              496M  8.0K  496M   1% /dev/shm
tmpfs                              496M  6.8M  489M   2% /run
tmpfs                              496M     0  496M   0% /sys/fs/cgroup
/dev/mapper/rhel_vm--73--192-root  3.5G  2.2G  1.4G  61% /
/dev/sda1                         1014M  136M  879M  14% /boot
tmpfs                              100M     0  100M   0% /run/user/0


Actual result:
According to v2v man page, the minimum free space for linux guest is 50MB, but the conversion is failed even if /boot of the guest still has 879M

Expected result:
Seems v2v conversion is also affected by limited space of /etc, should add related info to v2v man page

Additional info:

Comment 1 Richard W.M. Jones 2019-10-24 07:52:58 UTC
There's something very odd going on here:

command: mount '/dev/rhel_vm-73-192/root' '/sysroot//'
[    6.415530] XFS (dm-1): Mounting V5 Filesystem
[    6.568322] XFS (dm-1): Ending clean mount
libguestfs: trace: v2v: mount = 0
libguestfs: trace: v2v: touch "/q30vcb6l"
guestfsd: => mount (0x1) took 0.18 secs
guestfsd: <= touch (0x3) request length 56 bytes
guestfsd: error: open: /q30vcb6l: No space left on device
libguestfs: trace: v2v: touch = -1 (error)
virt-v2v: error: could not write to the guest filesystem: touch: open: 
/q30vcb6l: No space left on device

From the 'df' command above we can see that this filesystem clearly
has plenty of space.  I'm really not sure what is going on here.

Comment 2 Richard W.M. Jones 2019-10-24 07:56:45 UTC
Actually I think this might be an inode issue.  Is it possible you could
run this command inside the guest?

# df -i

It may show we have run out of inodes in the filesystem (rather than blocks).

Comment 3 mxie@redhat.com 2019-10-24 10:20:44 UTC
Hi rjones, 

    Check disk space as below

# df -i
Filesystem                         Inodes   IUsed  IFree IUse% Mounted on
devtmpfs                           123856     364 123492    1% /dev
tmpfs                              126839       3 126836    1% /dev/shm
tmpfs                              126839     483 126356    1% /run
tmpfs                              126839      16 126823    1% /sys/fs/cgroup
/dev/mapper/rhel_vm--73--192-root 1833024 1833024      0  100% /
/dev/sda1                          524288     324 523964    1% /boot
tmpfs                              126839       1 126838    1% /run/user/0

Comment 4 Richard W.M. Jones 2019-10-24 10:24:53 UTC
/dev/mapper/rhel_vm--73--192-root 1833024 1833024      0  100% /

The message is confusing but this is the cause of the error.  It has
run out of inodes on the root filesystem.  The fill-dir command causes
many small files to be created, each one using an inode, and if inodes
run out it stops new files from being created even if there is free
space available.

We might improve the error message, but the fact is that if there are
no inodes left then we cannot make v2v changes to the disk.

Comment 5 Pino Toscano 2019-11-27 12:45:44 UTC
Luckly it seems the bits of information needed to check this are already available:

https://github.com/libguestfs/virt-v2v/blob/9fca8345176d88934cf1f351633e5149175839fb/v2v/v2v.ml#L379-410

In particular, the "files", and "ffree" fields of the GuestFS.statvfs struct. I'll try to create a patch for it.

Comment 6 Pino Toscano 2019-11-27 17:00:47 UTC
Sent a simple patch for this:
https://www.redhat.com/archives/libguestfs/2019-November/msg00300.html

Comment 7 Pino Toscano 2019-11-27 17:18:27 UTC
(In reply to Pino Toscano from comment #6)
> Sent a simple patch for this:
> https://www.redhat.com/archives/libguestfs/2019-November/msg00300.html

While this is a good improvement/check IMHO, sadly it will not solve the actual case of this bug, i.e. that there is not even a single inode available.

Comment 10 RHEL Program Management 2021-04-23 07:31:22 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 11 Richard W.M. Jones 2021-04-23 08:32:26 UTC
This bug was closed by an automated process in error, my apologies.  I am reopening it.

Comment 12 Richard W.M. Jones 2021-04-27 15:52:34 UTC
This was actually fixed when we rebased to virt-v2v 1.42.

mxie: Could you ACK this please?

Comment 16 mxie@redhat.com 2021-05-20 05:27:13 UTC
Reproduce the bug with below builds:
virt-v2v-1.42.0-9.module+el8.4.0+9561+069bb9c1.x86_64
libguestfs-1.44.0-2.module+el8.4.0+10146+75917d2f.x86_64
libvirt-client-7.0.0-14.module+el8.4.0+10886+79296686.x86_64
qemu-kvm-5.2.0-16.module+el8.4.0+10806+b7d97207.x86_64

Steps to reproduce:

Scenario1:
1.1 Prepare a rhel8.3 guest whose root partition has about 69M free space on VMware
# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   890M     0  890M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M  8.9M  900M   1% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root  9.8G  9.8G   69M 100% /
/dev/sda1                                 1014M  179M  836M  18% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                Inodes IUsed  IFree IUse% Mounted on
devtmpfs                                  227667   329 227338    1% /dev
tmpfs                                     232560     1 232559    1% /dev/shm
tmpfs                                     232560   585 231975    1% /run
tmpfs                                     232560    17 232543    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root 187688 46894 140794   25% /
/dev/sda1                                 524288   365 523923    1% /boot
tmpfs                                     232560     5 232555    1% /run/user/0

1.2 Power off guest and convert the guest from VMware to rhv by v2v, but conversion is failed because there is no space left on device
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt Auto-esx7.0-rhel8.3-root-partition-limited-inode-available
[   0.5] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-root-partition-limited-inode-available -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.1] Creating an overlay to protect the source from being modified
[   2.8] Opening the overlay
[   7.1] Inspecting the overlay
[  18.8] Checking for sufficient free disk space in the guest
[  18.8] Estimating space required on target for each disk
[  18.8] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: error: libguestfs error: command: dracut: No '/dev/log' or 
'logger' included for syslog logging
pigz: abort: write error on <stdout> (No space left on device)
dracut: dracut: creation of /boot/initramfs-4.18.0-240.el8.x86_64.img 
failed

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Scenario2:
2.1 Prepare a rhel8.3 guest whose root partition has about 71M free space on VMware
# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   890M     0  890M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M  8.9M  900M   1% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root  9.8G  9.8G   71M 100% /
/dev/sda1                                 1014M  179M  836M  18% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                Inodes IUsed  IFree IUse% Mounted on
devtmpfs                                  227667   329 227338    1% /dev
tmpfs                                     232560     1 232559    1% /dev/shm
tmpfs                                     232560   585 231975    1% /run
tmpfs                                     232560    17 232543    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root 192472 46890 145582   25% /
/dev/sda1                                 524288   365 523923    1% /boot
tmpfs                                     232560     5 232555    1% /run/user/0

2.2 Power off guest and convert the guest from VMware to rhv by v2v, conversion can finish without error
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt Auto-esx7.0-rhel8.3-root-partition-limited-inode-available
[   0.5] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-root-partition-limited-inode-available -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.1] Creating an overlay to protect the source from being modified
[   3.0] Opening the overlay
[   7.4] Inspecting the overlay
[  19.0] Checking for sufficient free disk space in the guest
[  19.0] Estimating space required on target for each disk
[  19.0] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  64.4] Mapping filesystem data to avoid copying unused and blank areas
[  65.0] Closing the overlay
[  65.2] Assigning disks to buses
[  65.2] Checking if the guest needs BIOS or UEFI to boot
[  65.2] Initializing the target -o rhv-upload -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  66.6] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.p74jQk/nbdkit4.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 563.3] Creating output metadata
[ 565.2] Finishing off



Verify the bug with below builds:
virt-v2v-1.42.0-12.module+el8.5.0+10976+d40a93e9.x86_64
libguestfs-1.44.0-3.module+el8.5.0+10681+17a9b157.x86_64
libvirt-client-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64
qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64


Steps:
1.Convert the same guest of above scenario1 from VMware to rhv by v2v, get same result with virt-v2v-1.42.0-9
#  virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt Auto-esx7.0-rhel8.3-root-partition-limited-inode-available
[   0.6] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-root-partition-limited-inode-available -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.3] Creating an overlay to protect the source from being modified
[   3.1] Opening the overlay
[   8.2] Inspecting the overlay
[  21.3] Checking for sufficient free disk space in the guest
[  21.3] Estimating space required on target for each disk
[  21.3] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: error: libguestfs error: command: dracut: No '/dev/log' or 
'logger' included for syslog logging
pigz: abort: write error on <stdout> (No space left on device)
dracut: dracut: creation of /boot/initramfs-4.18.0-240.el8.x86_64.img 
failed

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.Convert the same guest of above scenario2 from VMware to rhv by v2v, get same result with virt-v2v-1.42.0-9
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt Auto-esx7.0-rhel8.3-root-partition-limited-inode-available
[   0.7] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-root-partition-limited-inode-available -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.3] Creating an overlay to protect the source from being modified
[   3.0] Opening the overlay
[   8.1] Inspecting the overlay
[  20.7] Checking for sufficient free disk space in the guest
[  20.7] Estimating space required on target for each disk
[  20.7] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  70.7] Mapping filesystem data to avoid copying unused and blank areas
[  71.4] Closing the overlay
[  71.7] Assigning disks to buses
[  71.7] Checking if the guest needs BIOS or UEFI to boot
[  71.7] Initializing the target -o rhv-upload -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  73.1] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.7vfiNh/nbdkit4.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 701.3] Creating output metadata
[ 703.5] Finishing off



Hi Richard,
    I can't find any difference between virt-v2v-1.42.0-9 and virt-v2v-1.42.0-12 for the bug, seems v2v needs at least 71M free space (maybe more than 70MB?) to convert linux guest, but v2v man page said the minimal free space for linux guest is 20MB, so I think we should improve minimal free space size for linux guest
    
#man virt-v2v
.....
   Free space for conversion
       Free space in the guest

       Virt-v2v checks there is sufficient free space in the guest filesystem to perform the conversion.  Currently it
       checks:

       Linux root filesystem
           Minimum free space: 20 MB

       Linux /boot
           Minimum free space: 50 MB

Comment 17 Richard W.M. Jones 2021-05-20 07:34:24 UTC
New 100M limit only applies to Windows guests!

However yes I agree that the Linux guest limit also needs to be increased.
I think I'm going to increase Linux root limit to 100 MB based on your
finding above.  This will need a further patch.

Comment 19 Richard W.M. Jones 2021-06-16 12:30:49 UTC
FWIW this is fixed in RHEL 9 virt-v2v-1.45.1-1.el9
I don't think we have a separate bug for that.

Comment 21 mxie@redhat.com 2021-07-05 07:33:36 UTC
Test the bug with below builds:
virt-v2v-1.42.0-13.module+el8.5.0+11638+3d8f9850.x86_64
libguestfs-1.44.0-3.module+el8.5.0+10681+17a9b157.x86_64
libvirt-client-7.5.0-1.module+el8.5.0+11664+59f87560.x86_64
qemu-kvm-6.0.0-21.module+el8.5.0+11555+e0ab0d09.x86_64


Steps:
1.Check minimum space info for linux guest in virt-v2v man page

#man virt-v2v
......
Free space for conversion
       Free space in the guest

       Virt-v2v checks there is sufficient free space in the guest filesystem to perform the
       conversion.  Currently it checks:

       Linux root filesystem
           Minimum free space: 100 MB

       Linux /boot
           Minimum free space: 50 MB

           This is because we need to build a new initramfs for some Enterprise Linux
           conversions.

Scenario1:
1.1 Prepare a rhel8.3 guest whose root partition has free space less than 100M on VMware

# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   890M     0  890M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M   17M  892M   2% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root  9.8G  9.7G   87M 100% /
/dev/sda1                                 1014M  179M  836M  18% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                Inodes IUsed  IFree IUse% Mounted on
devtmpfs                                  227667   329 227338    1% /dev
tmpfs                                     232560     1 232559    1% /dev/shm
tmpfs                                     232560   590 231970    1% /run
tmpfs                                     232560    17 232543    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root 225928 47681 178247   22% /
/dev/sda1                                 524288   365 523923    1% /boot
tmpfs                                     232560     5 232555    1% /run/user/0


1.2 Power off guest and convert the guest from VMware to rhv by v2v, conversion can finish without error
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt Auto-esx7.0-rhel8.3-root-partition-87M-available
[   0.9] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-root-partition-87M-available -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.6] Creating an overlay to protect the source from being modified
[   3.7] Opening the overlay
[  11.4] Inspecting the overlay
[  34.7] Checking for sufficient free disk space in the guest
[  34.7] Estimating space required on target for each disk
[  34.7] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 127.2] Mapping filesystem data to avoid copying unused and blank areas
[ 128.8] Closing the overlay
[ 129.1] Assigning disks to buses
[ 129.1] Checking if the guest needs BIOS or UEFI to boot
[ 129.1] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 130.6] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.BWbGtG/nbdkit4.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 666.1] Creating output metadata
[ 668.2] Finishing off

1.3 Check guest after v2v conversion, the checkpoints of guest are passed


Scenario2:
2.1 Prepare a linux guest and use guestfish to fill some constant to disk, make sure there is no info "No space left on device" shown after filling and  filesystem has less than 100 inodes 
#guestfish -a esx6.5-rhel8.4-x86_64-sda -i fill-dir /etc 5093000

# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   891M     0  891M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M  8.9M  900M   1% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root  9.8G  4.5G  5.4G  46% /
/dev/sda1                                 1014M  250M  765M  25% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                 Inodes   IUsed  IFree IUse% Mounted on
devtmpfs                                   227959     369 227590    1% /dev
tmpfs                                      232608       1 232607    1% /dev/shm
tmpfs                                      232608     550 232058    1% /run
tmpfs                                      232608      17 232591    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root 5134336 5134259     77  100% /
/dev/sda1                                  524288     311 523977    1% /boot
tmpfs                                      232608       5 232603    1% /run/user/0
                                 232608       5 232603    1% /run/user/0

2.2 Power off guest and convert the disk of guest to null by v2v
# virt-v2v -i disk rhel8.4-guestfish-less-100-inodes -o null
[   0.0] Opening the source -i disk rhel8.4-guestfish-less-100-inodes
[   0.0] Creating an overlay to protect the source from being modified
[   0.2] Opening the overlay
[   6.2] Inspecting the overlay
[  31.7] Checking for sufficient free disk space in the guest
virt-v2v: error: not enough available inodes for conversion on filesystem 
‘/’.  92 inodes available < 100 inodes needed

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Scenario3:
3.1 Prepare a linux guest and use guestfish to fill some constant to disk, make sure there is no info "No space left on device" shown after filling and  filesystem has more than 100 inodes
# guestfish -a esx6.5-rhel8.4-x86_64-sda-1 -i fill-dir /etc 5090000

# df -h 
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   891M     0  891M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M  8.9M  900M   1% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root  9.8G  4.6G  5.3G  47% /
/dev/vda1                                 1014M  250M  765M  25% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                 Inodes   IUsed  IFree IUse% Mounted on
devtmpfs                                   227930     360 227570    1% /dev
tmpfs                                      232579       1 232578    1% /dev/shm
tmpfs                                      232579     540 232039    1% /run
tmpfs                                      232579      17 232562    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root 5134336 5131256   3080  100% /
/dev/vda1                                  524288     311 523977    1% /boot
tmpfs                                      232579       5 232574    1% /run/user/0

3.2 Power off guest and convert the disk of guest to null by v2v
# virt-v2v -i disk rhel8.4-guestfish-more-100-inodes -o null
[   0.0] Opening the source -i disk rhel8.4-guestfish-more-100-inodes
[   0.0] Creating an overlay to protect the source from being modified
[   0.2] Opening the overlay
[   7.1] Inspecting the overlay
[  31.9] Checking for sufficient free disk space in the guest
[  31.9] Estimating space required on target for each disk
[  31.9] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown 
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[1512.2] Mapping filesystem data to avoid copying unused and blank areas
[1514.2] Closing the overlay
[1514.5] Assigning disks to buses
[1514.5] Checking if the guest needs BIOS or UEFI to boot
[1514.5] Initializing the target -o null
[1514.5] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[1541.8] Creating output metadata
[1541.8] Finishing off


Result:
      The minimum free space for Linux root filesystem has increased to 100M in v2v man page and virt-v2v can convert linux guest successfully when root partition of guest has about 100M free space, besides, v2v can report correct error info when guest filesystem has less than 100 inodes, so the bug has been fixed

Comment 23 Richard W.M. Jones 2021-07-19 08:10:50 UTC
The bug has been added to the erratum but for some reason
is in the wrong state.  I will try setting the state to ON_QA
manually.

Comment 24 mxie@redhat.com 2021-07-21 08:49:46 UTC
Verify the bug with below builds:
virt-v2v-1.42.0-14.module+el8.5.0+11846+77888a74.x86_64
libguestfs-1.44.0-3.module+el8.5.0+10681+17a9b157.x86_64
libvirt-libs-7.5.0-1.module+el8.5.0+11664+59f87560.x86_64
qemu-kvm-6.0.0-24.module+el8.5.0+11844+1e3017bd.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64
virtio-win-1.9.17-3.el8_4.noarch


Steps:
1.Check minimum space info for linux guest in virt-v2v man page

#man virt-v2v
......
Free space for conversion
       Free space in the guest

       Virt-v2v checks there is sufficient free space in the guest filesystem to perform the
       conversion.  Currently it checks:

       Linux root filesystem
           Minimum free space: 100 MB

       Linux /boot
           Minimum free space: 50 MB
......

Scenario1:
1.1 Prepare a rhel8.3 guest whose root partition has free space less than 100M on VMware

# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   890M     0  890M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M   17M  892M   2% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root  9.8G  9.7G   87M 100% /
/dev/sda1                                 1014M  179M  836M  18% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                Inodes IUsed  IFree IUse% Mounted on
devtmpfs                                  227667   329 227338    1% /dev
tmpfs                                     232560     1 232559    1% /dev/shm
tmpfs                                     232560   590 231970    1% /run
tmpfs                                     232560    17 232543    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--145-root 225928 47681 178247   22% /
/dev/sda1                                 524288   365 523923    1% /boot
tmpfs                                     232560     5 232555    1% /run/user/0


1.2 Power off guest and convert the guest from VMware to rhv by v2v, conversion can finish without error
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt Auto-esx7.0-rhel8.3-root-partition-87M-available
[   0.9] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-root-partition-87M-available -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.6] Creating an overlay to protect the source from being modified
[   3.7] Opening the overlay
[  11.4] Inspecting the overlay
[  34.7] Checking for sufficient free disk space in the guest
[  34.7] Estimating space required on target for each disk
[  34.7] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 127.2] Mapping filesystem data to avoid copying unused and blank areas
[ 128.8] Closing the overlay
[ 129.1] Assigning disks to buses
[ 129.1] Checking if the guest needs BIOS or UEFI to boot
[ 129.1] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 130.6] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.BWbGtG/nbdkit4.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 666.1] Creating output metadata
[ 668.2] Finishing off

1.3 Check guest after v2v conversion, the checkpoints of guest are passed


Scenario2:
2.1 Prepare a linux guest and use guestfish to fill some constant to disk, make sure there is no info "No space left on device" shown after filling and  filesystem has less than 100 inodes 
2.2.1 #guestfish -a esx6.5-rhel8.4-x86_64-sda -i fill-dir /etc 5093000

2.1.2 log into guest to check the disk space, make sure the inodes of root partition is less than 100
# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   891M     0  891M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M  8.9M  900M   1% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root  9.8G  4.5G  5.4G  46% /
/dev/sda1                                 1014M  250M  765M  25% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                 Inodes   IUsed  IFree IUse% Mounted on
devtmpfs                                   227959     369 227590    1% /dev
tmpfs                                      232608       1 232607    1% /dev/shm
tmpfs                                      232608     550 232058    1% /run
tmpfs                                      232608      17 232591    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root 5134336 5134259     77  100% /
/dev/sda1                                  524288     311 523977    1% /boot
tmpfs                                      232608       5 232603    1% /run/user/0
                                           232608       5 232603    1% /run/user/0

2.1.3 Rename the disk of guest as rhel8.4-guestfish-less-100-inodes

2.2 Power off guest and convert the disk of guest to null by v2v
# virt-v2v -i disk rhel8.4-guestfish-less-100-inodes -o null
[   0.0] Opening the source -i disk rhel8.4-guestfish-less-100-inodes
[   0.0] Creating an overlay to protect the source from being modified
[   0.2] Opening the overlay
[   6.2] Inspecting the overlay
[  31.7] Checking for sufficient free disk space in the guest
virt-v2v: error: not enough available inodes for conversion on filesystem 
‘/’.  92 inodes available < 100 inodes needed

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Scenario3:
3.1 Prepare a linux guest and use guestfish to fill some constant to disk, make sure there is no info "No space left on device" shown after filling and  filesystem has more than 100 inodes

3.1.1 # guestfish -a esx6.5-rhel8.4-x86_64-sda-1 -i fill-dir /etc 5090000

3.1.2 log into guest to check the disk space, make sure the inodes of root partition is more than 100
# df -h 
Filesystem                                 Size  Used Avail Use% Mounted on
devtmpfs                                   891M     0  891M   0% /dev
tmpfs                                      909M     0  909M   0% /dev/shm
tmpfs                                      909M  8.9M  900M   1% /run
tmpfs                                      909M     0  909M   0% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root  9.8G  4.6G  5.3G  47% /
/dev/vda1                                 1014M  250M  765M  25% /boot
tmpfs                                      182M     0  182M   0% /run/user/0

# df -i
Filesystem                                 Inodes   IUsed  IFree IUse% Mounted on
devtmpfs                                   227930     360 227570    1% /dev
tmpfs                                      232579       1 232578    1% /dev/shm
tmpfs                                      232579     540 232039    1% /run
tmpfs                                      232579      17 232562    1% /sys/fs/cgroup
/dev/mapper/rhel_bootp--73--199--244-root 5134336 5131256   3080  100% /
/dev/vda1                                  524288     311 523977    1% /boot
tmpfs                                      232579       5 232574    1% /run/user/0

3.1.3 Rename the disk of guest as rhel8.4-guestfish-more-100-inodes

3.2 Power off guest and convert the disk of guest to null by v2v
# virt-v2v -i disk rhel8.4-guestfish-more-100-inodes -o null
[   0.0] Opening the source -i disk rhel8.4-guestfish-more-100-inodes
[   0.1] Creating an overlay to protect the source from being modified
[   0.2] Opening the overlay
[   7.7] Inspecting the overlay
[  34.1] Checking for sufficient free disk space in the guest
[  34.1] Estimating space required on target for each disk
[  34.1] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown 
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[1583.3] Mapping filesystem data to avoid copying unused and blank areas
[1585.3] Closing the overlay
[1585.7] Assigning disks to buses
[1585.7] Checking if the guest needs BIOS or UEFI to boot
[1585.7] Initializing the target -o null
[1585.7] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[1678.9] Creating output metadata
[1678.9] Finishing off



Result:
      The minimum free space for Linux root filesystem has increased to 100M in v2v man page and virt-v2v can convert linux guest successfully when root partition of guest has about 100M free space, besides, v2v can report correct error info when guest filesystem has less than 100 inodes, move the bug from ON_QA to VERIFIED

Comment 26 errata-xmlrpc 2021-11-16 07:49:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684


Note You need to log in before you can comment on or make changes to this bug.