Description of problem: Background: There were several runs on VM migrations using CFME-5.11.4.1 and RHV conversion host VM based on UCI VM v2v-conversion-host-appliance-5.11.4.1-1_scratch.x86_64.qcow2 of 10 and 20 VMs from VMware. At some point the conversion host VMs space got full, ans when I tried to migrate 20 VMS, it failed in v2v wrapper log on: "OSError: [Errno 28] No space left on device: '/var/lib/uci/tmp3d7zh8vi.v2v.state'" $ df -h on the conversion host VMs showed usage for /dev/mapper/VG_UCI-lv_os was 97%: [root@rhev-node-10 173]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 84K 7.8G 1% /dev/shm tmpfs 7.8G 1.2M 7.8G 1% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/VG_UCI-lv_os 16G 16G 483M 97% / /dev/mapper/VG_UCI-lv_tmp 1014M 44M 971M 5% /tmp /dev/vda1 295M 155M 141M 53% /boot /dev/mapper/VG_UCI-lv_var_log 1014M 238M 777M 24% /var/log tmpfs 1.6G 0 1.6G 0% /run/user/0 shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/f7590c05b43073898428c6f0623bc636159ee83e5ac1548d95470092f195feb1/userdata/shm overlay 16G 16G 483M 97% /var/lib/containers/storag Here is an example of the leftover files, on the conversion host VM: drwx------. 2 root root 6 Mar 24 07:41 null.Oxdb1G drwx------. 2 root root 35 Mar 24 07:41 v2v.Qh4jHt drwx------. 2 root root 34 Mar 24 07:41 v2v.G3rsVJ drwx------. 2 root root 36 Mar 24 07:41 v2v.bj9D9Z drwx------. 2 root root 56 Mar 24 07:41 v2v.849dtd drwx------. 2 root root 30 Mar 24 07:41 rhvupload.ZeMbfX drwxr-xr-x. 2 root root 45 Mar 24 07:41 vddk.Gm33Ai drwx------. 2 root root 6 Mar 24 07:42 null.1JaZgA drwx------. 2 root root 34 Mar 24 07:42 v2v.zvgnOv drwx------. 2 root root 35 Mar 24 07:42 v2v.SXstqM drwx------. 2 root root 36 Mar 24 07:42 v2v.AKPicf drwx------. 2 root root 56 Mar 24 07:42 v2v.YBmB22 drwx------. 2 root root 30 Mar 24 07:42 rhvupload.kfhMEj drwxr-xr-x. 2 root root 45 Mar 24 07:42 vddk.9MM0j1 drwx------. 2 root root 6 Mar 24 07:42 null.By3gCJ drwx------. 2 root root 35 Mar 24 07:42 v2v.hQ5l2l drwx------. 2 root root 34 Mar 24 07:42 v2v.FEN7ae drwx------. 2 root root 36 Mar 24 07:42 v2v.3iQVj6 drwx------. 2 root root 56 Mar 24 07:42 v2v.iMgCTt drwx------. 2 root root 30 Mar 24 07:42 rhvupload.4neVKB drwxr-xr-x. 2 root root 45 Mar 24 07:42 vddk.dbfET0 drwx------. 2 root root 6 Mar 24 07:42 null.2FRf5z drwx------. 2 root root 35 Mar 24 07:42 v2v.ZkaRUA drwx------. 2 root root 34 Mar 24 07:42 v2v.MLf9bB drwx------. 2 root root 36 Mar 24 07:42 v2v.DPqutB drwx------. 2 root root 56 Mar 24 07:42 v2v.7wLBDA drwx------. 2 root root 30 Mar 24 07:42 rhvupload.xkWpmA drwxr-xr-x. 2 root root 45 Mar 24 07:42 vddk.DzK2dE -rw-------. 1 root root 107020288 Mar 24 07:43 v2vovl2cd2ce.qcow2 -rw-------. 1 root root 107020288 Mar 24 07:43 v2vovl744b53.qcow2 -rw-------. 1 root root 158466048 Mar 24 07:43 v2vovl9c512d.qcow2 -rw-------. 1 root root 459866112 Mar 24 07:45 v2vovl908870.qcow2 The space was taken by leftover files from previous migration VMs run, that might have failed or cancelled by CFME. Here's the solution Richard suggested to avoid such leftovers: Richard W.M. Jones: I'm of the opinion (also with the Kubevirt/NFS bug 1814611) that we need to have a way to tell virt-v2v where the overlay files must be placed, rather than defaulting always to cachedir. If we had this then we could just put the overlay temporaries in a particular subdirectory which the wrapper can then delete itself, in case virt-v2v is unable to. And it also solves the NFS thing since Kubevirt can tell virt-v2v to put the overlays only into its NFS directory, and the appliance on a regular filesystem.
Fixed upstream in https://github.com/libguestfs/virt-v2v/commit/717b808bc5cb632778973eb000600e87eaf5c31a You can now (optionally) set $VIRT_V2V_TMPDIR to put the large temporary files like the overlays in a separate directory.
Verify bug with builds: virt-v2v-1.42.0-9.module+el8.4.0+9561+069bb9c1.x86_64 qemu-kvm-5.2.0-4.module+el8.4.0+9676+589043b9.x86_64 Steps: Scenario 1: On standalone v2v conversion server,set normal storage to keep overlay files. 1.1 Set dir to keep overlay files: # export VIRT_V2V_TMPDIR=/home/test # echo $VIRT_V2V_TMPDIR /home/test 1.2 Use virt-v2v to convert guests from Vmware to RHV # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 esx7.0-win2019-x86_64 -o rhv-upload -os nfs_data -of raw -b ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cluster=Default -oo rhv-direct -ip /home/passwd -oo rhv-verifypeer=true -oo rhv-cafile=/home/ca.pem 1.3 During conversion,check the new dir # ls /home/test null.DDB809 v2vovl4f79dd.qcow2 Check the log: [...] check_host_free_space: large_tmpdir=/home/test free_space=689785180160 [ 2.2] Creating an overlay to protect the source from being modified qemu-img 'create' '-q' '-f' 'qcow2' '-b' 'nbd:unix:/tmp/v2vnbdkit.4ZU1N9/nbdkit2.sock:exportname=/' '-o' 'compat=1.1,backing_fmt=raw' '/home/test/v2vovld8ee4c.qcow2' [...] 1.4 After conversion finishing,check the cachedir,the overlay file have been deleted. Scenario 2: On rhv4.4 node,use nfs storage as cachedir 2.1 Set dir to keep overlay files: # export VIRT_V2V_TMPDIR=/home/nfs_data/test 2.2 Use virt-v2v to convert guest # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win10-x86_64 -o rhv-upload -os nfs_data -of raw -b ovirtmgmt -it vddk -io vddk-libdir=/home/vddk6.7/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cluster=Default -ip /home/passwd -oo rhv-verifypeer=true -oo rhv-cafile=/home/ca.pem -oo rhv-direct=true 2.3 During conversion,check the new dir # ls /home/nfs_data/test null.7q1PdX v2vovld94ef6.qcow2 Check the log: [...] check_host_free_space: large_tmpdir=/home/nfs_data/test free_space=742164733952 [ 2.2] Creating an overlay to protect the source from being modified qemu-img 'create' '-q' '-f' 'qcow2' '-b' 'nbd:unix:/tmp/v2vnbdkit.BG3ZgE/nbdkit2.sock:exportname=/' '-o' 'compat=1.1,backing_fmt=raw' '/home/nfs_data/test/v2vovl5e88bd.qcow2' [...] 2.4 After conversion finishing,check the cachedir,the overlay file have been deleted. Result: virt-v2v can set the dir to set overlay files.so change the bug from ON_QA to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098