Bug 1820282

Summary: Tell virt-v2v where the overlay files must be placed, rather than defaulting always to cachedir
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Ilanit Stein <istein>
Component: virt-v2vAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: liuzi <zili>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.2CC: mxie, mzhan, ptoscano, rjones, tyan, tzheng, virt-maint, xiaodwan, zili
Target Milestone: rcKeywords: Triaged
Target Release: 8.4Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: virt-v2v-1.42.0-7.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-25 06:42:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ilanit Stein 2020-04-02 16:30:02 UTC 
Description of problem:

Background:
There were several runs on VM migrations using CFME-5.11.4.1 and RHV conversion host VM based on UCI VM v2v-conversion-host-appliance-5.11.4.1-1_scratch.x86_64.qcow2 of 10 and 20 VMs from VMware.
At some point the conversion host VMs space got full,
ans when I tried to migrate 20 VMS, it failed in v2v wrapper log on:
"OSError: [Errno 28] No space left on device: '/var/lib/uci/tmp3d7zh8vi.v2v.state'"

$ df -h on the conversion host VMs showed usage for /dev/mapper/VG_UCI-lv_os was 97%:

[root@rhev-node-10 173]# df -h 
Filesystem                     Size  Used Avail Use% Mounted on 
devtmpfs                       7.8G     0  7.8G   0% /dev 
tmpfs                          7.8G   84K  7.8G   1% /dev/shm 
tmpfs                          7.8G  1.2M  7.8G   1% /run 
tmpfs                          7.8G     0  7.8G   0% /sys/fs/cgroup 
/dev/mapper/VG_UCI-lv_os        16G   16G  483M  97% / 
/dev/mapper/VG_UCI-lv_tmp     1014M   44M  971M   5% /tmp 
/dev/vda1                      295M  155M  141M  53% /boot 
/dev/mapper/VG_UCI-lv_var_log 1014M  238M  777M  24% /var/log 
tmpfs                          1.6G     0  1.6G   0% /run/user/0 
shm                             63M     0   63M   0% /var/lib/containers/storage/overlay-containers/f7590c05b43073898428c6f0623bc636159ee83e5ac1548d95470092f195feb1/userdata/shm 
overlay                         16G   16G  483M  97% /var/lib/containers/storag

Here is an example of the leftover files, on the conversion host VM:
drwx------. 2 root root         6 Mar 24 07:41 null.Oxdb1G 
drwx------. 2 root root        35 Mar 24 07:41 v2v.Qh4jHt 
drwx------. 2 root root        34 Mar 24 07:41 v2v.G3rsVJ 
drwx------. 2 root root        36 Mar 24 07:41 v2v.bj9D9Z 
drwx------. 2 root root        56 Mar 24 07:41 v2v.849dtd 
drwx------. 2 root root        30 Mar 24 07:41 rhvupload.ZeMbfX 
drwxr-xr-x. 2 root root        45 Mar 24 07:41 vddk.Gm33Ai 
drwx------. 2 root root         6 Mar 24 07:42 null.1JaZgA 
drwx------. 2 root root        34 Mar 24 07:42 v2v.zvgnOv 
drwx------. 2 root root        35 Mar 24 07:42 v2v.SXstqM 
drwx------. 2 root root        36 Mar 24 07:42 v2v.AKPicf 
drwx------. 2 root root        56 Mar 24 07:42 v2v.YBmB22 
drwx------. 2 root root        30 Mar 24 07:42 rhvupload.kfhMEj 
drwxr-xr-x. 2 root root        45 Mar 24 07:42 vddk.9MM0j1 
drwx------. 2 root root         6 Mar 24 07:42 null.By3gCJ 
drwx------. 2 root root        35 Mar 24 07:42 v2v.hQ5l2l 
drwx------. 2 root root        34 Mar 24 07:42 v2v.FEN7ae 
drwx------. 2 root root        36 Mar 24 07:42 v2v.3iQVj6 
drwx------. 2 root root        56 Mar 24 07:42 v2v.iMgCTt 
drwx------. 2 root root        30 Mar 24 07:42 rhvupload.4neVKB 
drwxr-xr-x. 2 root root        45 Mar 24 07:42 vddk.dbfET0 
drwx------. 2 root root         6 Mar 24 07:42 null.2FRf5z 
drwx------. 2 root root        35 Mar 24 07:42 v2v.ZkaRUA 
drwx------. 2 root root        34 Mar 24 07:42 v2v.MLf9bB 
drwx------. 2 root root        36 Mar 24 07:42 v2v.DPqutB 
drwx------. 2 root root        56 Mar 24 07:42 v2v.7wLBDA 
drwx------. 2 root root        30 Mar 24 07:42 rhvupload.xkWpmA 
drwxr-xr-x. 2 root root        45 Mar 24 07:42 vddk.DzK2dE 
-rw-------. 1 root root 107020288 Mar 24 07:43 v2vovl2cd2ce.qcow2 
-rw-------. 1 root root 107020288 Mar 24 07:43 v2vovl744b53.qcow2 
-rw-------. 1 root root 158466048 Mar 24 07:43 v2vovl9c512d.qcow2 
-rw-------. 1 root root 459866112 Mar 24 07:45 v2vovl908870.qcow2


The space was taken by leftover files from previous migration VMs run, that might have failed or cancelled by CFME.

Here's the solution Richard suggested to avoid such leftovers:

Richard W.M. Jones:
I'm of the opinion (also with the Kubevirt/NFS bug 1814611) that we need to
have a way to tell virt-v2v where the overlay files must be placed,
rather than defaulting always to cachedir.  If we had this then we
could just put the overlay temporaries in a particular subdirectory
which the wrapper can then delete itself, in case virt-v2v is unable
to.  And it also solves the NFS thing since Kubevirt can tell virt-v2v
to put the overlays only into its NFS directory, and the appliance on
a regular filesystem.
Comment 2 Richard W.M. Jones 2021-01-05 09:46:32 UTC 
Fixed upstream in
https://github.com/libguestfs/virt-v2v/commit/717b808bc5cb632778973eb000600e87eaf5c31a

You can now (optionally) set $VIRT_V2V_TMPDIR to put the large temporary
files like the overlays in a separate directory.
Comment 8 liuzi 2021-02-02 13:07:30 UTC 
Verify bug with builds:
virt-v2v-1.42.0-9.module+el8.4.0+9561+069bb9c1.x86_64
qemu-kvm-5.2.0-4.module+el8.4.0+9676+589043b9.x86_64


Steps:
Scenario 1: On standalone v2v conversion server,set normal storage to keep overlay files.
1.1 Set dir to keep overlay files:
# export VIRT_V2V_TMPDIR=/home/test
# echo $VIRT_V2V_TMPDIR
/home/test

1.2 Use virt-v2v to convert guests from Vmware to RHV
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1  esx7.0-win2019-x86_64   -o rhv-upload -os nfs_data -of raw -b ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cluster=Default -oo rhv-direct -ip /home/passwd -oo rhv-verifypeer=true -oo rhv-cafile=/home/ca.pem

1.3 During conversion,check the new dir
# ls /home/test
null.DDB809  v2vovl4f79dd.qcow2

Check the log:
[...]
check_host_free_space: large_tmpdir=/home/test free_space=689785180160
[   2.2] Creating an overlay to protect the source from being modified
qemu-img 'create' '-q' '-f' 'qcow2' '-b' 'nbd:unix:/tmp/v2vnbdkit.4ZU1N9/nbdkit2.sock:exportname=/' '-o' 'compat=1.1,backing_fmt=raw' '/home/test/v2vovld8ee4c.qcow2'
[...]


1.4 After conversion finishing,check the cachedir,the overlay file have been deleted.

Scenario 2: On rhv4.4 node,use nfs storage as cachedir
2.1 Set dir to keep overlay files:
 # export VIRT_V2V_TMPDIR=/home/nfs_data/test

2.2 Use virt-v2v to convert guest 
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win10-x86_64  -o rhv-upload -os nfs_data -of raw -b ovirtmgmt -it vddk -io vddk-libdir=/home/vddk6.7/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cluster=Default -ip /home/passwd -oo rhv-verifypeer=true -oo rhv-cafile=/home/ca.pem  -oo rhv-direct=true 

2.3  During conversion,check the new dir
# ls /home/nfs_data/test
null.7q1PdX  v2vovld94ef6.qcow2

Check the log:
[...]
check_host_free_space: large_tmpdir=/home/nfs_data/test free_space=742164733952
[   2.2] Creating an overlay to protect the source from being modified
qemu-img 'create' '-q' '-f' 'qcow2' '-b' 'nbd:unix:/tmp/v2vnbdkit.BG3ZgE/nbdkit2.sock:exportname=/' '-o' 'compat=1.1,backing_fmt=raw' '/home/nfs_data/test/v2vovl5e88bd.qcow2'
[...]

2.4 After conversion finishing,check the cachedir,the overlay file have been deleted.

Result:
virt-v2v can set the dir to set overlay files.so change the bug from ON_QA to VERIFIED.
Comment 10 errata-xmlrpc 2021-05-25 06:42:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098