Bug 1685032
| Summary: | Virt-v2v openstack conversion will be failed sometimes because of operation timed out | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | mxie <mxie> | ||||||||
| Component: | virt-v2v | Assignee: | Richard W.M. Jones <rjones> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | mxie <mxie> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | 8.0 | CC: | jsuchane, juzhou, knoel, mzhan, ptoscano, rjones, tzheng, xiaodwan | ||||||||
| Target Milestone: | rc | Keywords: | Reopened, Triaged | ||||||||
| Target Release: | 8.1 | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | V2V | ||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2021-04-27 15:55:47 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1771318 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 1540891 [details]
v2v-operation-timeout-reproduce-second-time.log
As 60 seconds is quite a short timeout there doesn't appear to be any harm with increasing this to 5 minutes: https://www.redhat.com/archives/libguestfs/2020-February/msg00002.html Upstream in virt-v2v commit e2ce290f6e366716f857eeaddc1dc680e5608c80. I think for downstream RHEL AV we should go with this patch and see if it appears to solve the problem or if we find further issues. Verify the bug with below builds:
virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64
libguestfs-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64
libvirt-client-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
qemu-kvm-4.2.0-12.module+el8.2.0+5858+afd073bc.x86_64
nbdkit-1.16.2-2.module+el8.2.0+5664+dd92f997.x86_64
python3-openstackclient-4.0.0-0.20191025160014.aa64eb6.el8ost.noarch
python3-openstacksdk-0.36.0-0.20191004153514.8b85e8c.el8ost.noarch
virtio-win-1.9.10-3.el8.noarch
Steps:
1.Set up environment:
1.1 Prepare a guest which have installed python3-openstackclient and virt-v2v on openstack environment
# openstack server list
+--------------------------------------+---------------------------------+---------+-----------------------+-----------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------------------------------+---------+-----------------------+-----------------------+-----------+
| 815aea36-98aa-4fcf-adfa-2cb3e7aa6bb3 | v2v-rhel8.2-conversion-server | ACTIVE | public01=10.73.224.3 | esx6.7-rhel8.2-x86_64 | m1.large |
2.Copy keystone_admin from openstack server to guest "v2v-rhel8.2-conversion-server", source keystone_admin to authenticate with openstack env and enable selinux on guest "v2v-rhel8.2-conversion-server"
#source keystone_admin
#getenforce
Enforcing
#echo $LIBGUESTFS_BACKEND
nothing
3.Convert a guest from vmware to openstack by virt-v2v with vddk
3.1 # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel8.2-x86_64 -o openstack -ip /home/passwd -oo server-id=v2v-rhel8.2-conversion-server
[ 2.2] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel8.2-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[ 3.9] Creating an overlay to protect the source from being modified
[ 7.3] Opening the overlay
[ 59.1] Inspecting the overlay
[ 117.8] Checking for sufficient free disk space in the guest
[ 117.8] Estimating space required on target for each disk
[ 117.8] Converting Red Hat Enterprise Linux 8.2 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1133.3] Mapping filesystem data to avoid copying unused and blank areas
[1151.7] Closing the overlay
[1152.5] Assigning disks to buses
[1152.5] Checking if the guest needs BIOS or UEFI to boot
[1152.5] Initializing the target -o openstack
Failed to set volume read-only access mode flag: Invalid volume: Volume 3491e67d-f541-46e0-87b5-338d357d628d status must be available to update readonly flag, but current status is: creating. (HTTP 400) (Request-ID: req-8a54e749-8e6f-4820-936d-b8ab0375135b)
[1194.6] Copying disk 1/1 to /dev/disk/by-id/virtio-3491e67d-f541-46e0-8 (raw)
(100.00/100%)
[2256.9] Creating output metadata
[2288.4] Finishing off
3.2 Launch the volume as instance after finishing conversion on OSP env and checkpoint of guest are passed except bug1455087
4.Convert a guest from vmware to openstack by virt-v2v without vddk, but need to set $LIBGUESTFS_BACKEND because of bug1804750
4.1 #export LIBGUESTFS_BACKEND=direct
4.2 # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win2019-x86_64 -o openstack -ip /home/passwd -oo server-id=v2v-rhel8.2-conversion-server
[ 2.3] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win2019-x86_64
[ 3.8] Creating an overlay to protect the source from being modified
[ 4.5] Opening the overlay
[ 78.7] Inspecting the overlay
[ 261.1] Checking for sufficient free disk space in the guest
[ 261.1] Estimating space required on target for each disk
[ 261.1] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0
x86_64). virt-v2v looks for this driver in
/usr/share/virtio-win/virtio-win.iso
The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[ 389.9] Mapping filesystem data to avoid copying unused and blank areas
[ 407.9] Closing the overlay
[ 408.7] Assigning disks to buses
[ 408.7] Checking if the guest needs BIOS or UEFI to boot
[ 408.7] Initializing the target -o openstack
Failed to set volume read-only access mode flag: Invalid volume: Volume 6fd95243-e799-4ab9-ac30-ad3155c10d04 status must be available to update readonly flag, but current status is: creating. (HTTP 400) (Request-ID: req-6ede138e-06f0-4be9-bf5f-01ec78017f33)
[ 448.2] Copying disk 1/1 to /dev/disk/by-id/virtio-6fd95243-e799-4ab9-a (raw)
(100.00/100%)
[1708.9] Creating output metadata
[1733.0] Finishing off
4.3 Launch the volume as instance after finishing conversion on OSP env and checkpoint of guest are passed
5.Convert three different guests from VMware to openstack by virt-v2v at the same time. one v2v conversion is failed due to operation timed out
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-win10-x86_64 -o openstack -ip /home/passwd -oo server-id=v2v-rhel8.2-conversion-server -oo guest-id=mxie-guest-id-win10-1231312 -oo verify-server-certificate=true
[ 2.3] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win10-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[ 4.1] Creating an overlay to protect the source from being modified
[ 7.4] Opening the overlay
[ 45.0] Inspecting the overlay
[ 86.5] Checking for sufficient free disk space in the guest
[ 86.5] Estimating space required on target for each disk
[ 86.5] Converting Windows 10 Enterprise to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0
x86_64). virt-v2v looks for this driver in
/usr/share/virtio-win/virtio-win.iso
The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[ 213.4] Mapping filesystem data to avoid copying unused and blank areas
[ 231.6] Closing the overlay
[ 232.7] Assigning disks to buses
[ 232.7] Checking if the guest needs BIOS or UEFI to boot
[ 232.7] Initializing the target -o openstack
virt-v2v: error: waiting for cinder volume
b342abed-ed92-4b3a-b345-2e5ed9976a1c to attach to the conversion appliance:
operation timed out
If reporting bugs, run virt-v2v with debugging enabled and include the
complete output:
virt-v2v -v -x [...]
Failed to delete volume with name or ID 'b342abed-ed92-4b3a-b345-2e5ed9976a1c': Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-32c1123a-fdd4-46c3-b5c1-a40307d314f3)
1 of 1 volumes failed to delete.
6. Install moreuitils to add timestamp to v2v debug log, continue to convert different guests from VMware to openstack by virt-v2v at the same time, get virt-v2v debug log about operation timed out as below after trying 6 times.
6.1 # rpm -q moreutils
moreutils-0.63-1.el8.x86_64
6.2 # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-win10-x86_64 -o openstack -ip /home/passwd -oo server-id=v2v-rhel8.2-conversion-server -oo guest-id=mxie-guest-id-win10-1231312 -oo verify-server-certificate=true -v -x |& ts
......
......
Feb 28 08:48:45 openstack [...] server add volume v2v-rhel8.2-conversion-server b4a51229-4f05-42df-985d-56403b5770c3
Feb 28 08:53:54 virt-v2v: error: waiting for cinder volume
Feb 28 08:53:54 b4a51229-4f05-42df-985d-56403b5770c3 to attach to the conversion appliance:
Feb 28 08:53:54 operation timed out
Feb 28 08:53:54 openstack [...] server remove volume v2v-rhel8.2-conversion-server b4a51229-4f05-42df-985d-56403b5770c3
Feb 28 08:54:02 openstack [...] volume delete b4a51229-4f05-42df-985d-56403b5770c3
Feb 28 08:54:06 Failed to delete volume with name or ID 'b4a51229-4f05-42df-985d-56403b5770c3': Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-cde21544-ba52-44db-b0e4-1f037f0037ce)
Feb 28 08:54:06 1 of 1 volumes failed to delete.
Feb 28 08:54:06 rm -rf '/var/tmp/vddk.a2UoBX'
Feb 28 08:54:06 rm -rf '/var/tmp/null.H1EHjI'
Feb 28 08:54:06 nbdkit: debug: vddk: unload plugin
Feb 28 08:54:06 nbdkit: debug: VDDK call: VixDiskLib_Exit ()
Feb 28 08:54:06 nbdkit: debug: VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.
Feb 28 08:54:07 nbdkit: debug: VixDiskLibVim: VixDiskLibVim_Exit: Clean up.
Hi rjones,
According to result of step6.2, the time of adding volume to openstack has been changed from 60 sec to 500 sec. so bug has been fixed with virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b. But I have a concern about success rate of openstack conversion, below is the spec of v2v conversion appliance "v2v-rhel8.2-conversion-server" on openstack env, if only convert one guest from foreign hypervisor to openstack by v2v, the openstack conversion usually can be finished with 100% success rate, if convert three or more guests from foreign hypervisor to openstack by v2v at the same time, success rate of openstack conversion will be drop to 70~80%, the v2v conversion still will be failed due to operation timed out sometimes, do you think such success rate is acceptable？
Flavor Name: m1.large
RAM: 8GB
VCPUs: 4 VCPU
Disk: 80GB
Created attachment 1666411 [details]
openstack-attach_time_virt-v2v-1.40.2-21.module+el8.2.0+5851.log
Correct wrong statement of comment11, the time of adding volume to openstack has been changed from 60 sec to 300 sec. pls refer to the time in debug log of different v2v version . Test with virt-v2v-1.40.2-1.el7.x86_64 Mar 04 22:10:52 openstack [...] server add volume v2v-mxie 1de8a2d2-4991-480f-b581-f89bd48f9e74 Mar 04 22:10:52 /usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version! Mar 04 22:10:52 RequestsDependencyWarning) Mar 04 22:12:01 virt-v2v: error: waiting for cinder volume Test with virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b Feb 28 08:48:45 openstack [...] server add volume v2v-rhel8.2-conversion-server b4a51229-4f05-42df-985d-56403b5770c3 Feb 28 08:53:54 virt-v2v: error: waiting for cinder volume In answer to the question, no, 70% success rate is not good. I need to ask Nenad to get access to OSP so I can actually test this, but at the moment this bug isn't fixed. Moving this to the backlog because priorities have changed. I think the bug is related to the stability and capabilities of RHOSP environment because I can't reproduce the bug on OSP16.1 environment which is set up on three physical machines. But I still can reproduce the bug on OSP14 which is set up on one physical machine Package version of v2v conversion appliance of OSP16.1 and OSP14: virt-v2v-1.42.0-5.module+el8.3.0+7152+ab3787c3.x86_64 libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64 libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64 qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64 nbdkit-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 Configure info of v2v conversion appliance of OSP16.1 and OSP14: OSP14: 4 VCPU, 8GB RAM, 80G disk OSP14: 4 VCPU, 8GB RAM, 50G disk Steps: 1. Convert 5 guests from foreign hypervisor to OSP16.1 via openstack by v2v at the same time, all v2v conversion can finished successfully 1.1 # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd esx7.0-opensuse42.3-x86_64 -o openstack -oo server-id=rhel8.3-v2v-osp16-server 1.2 # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd esx7.0-rhel8.2-x86_64 -o openstack -oo server-id=rhel8.3-v2v-osp16-server 1.3 # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd esx7.0-win2019-static-ip -o openstack -oo server-id=rhel8.3-v2v-osp16-server 1.4 # virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel5.11-x86_64 -o openstack -oo server-id=rhel8.3-v2v-osp16-server 1.5 # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel6.10-x86_64 -ip /home/passwd -o openstack -oo server-id=rhel8.3-v2v-osp16-server 2. Convert 3 guests from foreign hypervisor to OSP14 via openstack by v2v at the same time, two of the three v2v conversions are failed with virt-v2v: error: waiting for cinder volume xxxxxx to attach to the conversion appliance: operation timed out 2.1 # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd esx7.0-opensuse42.3-x86_64 -o openstack -oo server-id=rhel8.3-v2v-server-osp14 2.2 # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel6.10-x86_64 -ip /home/passwd -o openstack -oo server-id=rhel8.3-v2v-server-osp14 2.3 # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd esx7.0-rhel8.2-x86_64 -o openstack -oo server-id=rhel8.3-v2v-server-osp14 After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. My apologies, this bug was closed by an automated process that we have no control over, and it should not have been. I am reopening it. I think this was probably fixed, as far as it's possible to fix it from virt-v2v, so I'm closing it as current release. See also comment 17. |
Created attachment 1540545 [details] openstack-operation-timeout.log Description of problem: Virt-v2v openstack conversion will be failed sometimes because of operation timed out Version-Release number of selected component (if applicable): virt-v2v-1.40.2-1.el7.x86_64 libguestfs-1.40.2-1.el7.x86_64 libvirt-client-4.5.0-10.el7_6.6.x86_64 qemu-kvm-1.5.3-160.el7.x86_64 nbdkit-1.8.0-1.el7.x86_64 python2-openstackclient-3.16.1-0.20180925042058.3e5a2d2.el7ost.noarch How reproducible: 50% Steps to reproduce: 1.Prepare two guests "rhel7.6-v2v-conversion-server" and "rhel7.6-mxie"on openstack environment, both of guests have installed python2-openstackclient and virt-v2v # openstack server list +--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+ | ff074bf3-51ed-4897-9a81-2ba97a69d81e | rhel7.6-v2v-conversion-server | ACTIVE | public01=10.73.224.5 | esx6.7-rhel7.6-x86_64 | m1.medium | | d5197071-0568-402b-aafa-a0f2c8fe78f4 | rhel7.6-mxie | ACTIVE | public01=10.73.224.10 | esx6.7-rhel7.6-x86_64 | m1.small | +--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+ 2.Copy keystone_admin from openstack server to guest "rhel7.6-v2v-conversion-server", then source keystone_admin to solve openstack authentication #source keystone_admin 3.Convert a guest from vmware to openstack by virt-v2v but set server-id as "rhel7.6-mxie" # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel7.6-x86_64 --password-file /tmp/passwd -o openstack -oo server-id=rhel7.6-mxie -on esx6.7-rhel7.6-o-openstack-performance [ 1.8] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.6-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA [ 3.7] Creating an overlay to protect the source from being modified [ 7.0] Opening the overlay [ 89.4] Inspecting the overlay [ 203.8] Checking for sufficient free disk space in the guest [ 203.8] Estimating space required on target for each disk [ 203.8] Converting Red Hat Enterprise Linux Server 7.6 (Maipo) to run on KVM virt-v2v: This guest has virtio drivers installed. [3766.0] Mapping filesystem data to avoid copying unused and blank areas [3810.1] Closing the overlay [3812.2] Assigning disks to buses [3812.2] Checking if the guest needs BIOS or UEFI to boot [3812.2] Initializing the target -o openstack virt-v2v: error: waiting for cinder volume 8e874c5e-938a-4520-a53f-77ab9548e05f to attach to the conversion appliance: operation timed out If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Actual results: As description Expected results: Virt-v2v openstack conversion can be finished successfully with 100 percentage Additional info: Actually,the problem also can be reproduced by setting server-id as "rhel7.6-v2v-conversion-server" in step3 sometimes, just setting server-id as "rhel7.6-mxie" can reproduce the bug more easily