Created attachment 1480248 [details] evm automation virt-v2v virt-v2v-wrapper logs Description of problem: During a run of 40x100GB VMs migration, 3 VM's failed with the following error with virt-v2v: [ 0.3] Opening the source -i vmx ssh://root.69.75/vmfs/volumes/netapp-v2v-datastore/file:///[] /vmfs/volumes/5af94d9b-8e85fe68-9b02-b8ca3a6382e0/100_VM_TEST_15/100_VM_TEST_15.vmx virt-v2v: error: remote vmx ‘ssh://root.69.75/vmfs/volumes/netapp-v2v-datastore/file:///[] /vmfs/volumes/5af94d9b-8e85fe68-9b02-b8ca3a6382e0/100_VM_TEST_15/100_VM_TEST_15.vmx’ could not be parsed as a URI V2V wrapper invokes virt-v2v with the following command: # /usr/bin/virt-v2v -v -x ssh://root.69.75/vmfs/volumes/netapp-v2v-datastore/file:///[] /vmfs/volumes/5af94d9b-8e85fe68-9b02-b8ca3a6382e0/100_VM_TEST_15/100_VM_TEST_15.vmx -of raw --bridge ovirtmgmt --root first -i vmx -it ssh -o rhv-upload -oc https://b01-h21-r620.rhev.openstack.engineering.redhat.com/ovirt-engine/api -os fc_v2v_datastore_2 -op /tmp/tmpfxRl4d.v2v -oo rhv-cafile=/etc/pki/vdsm/certs/cacert.pem -oo rhv-cluster=sep_cluster -oo rhv-direct -oo rhv-verifypeer=false -oa preallocated --bridge VM Network:ovirtmgmt The source URI that the wrapper sends is invalid. Version-Release number of selected component (if applicable): CFME 5.9.4.5.20180821202240_ced41ae RHV Version 4.2.6.4-0.1 How reproducible: Currently unknown. I only reproduced it once, but it my be reproducible. Steps to Reproduce: 1. Run a plan with 40x100 VMs on our environment. Actual results: Error as described above. Expected results: Should send valid URI. Additional info:
There is a bug in Automation code that builds the URI > ssh://root.69.75/vmfs/volumes/netapp-v2v-datastore/file:///[] \~~~~~~~~~~~~~~~~~~~~/\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~/ \~~~~~~~~/ this part is OK probably a duplicate of following some nonsense > /vmfs/volumes/5af94d9b-8e85fe68-9b02-b8ca3a6382e0/100_VM_TEST_15/ > 100_VM_TEST_15.vmx the rest is is OK I guess that /vmfs/volumes/netapp-v2v-datastore is symlink to /vmfs/volumes/5af94d9b-8e85fe68-9b02-b8ca3a6382e0, so the URI should be either one of: ssh://root.69.75/vmfs/volumes/netapp-v2v-datastore/100_VM_TEST_15/100_VM_TEST_15.vmx ssh://root.69.75/vmfs/volumes/5af94d9b-8e85fe68-9b02-b8ca3a6382e0/100_VM_TEST_15/100_VM_TEST_15.vmx
Hi Fabian, We are currently in the middle of RHV sprint, an not able to work on this. We are planning to do IMS regressions towards end of next week or week after If you need info/reproduction before it i suggest try to get it from Functional QE
@satoe, we never reproduced this in our environment, so we didn't implement any change.
We executed number of times 20 parallel migrations in 5.9.6 with no problem, Was it fixes in 5.9.6 or only in 5.10? We did not tested 5.10 ims yet in RHV QE
Assigned Ilanit, we will plan to do performance regression on 5.10 V2V later this this month. We need to ensure that functional team tested functionally CFME 5.10 with RHV 4.2.8( which is the current RHV version) Adding Need Info on KK
Moving to Verified, since we did not encounter this recently on CFME-5.9.6/5.10.0.29. Will keep on tracking it, as we run coming CFME/RHV versions.
Reopening according to the feedback from bug 1694589. Pasting the relevant information here: ============== CFME/RHV hosts logs can be found here: https://drive.google.com/drive/u/0/folders/11ADeirgAefu97W8_4qpj3cZEdpWmQUm8 2019-03-31_10-46-39_cloudforms_v2v_test-cfme.tar.gz 2019-03-31_14-46-58_b02-h21-r620_v2v_test-conv_host.tar.gz 2019-03-31_14-47-14_b02-h23-r620_v2v_test-conv_host.tar.gz ============== (In reply to Ilanit Stein from comment #8) > CFME/RHV hosts logs can be found here: > https://drive.google.com/drive/u/0/folders/11ADeirgAefu97W8_4qpj3cZEdpWmQUm8 > > 2019-03-31_10-46-39_cloudforms_v2v_test-cfme.tar.gz In this archive, I can see in files/evm.log various lines like: 40377:[----] W, [2019-03-31T06:54:04.543363 #29793:1128f58] WARN -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::RefreshParser.vm_inv_to_hashes) Warning: [path, '[] /vmfs/volumes/5bfe7583-fe7f9906-f066-b8ca3a638940/v2v_migration_vm_03/v2v_migration_vm_03.vmx', is malformed] So the URL is already malformed in CFME, and from there it is sent (invalid) to virt-v2v-wrapper, and then to virt-v2v (which rightfully fails).
*** Bug 1694589 has been marked as a duplicate of this bug. ***
@Ilanit, thanks for the logs, but I would really need access to the appliance to see what's in the VMDB. As I mentioned, none of our VMs has 'file://' in their location, so I'd like to see it on my own to understand the difference between your lab and ours.
@Fabien, ack. I suggest that till our lab, maybe you can try to reproduce with these essential steps (if not tried it already): 1. Use SSH method 2. Migrate 20 VMs in parallel (provider max concurrent=20, host max concurrent=10) Thanks.
Fabien, Our main lab is still down, but we will try to reproduce it at our RHV CI production lab once it is free later this week once it will be free from RHV GA tests
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days