Created attachment 1522236 [details]
Description of problem:
VM migration with latest CloudForms build fail after the Disk conversion is finished.
My observation is Disk conversion finishes 100%, virt-v2v finishes with return code 0, but migration fails to create instance and Migration plan fails. Virt-v2v and virt-v2v-wrapper logs are attached.
Version-Release number of selected component (if applicable):
# Wrapper version
VERSION = "11"
100% ( tried 4 different VM migrations all failed same )
Steps to Reproduce:
1.Configure CFME appliance and Conversion Instance for migration
2.Create Infra mapping and migration plans
3.Execute migration plan and wait for completion
See logs and attached screenshot, migration is failed after 100% conversion of disk
Migration should finish successfully and launch instance.
Logs are attached only for 1 migration, but all 4 migrations that were run, failed similarly.
Created attachment 1522237 [details]
Created attachment 1522238 [details]
I've run a migration with the latest appliance and it failed as described in this BZ.
Environment is composed of:
- OpenStack 13
- CloudForms 188.8.131.52.20190108221820_a0968c8
- rhos-v2v-appliance 14.0-20190116.1, with:
- ovirt-ansible-v2v-conversion-host 1.9.0-3
The migration of this specific VM worked earlier with a conversion host created manually from RHEL 7.6.
The difference between the migrations is that virt-v2v-wrapper.py is unable to identify the volume id in virt-v2v log.
A successful identification will produce the following kind of line in the wrapper log:
2019-01-15 02:17:40,913:DEBUG: Volume at index 1 has id='310718f5-048e-49cb-a65c-513904e068cc' (virt-v2v-wrapper:912)
With the latest appliance, it doesn't appear in the log. And that's caused because the regex fails to catch the line in virt-v2v log. For the failed migration, the line is:
openstack [...] volume set --description hridbxprd001 disk 1/1 converted by virt-v2v --property virt_v2v_version=1.38.2rhel=7,release=12.28.lp.el7ev,libvirt --property virt_v2v_conversion_date=2019/01/22 08:55:51 --property virt_v2v_guest_name=hridbxprd001 --property virt_v2v_disk_index=1/1 --property virt_v2v_guest_id=d5d9f7bf-ffc5-4f82-ac0e-e9d961fc569d ab42e303-4d31-4932-9589-9f3726ebdae7
In virt-v2v-wrapper.py, the regex is the following:
OSP_VOLUME_PROPS = re.compile(
br'openstack .*\'volume\' \'set\'.*'
The other difference I noticed between a conversion host where it was successful and the one based on rhos-v2v-appliance is the version of nbdkit, which is 1.2.6-1.1 on the successful conversion host and 1.2.6 on the failing conversion host.
I checked the virt-v2v log line on the machine that succeeded and the format is different:
openstack '--os-username=admin' '--os-identity-api-version=3' '--os-user-domain-name=default' '--os-auth-url=http://controller.v2v.bos.redhat.com:5000/v3' '--os-project-name=admin' '--os-password=100Root-' 'volume' 'set' '--description' 'hriprdweb001 disk 1/1 converted by virt-v2v' '--property' 'virt_v2v_version=1.38.2rhel=7,release=12.22.lp.el7ev,libvirt' '--property' 'virt_v2v_conversion_date=2019/01/15 07:13:30' '--property' 'virt_v2v_guest_name=hriprdweb001' '--property' 'virt_v2v_disk_index=1/1' '--property' 'virt_v2v_guest_id=ff908ae6-385f-4edd-ba87-b0cd82189313' '310718f5-048e-49cb-a65c-513904e068cc'
You can see that there are single quotes when they are not there on failing VM.
FYI. Found volumes are not getting cleaned up in this case despite it is Failed. While cancel migration issue (BZ1666799) assumption was failed migration should cleanups failed volumes (which it was also doing earlier before this case).
(In reply to Fabien Dupont from comment #6)
> I checked the virt-v2v log line on the machine that succeeded and the format
> is different:
> openstack '--os-username=admin' '--os-identity-api-version=3'
> '--os-project-name=admin' '--os-password=100Root-' 'volume' 'set'
> '--description' 'hriprdweb001 disk 1/1 converted by virt-v2v' '--property'
> 'virt_v2v_version=1.38.2rhel=7,release=12.22.lp.el7ev,libvirt' '--property'
> 'virt_v2v_conversion_date=2019/01/15 07:13:30' '--property'
> 'virt_v2v_guest_name=hriprdweb001' '--property' 'virt_v2v_disk_index=1/1'
> '--property' 'virt_v2v_guest_id=ff908ae6-385f-4edd-ba87-b0cd82189313'
> You can see that there are single quotes when they are not there on failing
I see - this is indeed a side effect of fixing bug 1664310:
A different function is used to print the command, which doesn't quote.
I think the real lesson from this is that we should write some kind of metadata
file which the wrapper can pick up, rather than relying on parsing the debug
logs which are not intended for this and are just going to cause ongoing
problems of this sort in future.
(In reply to Yadnyawalk Tale from comment #7)
> FYI. Found volumes are not getting cleaned up in this case despite it is
> Failed. While cancel migration issue (BZ1666799) assumption was failed
> migration should cleanups failed volumes (which it was also doing earlier
> before this case).
This is another side effect of the current bug.
I have edit virt-v2v-wrapper.py with following fix and pass the migration
Moiviung over to RHV as fix is in virt-v2v-wrapper. Cleared blocker flag. Will ship fix async for RHV 4.2.8 and OSP 14 conversion host image.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.