Created attachment 1504928 [details] logs and screenshot Description of problem: A failure to live merge the auto generated snapshot that was created during live storage migration. Version-Release number of selected component (if applicable): rhvm-4.2.7.4-0.1.el7ev.noarch vdsm-4.20.43-1.el7ev.x86_64 libvirt-4.5.0-10.el7_6.2.x86_64 qemu-kvm-rhev-2.12.0-18.el7_6.1.x86_64 kernel 3.10.0-957.el7.x86_64 How reproducible: Happened after the below scenario, havn't tried it again Steps to Reproduce: Happened after the following scenario: - Created a VM created with a disk on iSCSI (SD1) - Started the VM - Live migrated the VM disk to another iSCSI domain (SD2) - Attached a direct LUN with virtio-scsi enabled - Live migrated the VM disk to another iSCSI domain (SD3) Actual results: Failed to delete snapshot 'disk_create_and_attach_disk_1223324525 Auto-generated for Live Storage Migration' for VM 'vm_create_vm_1223264526': 2018-11-12 23:38:59,575+02 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [disks_syncAction_f7107428-c02c-4b5d] Command 'org.ovirt.engine.core.bll.MergeStatusCommand' failed: Invalid UUID string: mapper 2018-11-12 23:38:59,576+02 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [disks_syncAction_f7107428-c02c-4b5d] Exception: java.lang.IllegalArgumentException: Invalid UUID string: mapper at java.util.UUID.fromString(UUID.java:194) [rt.jar:1.8.0_181] at org.ovirt.engine.core.compat.Guid.<init>(Guid.java:67) [compat.jar:] at org.ovirt.engine.core.compat.Guid.createGuidFromStringWithDefault(Guid.java:87) [compat.jar:] at org.ovirt.engine.core.compat.Guid.createGuidFromString(Guid.java:76) [compat.jar:] at org.ovirt.engine.core.bll.MergeStatusCommand.getVolumeChain(MergeStatusCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.MergeStatusCommand.attemptResolution(MergeStatusCommand.java:75) [bll.jar:] at org.ovirt.engine.core.bll.MergeStatusCommand.executeCommand(MergeStatusCommand.java:59) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1146) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1303) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1952) [bll.jar:] Expected results: Live merge should succeed. Not sure if related, but later operations on that disk and VM, like VM migration will fail too (in logs, host2, the destination host for VM migration. The source host was host1): 2018-11-12 23:47:49,389+0200 ERROR (vm/faaab84f) [virt.vm] (vmId='faaab84f-85be-49f4-afae-34a95f49ecb3') The vm start process failed (vm:948) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 877, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2802, in _run self._devices = self._make_devices() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2643, in _make_devices disk_objs = self._perform_host_local_adjustment() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2716, in _perform_host_local_adjustment self._preparePathsForDrives(disk_params) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1022, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 417, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'address': {'slot': '0x09', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'serial': 'be0ff91b-a137-4f4d-87b3-1ec5825b29a0', 'index': 0, 'iface': 'vi rtio', 'apparentsize': '1207959552', 'specParams': {'pinToIoThread': '1'}, 'cache': 'none', 'imageID': 'be0ff91b-a137-4f4d-87b3-1ec5825b29a0', 'truesize': '1207959552', 'type': 'disk', 'domainID': '5901a2ff-e298 -43ea-b183-828c9645eb28', 'reqsize': '0', 'format': 'cow', 'poolID': '33153d4d-f83e-4163-9112-ca7b2333aaa2', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/5901a2ff-e298-43ea-b183-828c9645eb28/images/b e0ff91b-a137-4f4d-87b3-1ec5825b29a0/23b99597-7edf-4950-891a-5d2b3eb89184', 'propagateErrors': 'off', 'name': 'vda', 'volumeID': '23b99597-7edf-4950-891a-5d2b3eb89184', 'diskType': 'block', 'alias': 'ua-be0ff91b- a137-4f4d-87b3-1ec5825b29a0', 'discard': False, 'volumeChain': [{'domainID': '5901a2ff-e298-43ea-b183-828c9645eb28', 'leaseOffset': 112197632, 'volumeID': '23b99597-7edf-4950-891a-5d2b3eb89184', 'leasePath': '/d ev/5901a2ff-e298-43ea-b183-828c9645eb28/leases', 'imageID': 'be0ff91b-a137-4f4d-87b3-1ec5825b29a0', 'path': '/rhev/data-center/mnt/blockSD/5901a2ff-e298-43ea-b183-828c9645eb28/images/be0ff91b-a137-4f4d-87b3-1ec5 825b29a0/23b99597-7edf-4950-891a-5d2b3eb89184'}]} Additional info: Logs and screenshot from webadmin with all the operations done for the relevant VM.
This bug has not been marked as blocker for oVirt 4.3.0. Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.
I couldn't reproduce this issue. Arik, Can you please take a look? It looks similar to - bug 1598594. I suspect maybe this patch already fixed it - https://gerrit.ovirt.org/#/c/93993/
After reverting the code to not include bug 1598594 fix I managed to reproduce the issue. So patch - https://gerrit.ovirt.org/#/c/93993/ fix this failure as well. Closing this bug as 'WORKSFORME'.
*** This bug has been marked as a duplicate of bug 1598594 ***
Actually, since bug 1598594 was reported downstream, re-opening this one for upstream
Wrong link to the patch was added. This is the patch that fixed the issue - https://gerrit.ovirt.org/#/c/93036/
Verifying based on https://bugzilla.redhat.com/show_bug.cgi?id=1598594#c18
This bugzilla is included in oVirt 4.3.1 release, published on February 28th 2019. Since the problem described in this bug report should be resolved in oVirt 4.3.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.