Bug 1649091 - LSM's live merge fails with "java.lang.IllegalArgumentException: Invalid UUID string: mapper"
Summary: LSM's live merge fails with "java.lang.IllegalArgumentException: Invalid UUID...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.2.7.1
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.3.1
: ---
Assignee: Eyal Shenitzky
QA Contact: Elad
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-12 22:27 UTC by Elad
Modified: 2019-03-01 10:17 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2019-03-01 10:17:53 UTC
oVirt Team: Storage
Embargoed:
rule-engine: ovirt-4.3+


Attachments (Terms of Use)
logs and screenshot (3.88 MB, application/x-gzip)
2018-11-12 22:27 UTC, Elad
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1652698 0 unspecified CLOSED Live merge fails on the RHV-M Engine with "Invalid UUID string: mapper" followed by exception 2022-03-13 17:07:29 UTC

Internal Links: 1652698

Description Elad 2018-11-12 22:27:17 UTC
Created attachment 1504928 [details]
logs and screenshot

Description of problem:
A failure to live merge the auto generated snapshot that was created during live storage migration.


Version-Release number of selected component (if applicable):
rhvm-4.2.7.4-0.1.el7ev.noarch
vdsm-4.20.43-1.el7ev.x86_64
libvirt-4.5.0-10.el7_6.2.x86_64
qemu-kvm-rhev-2.12.0-18.el7_6.1.x86_64
kernel 3.10.0-957.el7.x86_64 


How reproducible:
Happened after the below scenario, havn't tried it again

Steps to Reproduce:
Happened after the following scenario:

- Created a VM created with a disk on iSCSI (SD1)
- Started the VM
- Live migrated the VM disk to another iSCSI domain (SD2) 
- Attached a direct LUN with virtio-scsi enabled 
- Live migrated the VM disk to another iSCSI domain (SD3)


Actual results:
Failed to delete snapshot 'disk_create_and_attach_disk_1223324525 Auto-generated for Live Storage Migration' for VM 'vm_create_vm_1223264526':


2018-11-12 23:38:59,575+02 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [disks_syncAction_f7107428-c02c-4b5d] Command 'org.ovirt.engine.core.bll.MergeStatusCommand' failed: Invalid UUID string: mapper
2018-11-12 23:38:59,576+02 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [disks_syncAction_f7107428-c02c-4b5d] Exception: java.lang.IllegalArgumentException: Invalid UUID string: mapper
        at java.util.UUID.fromString(UUID.java:194) [rt.jar:1.8.0_181]
        at org.ovirt.engine.core.compat.Guid.<init>(Guid.java:67) [compat.jar:]
        at org.ovirt.engine.core.compat.Guid.createGuidFromStringWithDefault(Guid.java:87) [compat.jar:]
        at org.ovirt.engine.core.compat.Guid.createGuidFromString(Guid.java:76) [compat.jar:]
        at org.ovirt.engine.core.bll.MergeStatusCommand.getVolumeChain(MergeStatusCommand.java:152) [bll.jar:]
        at org.ovirt.engine.core.bll.MergeStatusCommand.attemptResolution(MergeStatusCommand.java:75) [bll.jar:]
        at org.ovirt.engine.core.bll.MergeStatusCommand.executeCommand(MergeStatusCommand.java:59) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1146) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1303) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1952) [bll.jar:]




Expected results:
Live merge should succeed. Not sure if related, but later operations on that disk and VM, like VM migration will fail too (in logs, host2, the destination host for VM migration. The source host was host1):


2018-11-12 23:47:49,389+0200 ERROR (vm/faaab84f) [virt.vm] (vmId='faaab84f-85be-49f4-afae-34a95f49ecb3') The vm start process failed (vm:948)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 877, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2802, in _run
    self._devices = self._make_devices()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2643, in _make_devices
    disk_objs = self._perform_host_local_adjustment()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2716, in _perform_host_local_adjustment
    self._preparePathsForDrives(disk_params)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1022, in _preparePathsForDrives
    drive['path'] = self.cif.prepareVolumePath(drive, self.id)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 417, in prepareVolumePath
    raise vm.VolumeError(drive)
VolumeError: Bad volume specification {'address': {'slot': '0x09', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'serial': 'be0ff91b-a137-4f4d-87b3-1ec5825b29a0', 'index': 0, 'iface': 'vi
rtio', 'apparentsize': '1207959552', 'specParams': {'pinToIoThread': '1'}, 'cache': 'none', 'imageID': 'be0ff91b-a137-4f4d-87b3-1ec5825b29a0', 'truesize': '1207959552', 'type': 'disk', 'domainID': '5901a2ff-e298
-43ea-b183-828c9645eb28', 'reqsize': '0', 'format': 'cow', 'poolID': '33153d4d-f83e-4163-9112-ca7b2333aaa2', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/5901a2ff-e298-43ea-b183-828c9645eb28/images/b
e0ff91b-a137-4f4d-87b3-1ec5825b29a0/23b99597-7edf-4950-891a-5d2b3eb89184', 'propagateErrors': 'off', 'name': 'vda', 'volumeID': '23b99597-7edf-4950-891a-5d2b3eb89184', 'diskType': 'block', 'alias': 'ua-be0ff91b-
a137-4f4d-87b3-1ec5825b29a0', 'discard': False, 'volumeChain': [{'domainID': '5901a2ff-e298-43ea-b183-828c9645eb28', 'leaseOffset': 112197632, 'volumeID': '23b99597-7edf-4950-891a-5d2b3eb89184', 'leasePath': '/d
ev/5901a2ff-e298-43ea-b183-828c9645eb28/leases', 'imageID': 'be0ff91b-a137-4f4d-87b3-1ec5825b29a0', 'path': '/rhev/data-center/mnt/blockSD/5901a2ff-e298-43ea-b183-828c9645eb28/images/be0ff91b-a137-4f4d-87b3-1ec5
825b29a0/23b99597-7edf-4950-891a-5d2b3eb89184'}]}


Additional info:
Logs and screenshot from webadmin with all the operations done for the relevant VM.

Comment 1 Sandro Bonazzola 2019-01-28 09:34:43 UTC
This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

Comment 2 Eyal Shenitzky 2019-02-25 07:41:03 UTC
I couldn't reproduce this issue.

Arik, 
Can you please take a look?
It looks similar to - bug 1598594.

I suspect maybe this patch already fixed it - https://gerrit.ovirt.org/#/c/93993/

Comment 3 Eyal Shenitzky 2019-02-25 08:10:47 UTC
After reverting the code to not include bug 1598594 fix I managed to reproduce the issue.

So patch - https://gerrit.ovirt.org/#/c/93993/ fix this failure as well.

Closing this bug as 'WORKSFORME'.

Comment 4 Elad 2019-02-25 08:15:19 UTC

*** This bug has been marked as a duplicate of bug 1598594 ***

Comment 5 Elad 2019-02-25 08:19:35 UTC
Actually, since bug 1598594 was reported downstream, re-opening this one for upstream

Comment 8 Eyal Shenitzky 2019-02-25 08:35:22 UTC
Wrong link to the patch was added.
This is the patch that fixed the issue - https://gerrit.ovirt.org/#/c/93036/

Comment 9 Elad 2019-02-25 09:10:48 UTC
Verifying based on https://bugzilla.redhat.com/show_bug.cgi?id=1598594#c18

Comment 10 Sandro Bonazzola 2019-03-01 10:17:53 UTC
This bugzilla is included in oVirt 4.3.1 release, published on February 28th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.