Bimal, can you please add steps to reproduce this bug?

(Originally by Eyal Shenitzky)

I was able to reproduce this issue. The issue happens when the cluster is upgraded from 4.1 to 4.2. All the VMs which is having the payload (started as run once with cloud-init) will face this issue if it was UP during the cluster upgrade.

Steps to Reproduce:

1. In a 4.1 cluster, start a VM in RunOnce mode with cloud-init initialization. 

2. Confirm the VM is having "payload" attached using the command virsh -r domblklist vmname.

3. Upgrade the cluster compatibility to 4.2.

4. The VM will be having "Next run" configuration after upgrading the cluster to 4.2.

5. Create a snapshot on the VM. This will work fine.

6. Delete the created snapshot online. It will fail with the error below.

2018-07-10 07:40:11,046-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [bc03c305-9c1e-4e84-8ad8-28656dadfa88] START, DumpXmlsVDSCommand(HostName = 10.74.130.120, Params:{hostId='9b2d8619-7f8a-45d3-820b-d3136804f2e1', vmIds='[4654683b-c315-4ef6-976a-6f504eb06a4e]'}), log id: fc8a7b
2018-07-10 07:40:11,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [bc03c305-9c1e-4e84-8ad8-28656dadfa88] FINISH, DumpXmlsVDSCommand, return: [{vmId=4654683b-c315-4ef6-976a-6f504eb06a4e, devices=[Ljava.util.Map;@2a36f34f, guestDiskMapping={}}], log id: fc8a7b
2018-07-10 07:40:11,062-04 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [bc03c305-9c1e-4e84-8ad8-28656dadfa88] Exception: java.lang.IllegalArgumentException: Invalid UUID string: payload

If I start a new VM as run once with cloud-init in 4.2 compatibility, you would only see below warning and the snapshot delete will complete successfully.

2018-07-10 07:52:13,766-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [263f9885-0248-4d28-be8a-2904da5a11dc] START, DumpXmlsVDSCommand(HostName = 10.74.130.120, Params:{hostId='9b2d8619-7f8a-45d3-820b-d3136804f2e1', vmIds='[d44e901a-6280-439a-b983-1212ff6a9b1f]'}), log id: 611e7c67
2018-07-10 07:52:13,785-04 WARN  [org.ovirt.engine.core.vdsbroker.libvirt.VmDevicesConverter] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [263f9885-0248-4d28-be8a-2904da5a11dc] unmanaged disk with path '/var/run/vdsm/payload/d44e901a-6280-439a-b983-1212ff6a9b1f.572cdbd5eb01e684f008fb9e8bdc8867.img' is ignored
2018-07-10 07:52:13,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [263f9885-0248-4d28-be8a-2904da5a11dc] FINISH, DumpXmlsVDSCommand, return: [{vmId=d44e901a-6280-439a-b983-1212ff6a9b1f, devices=[Ljava.util.Map;@e7c97d, guestDiskMapping={}}], log id: 611e7c67


So the issue is with the live merge on a VMs which are having RunOnce configuration and was UP during the cluster upgrade.

(Originally by Nijin Ashok)

Eyal,

Nijin++ added a reproducer procedure.

- Bimal.

(Originally by Bimal Chollera)

(In reply to Bimal Chollera from comment #11)
> Eyal,
> 
> Nijin++ added a reproducer procedure.
> 
> - Bimal.

Thanks for the detailed information, I will investigate it.

(Originally by Eyal Shenitzky)

Arik, can you please take a look?

I think that it may be related to - https://gerrit.ovirt.org/#/c/92599/

(Originally by Eyal Shenitzky)

(In reply to Eyal Shenitzky from comment #13)
> Arik, can you please take a look?
> 
> I think that it may be related to - https://gerrit.ovirt.org/#/c/92599/

Nah, this bug occurred on 4.2.4, before that patch got in.
As a matter of fact, this patch might actually prevent this issue.
Anyway, posted a patch that fixes incorrect logic in MergeStatusCommand that would surely solve this.
Eyal, could you please verify that?

(Originally by Arik Hadas)

Sure, thanks.

(Originally by Eyal Shenitzky)

WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Tested using 4.2.5.2-0.1 against 4.2.5.1-0.1. 

The following steps:
1) On 4.1 DC and cluster, create a VM with a disk on iSCSI, with cloud init initialization and as run once.
2) While the VM is running, upgraded the cluster and DC to 4.2, a 'next run' snapshot created for the VM
3) Created live snapshot
4) live merged the snapshot


On 4.2.5.1-0.1, live merged failed with:

2018-07-25 17:15:07,886+03 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-5) [41801c7e-1da1-4a55-bc92-369a3a1d12bc] Exception: java.lang.IllegalArgumentEx
ception: Invalid UUID string: payload
        at java.util.UUID.fromString(UUID.java:194) [rt.jar:1.8.0_171]
        at org.ovirt.engine.core.compat.Guid.<init>(Guid.java:67) [compat.jar:]

And on 4.2.5.2-0.1, live merge succeeded.


============================
Used:
rhvm-4.2.5.2-0.1.el7ev.noarch
vdsm-4.20.35-1.el7ev.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2318