Bug 1583046

Hi Marcin,

Can you please also attach the REST request which you sent (assuming you are using REST/SDK scripts to hot unplug and detach the disks)
It seems like you first try to update the disk and amend it.

Sure - we use SDK not API directly.

We're waiting for disk to UNLOCK based on its status and
later send deactivation reequest. "active" variable is false.

waitForDiskUnlock(diskId, c);
final boolean active = false;
attachmentService.update().diskAttachment(
                    diskAttachment()
                            .id(attachment.id())
                            .active(active))
                    .send().diskAttachment();

Hey Marcin,

The engine log that attached does not contain the relevant log that you mentioned in the bug description, can you please add the relevant log?

Also, can you please add more specific steps to reproduce (e.g if you take a snapshot of the VM, is the VM running while you attach the disk, what was the disk type file/block, ...)

Created attachment 1443073 [details]
Earlier log

(In reply to Eyal Shenitzky from comment #4)
> Hey Marcin,
> 
> The engine log that attached does not contain the relevant log that you
> mentioned in the bug description, can you please add the relevant log?
> 
> Also, can you please add more specific steps to reproduce (e.g if you take a
> snapshot of the VM, is the VM running while you attach the disk, what was
> the disk type file/block, ...)

Can you please add the full steps to reproduce?

1. We take snapshot of a VM (it is running, disks are QCOw2-based)

        Snapshot snapshot = c.getSnapshotsSvc(vmId).add().snapshot(
                snapshot().description(snapshotDescription)
                        .persistMemorystate(false))
                .send().snapshot();
        return waitForSnapshotCreate(snapshot.id(), vmId, timeout);

2. Attach disk snapshots to our Proxy VM

        DiskBuilder diskBuilder = disk().id(diskId);
        if (snapshotId != null) {
            diskBuilder.snapshot(snapshot().id(snapshotId));
        }
        DiskAttachment attachment = c.getDiskAttachmentsSvc(vmId).add().attachment(
                diskAttachment().disk(diskBuilder)
                        .active(true)
                        .bootable(bootable)
                        .interface_(diskInterface))
                .send().attachment();

3. Dump data from proxy

4. We try to deactivate disk

        c.getDiskAttachmentSvc(attachment.id(), vmId)
                .remove().detachOnly(true).send();

5. we receive "Conflict" just when we invoke send() in the above-mentioned code

It doesn't happen always - most probable when multiple attach/detach operations happen at once.

In the log, we can see that the disk was updated and amended.
Did you try to update the disk?
Is there was another thread that tried to do is simultaneously on the same snapshot disks?

No - only one thread is accessing it.

From SDK perspective - we don't have anything related to "amend". Maybe this "update" is a result of our "deactivate" action.

This are the specific steps to reproduce:

1. Create a VM [VM1] with disk
2. Create a snapshot to VM1
3. Create another VM [VM2] with disk
4. Attach the snapshot disk created in step 2 to VM2
5. Run VM2
6. Using REST-API, hot-unplug the snapshot disk attached to VM2

* also failed when trying to edit the snapshot disk that attached to VM2 vi the UI

Marcin, I failed to reproduce this issue via REST-API,
Can you please supply more details on your environment, is this an upgraded environment? did you try any other operation on the VM/disk while that snapshot disk was attached to the VM?

Because the issue doesn't reproduce in the REST-API only via the UI, I restore the old bug description.

Eyal, we are using SDK for Java (4.2.1), and the REST calls are hidden from us. And it doesn't happen always, but I guess if RHV enviornment is simultaneously attaching and detaching disks, we end up quite often in this Conflict error. 

RHV: 4.2.3.8-0.1.el7

"Conflict" message happens sometimes in other moments, like snapshot removal (even in old API and 3.x oVirt/RHV). Simple retry after some time was in general the right way to get rid of the snapshots, but in general it looks like RHV does some internal processing and our calls happen maybe too fast, which causes "conflict". And even when we've been waiting on "Locked" flags (on VMs/Clones/Snapshots etc.) - it still was sometimes reported as "OK" (= not locked), we proceeded with snapshot removals (or like in this bug, with deactivation), but eventually we still end-up with "Conflict".

(In reply to Marcin from comment #13)
> Eyal, we are using SDK for Java (4.2.1), and the REST calls are hidden from
> us. And it doesn't happen always, but I guess if RHV enviornment is
> simultaneously attaching and detaching disks, we end up quite often in this
> Conflict error. 

You could see the REST API calls @ /var/log/httpd/ovirt-requests-log on the Engine.

> 
> RHV: 4.2.3.8-0.1.el7
> 
> "Conflict" message happens sometimes in other moments, like snapshot removal
> (even in old API and 3.x oVirt/RHV). Simple retry after some time was in
> general the right way to get rid of the snapshots, but in general it looks
> like RHV does some internal processing and our calls happen maybe too fast,
> which causes "conflict". And even when we've been waiting on "Locked" flags
> (on VMs/Clones/Snapshots etc.) - it still was sometimes reported as "OK" (=
> not locked), we proceeded with snapshot removals (or like in this bug, with
> deactivation), but eventually we still end-up with "Conflict".

(In reply to Marcin from comment #3)
> Sure - we use SDK not API directly.
> 
> We're waiting for disk to UNLOCK based on its status and
> later send deactivation reequest. "active" variable is false.
> 
> waitForDiskUnlock(diskId, c);
> final boolean active = false;
> attachmentService.update().diskAttachment(
>                     diskAttachment()
>                             .id(attachment.id())
>                             .active(active))
>                     .send().diskAttachment();

In the comment above you really 'deactivate' the snapshot disk.

(In reply to Marcin from comment #7)
> 1. We take snapshot of a VM (it is running, disks are QCOw2-based)
> 
>         Snapshot snapshot = c.getSnapshotsSvc(vmId).add().snapshot(
>                 snapshot().description(snapshotDescription)
>                         .persistMemorystate(false))
>                 .send().snapshot();
>         return waitForSnapshotCreate(snapshot.id(), vmId, timeout);
> 
> 2. Attach disk snapshots to our Proxy VM
> 
>         DiskBuilder diskBuilder = disk().id(diskId);
>         if (snapshotId != null) {
>             diskBuilder.snapshot(snapshot().id(snapshotId));
>         }
>         DiskAttachment attachment =
> c.getDiskAttachmentsSvc(vmId).add().attachment(
>                 diskAttachment().disk(diskBuilder)
>                         .active(true)
>                         .bootable(bootable)
>                         .interface_(diskInterface))
>                 .send().attachment();
> 
> 3. Dump data from proxy
> 
> 4. We try to deactivate disk
> 
>         c.getDiskAttachmentSvc(attachment.id(), vmId)
>                 .remove().detachOnly(true).send();
> 
> 5. we receive "Conflict" just when we invoke send() in the above-mentioned
> code
> 
> It doesn't happen always - most probable when multiple attach/detach
> operations happen at once.

But in step 4 in this comment you used the remove action with detachOnly=true, this operation will cause the snapshot disk to detach from the VM.
Maybe there is a 'conflict' because you are trying to detach the disk twice?

No chance:
1. request to deactivate
2. wait 1 minute and check periodically status (or like in our case - we receive "Conflict" exception in the request inself)
3. repeat several times 1. and 2. - if "Conflict" or minute has gone off.

(In reply to Marcin from comment #16)
> No chance:
> 1. request to deactivate
> 2. wait 1 minute and check periodically status (or like in our case - we
> receive "Conflict" exception in the request inself)
> 3. repeat several times 1. and 2. - if "Conflict" or minute has gone off.

So in which call are you using in the script?

c.getDiskAttachmentSvc(attachment.id(), vmId)
                .remove().detachOnly(true).send();

or - 

waitForDiskUnlock(diskId, c);
final boolean active = false;
attachmentService.update().diskAttachment(
                    diskAttachment()
                            .id(attachment.id())
                            .active(active))
                    .send().diskAttachment();

In the 1st step - more precisely: we first try to deactivate disk and only if it succeeded we try to detach. But we end up quite frequently with the problem with deactivate call - it resultts in exception, so we have to catch it check if it is "Conflict" and try to repeat. Next time we usually are more successful with deactivation and detachment.

OK, so the steps are in the following order?

1) c.getDiskAttachmentSvc(attachment.id(), vmId)
                .remove().detachOnly(true).send();

2) waitForDiskUnlock(diskId, c);
final boolean active = false;
attachmentService.update().diskAttachment(
                    diskAttachment()
                            .id(attachment.id())
                            .active(active))
                    .send().diskAttachment();

If so, the first command detach the disk from the VM but not remove it (the disk is now 'floating' disk and not attached to any VM)
And the second command really tries to deactivate the un-attached disk

The opposite - you can't detach active disk. So we first try to deactivate disk and this throws "Conflict". Then we retry after a while and it succeeds.

And why you are waiting for the disk to unlock?
What it the operation that previous to the disk deactivation?

Just data export. This is just to make sure that disk is in OK status before proceeding with deactivation. Otherwise we could get conflict anyway.

Hey Marcin,

I created a VM with a disk and write a script that does the following steps:

1. deactivate the disk using - 
attachmentService.update().diskAttachment(
                    diskAttachment()
                            .id(attachment.id())
                            .active(active))
                    .send().diskAttachment();

2. detach the disk from the VM using:
c.getDiskAttachmentSvc(attachment.id(), vmId)
                .remove().detachOnly(true).send();

3. attach the disk back to the VM
4. returns this steps 1000 times

I didn't fail on a 'conflict' but, I found this bug - bug 1588461 which you maybe encounter according to the logs you've attached.

So when this bug will fix I will close the bug and if you manage to reproduce it please reopen.

Is it sounds OK?

I also find bug 1589749

Marcin, did you attached the disk to the VM using 'virtio' interface?

Yes (Virtio-SCSI)

This bug depends on Virt bug 1589749 which is not targeted yet, moving target release to 4.3 for now

This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

This bug/RFE is more than 2 years old and it didn't get enough attention so far, and is now flagged as pending close. 
Please review if it is still relevant and provide additional details/justification/patches if you believe it should get more attention for the next oVirt release.

This bug didn't get any attention in a long time, and it's not planned in foreseeable future. oVirt development team has no plans to work on it.
Please feel free to reopen if you have a plan how to contribute this feature/bug fix.