Bug 1905108 - Cannot hotplug disk reports libvirtError: Requested operation is not valid: Domain already contains a disk with that address
Summary: Cannot hotplug disk reports libvirtError: Requested operation is not valid: D...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.4.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.4.5
: 4.4.5
Assignee: Arik
QA Contact: Qin Yuan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-07 14:31 UTC by Frank DeLorey
Modified: 2021-08-04 22:16 UTC (History)
5 users (show)

Fixed In Version: ovirt-engine-4.4.5
Doc Type: Bug Fix
Doc Text:
Previously, plugging several virtual disks to a running virtual machine over a short time interval could cause a failure to plug some of the disks, and issued an error message: "Domain already contains a disk with that address". In this release, this is avoided by making sure that a disk that is being plugged to a running virtual machine is not assigned with an address that has already been assigned to another disk that was previously plugged to the virtual machine.
Clone Of:
Environment:
Last Closed: 2021-04-14 11:39:56 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 6240381 0 None None None 2021-08-04 22:16:42 UTC
Red Hat Product Errata RHSA-2021:1169 0 None None None 2021-04-14 11:40:29 UTC
oVirt gerrit 112614 0 master MERGED core: cleanup in AttachDiskToVm 2021-02-21 23:27:54 UTC
oVirt gerrit 112615 0 master MERGED core: ensure unique address is assigned in attach-disk 2021-02-21 23:27:54 UTC
oVirt gerrit 112616 0 master MERGED core: cleanup in detach-disk 2021-02-21 23:27:54 UTC
oVirt gerrit 112617 0 master MERGED core: make detach-disk non-transactive 2021-02-21 23:27:54 UTC

Comment 1 Arik 2020-12-10 17:32:26 UTC 
The error is indeed the same as in bz 1855305 (libvirt.libvirtError: Requested operation is not valid: Domain already contains a disk with that address) but the cause is different.

In bz 1855305 we fixed the problem that when a disk was plugged it could have been assigned with an address that a passthrough disk is already assigned with.

Here, however, there's no passthrough disk involved.
Three hot-plug operations that are relevant here:

1. The one that started at 03:01:33,478+03:
2020-12-07 03:01:33,478+03 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1176) [99ec7869-2290-4a53-b411-e255bc0949f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[56bed5f9-a399-4b4a-aca7-0da9f
And finished at 03:01:38,958+03:
2020-12-07 03:01:38,958+03 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1176) [99ec7869-2290-4a53-b411-e255bc0949f9] Lock freed to object 'EngineLock:{exclusiveLocks='[56bed5f9-a399-4b4a-aca7-0da9fe77ebd7=DISK]', sharedLocks=''}'

2. The one that started at 03:01:39,420+03:
2020-12-07 03:01:39,420+03 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1176) [04d598d0-3dfe-43f0-ae38-582a0482ca82] Lock Acquired to object 'EngineLock:{exclusiveLocks='[0635be6d-50cf-4c3f-b2ee-b9dbe
And finished at:
2020-12-07 03:01:44,783+03 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1176) [04d598d0-3dfe-43f0-ae38-582a0482ca82] Lock freed to object 'EngineLock:{exclusiveLocks='[0635be6d-50cf-4c3f-b2ee-b9dbebe8c6cc=DISK]', sharedLocks=''}'

3. The one that started at:
2020-12-07 03:01:46,560+03 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1176) [9a45b07d-bedc-45e3-b4f9-0fa8b899b5c4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[141f768b-3130-41af-ab7f-a3e0f57cc38f=DISK]', sharedLocks=''}'
And finished at:
2020-12-07 03:02:09,727+03 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1176) [9a45b07d-bedc-45e3-b4f9-0fa8b899b5c4] Lock freed to object 'EngineLock:{exclusiveLocks='[141f768b-3130-41af-ab7f-a3e0f57cc38f=DISK]', sharedLocks=''}'

The first succeeded:
2020-12-07 03:01:38,958+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1176) [99ec7869-2290-4a53-b411-e255bc0949f9] EVENT_ID: USER_ATTACH_DISK_TO_VM(2,016), Disk ... was successfully attached to VM ...
And the disk was assigned with address:
<address bus="0" controller="0" unit="5" type="drive" target="0"/>

The other two failed because they tried to take the same address.

The next operation at 2020-12-07 03:02:10,488+03 INFO succeeded - and the disk was assigned with address:
<address bus="0" controller="0" unit="6" type="drive" target="0"/>

So I suspect the following:
The first command allocated the address with unit=5 to its disk.
For some reason, probably something to do with the interaction with postgres, it took longer-than-expected to update that address in the database (probably ~7 sec after the first command was finished).
That's why the other two commands didn't notice that this address was already allocated and tried to allocate it again, which lead to a conflict in libvirt.
Comment 5 Qin Yuan 2021-01-26 09:30:17 UTC 
Verified with:
ovirt-engine-4.4.5.2-0.1.el8ev.noarch

Steps:
1. Create and run a VM(use 8.3 template)
2. Create 5 new disks
3. Attach the 5 disks to the running VM at the same time(the interfaces are all VirtIO-SCSI)

Results:
All 5 disks are successfully attached to the running VM.

engine log:

2021-01-23 10:46:35,600+02 INFO  [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-8) [ab4826f7-250d-423f-8573-d3b0cc502d36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d6203d16-ff6a-41b2-85cf-4ad19efe3a76=DISK]', sharedLocks=''}'
2021-01-23 10:46:35,673+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-8) [ab4826f7-250d-423f-8573-d3b0cc502d36] Disk hot-plug: <?xml version="1.0" encoding="UTF-8"?><hotplug>
  <devices>
    <disk snapshot="no" type="block" device="disk">
      <target dev="sda" bus="scsi"/>
      <alias name="ua-d6203d16-ff6a-41b2-85cf-4ad19efe3a76"/>
      <address bus="0" controller="0" unit="0" type="drive" target="0"/>
2021-01-23 10:46:36,972+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-8) [ab4826f7-250d-423f-8573-d3b0cc502d36] EVENT_ID: USER_ATTACH_DISK_TO_VM(2,016), Disk test_disk1 was successfully attached to VM test by admin@internal-authz.
...
2021-01-23 10:46:37,443+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-8) [8990b1b6-d48a-439e-b9c5-9c4057e55609] Disk hot-plug: <?xml version="1.0" encoding="UTF-8"?><hotplug>
  <devices>
    <disk snapshot="no" type="block" device="disk">
      <target dev="sda" bus="scsi"/>
      <alias name="ua-82c8ab80-5bb0-4dfa-9272-99e000381cf0"/>
      <address bus="0" controller="0" unit="1" type="drive" target="0"/>
...
2021-01-23 10:46:38,707+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-8) [8990b1b6-d48a-439e-b9c5-9c4057e55609] EVENT_ID: USER_ATTACH_DISK_TO_VM(2,016), Disk test_disk2 was successfully attached to VM test by admin@internal-authz.
...
2021-01-23 10:46:39,298+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-8) [7a78dcd7-0250-43f2-bda4-17f58f993abf] Disk hot-plug: <?xml version="1.0" encoding="UTF-8"?><hotplug>
  <devices>
    <disk snapshot="no" type="block" device="disk">
      <target dev="sda" bus="scsi"/>
      <alias name="ua-f79ab334-be86-43c0-a6e8-d1deeb045a5a"/>
      <address bus="0" controller="0" unit="2" type="drive" target="0"/>
...
2021-01-23 10:46:40,517+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-8) [7a78dcd7-0250-43f2-bda4-17f58f993abf] EVENT_ID: USER_ATTACH_DISK_TO_VM(2,016), Disk test_disk3 was successfully attached to VM test by admin@internal-authz.
...
2021-01-23 10:46:41,032+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-8) [b2b44415-62a2-4ff6-ab0c-7421c7be3e8d] Disk hot-plug: <?xml version="1.0" encoding="UTF-8"?><hotplug>
  <devices>
    <disk snapshot="no" type="block" device="disk">
      <target dev="sda" bus="scsi"/>
      <alias name="ua-597829b3-06c0-4681-aa0b-3e4922212156"/>
      <address bus="0" controller="0" unit="3" type="drive" target="0"/>
2021-01-23 10:46:42,307+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-8) [b2b44415-62a2-4ff6-ab0c-7421c7be3e8d] EVENT_ID: USER_ATTACH_DISK_TO_VM(2,016), Disk test_disk4 was successfully attached to VM test by admin@internal-authz.
...
2021-01-23 10:46:42,749+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-12) [e4eb4def-dba8-4389-b26f-226aab1b93a1] Disk hot-plug: <?xml version="1.0" encoding="UTF-8"?><hotplug>
  <devices>
    <disk snapshot="no" type="block" device="disk">
      <target dev="sda" bus="scsi"/>
      <driver name="qemu" io="native" type="raw" error_policy="stop" cache="none"/>
      <alias name="ua-a9c437c2-4529-41f2-aa47-9193509a1709"/>
      <address bus="0" controller="0" unit="4" type="drive" target="0"/>
2021-01-23 10:46:44,061+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-12) [e4eb4def-dba8-4389-b26f-226aab1b93a1] EVENT_ID: USER_ATTACH_DISK_TO_VM(2,016), Disk test_disk5 was successfully attached to VM test by admin@internal-authz.
Comment 12 errata-xmlrpc 2021-04-14 11:39:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) 4.4.z [ovirt-4.4.5] security, bug fix, enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1169
Note You need to log in before you can comment on or make changes to this bug.