Bug 1529460 - Failed to run a VM after adding new non-bootable disk with custom alias
Summary: Failed to run a VM after adding new non-bootable disk with custom alias
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.2.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.2.2
: 4.2.2.2
Assignee: Tal Nisan
QA Contact: Kevin Alon Goldblatt
URL:
Whiteboard:
: 1535907 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-28 08:29 UTC by Eyal Shenitzky
Modified: 2018-03-29 11:12 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-29 11:12:41 UTC
oVirt Team: Storage
Embargoed:
rule-engine: ovirt-4.2+
ylavi: exception+


Attachments (Terms of Use)
engine and vdsm logs (515.34 KB, application/zip)
2017-12-28 08:29 UTC, Eyal Shenitzky
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 88255 0 master MERGED core: Mind guest address when sorting VM disks in Libvirt XML 2020-10-28 17:05:30 UTC
oVirt gerrit 88297 0 ovirt-engine-4.2 MERGED core: Mind guest address when sorting VM disks in Libvirt XML 2020-10-28 17:05:17 UTC

Description Eyal Shenitzky 2017-12-28 08:29:14 UTC
Created attachment 1373107 [details]
engine and vdsm logs

Description of problem:

When having a VM without a bootable disk.
Attaching new disk causes a failure to run the VM.

This scenario will occur when the VM has a disk which already received an address from VDSM (after running the VM) and adding newly non-bootable disk with alias which is previous (in the alphabetical order) to the older VM disk alias.

This occurs because of the fact that when running the VM, the disks are sorting by name (alias) and by the bootable field. By switching the aliases order the VDSM will try to allocate address which is already in use:

2017-12-28 10:07:20,075+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-10) [] EVENT_ID: VM_DOWN_ERROR(119), VM req_vm is down with error. Exit message: unsupported configuration: Found duplicate drive address for disk with target name 'sda' controller='0' bus='0' target='0' unit='0'.

Error on VDSM:
2017-12-28 09:53:47,264+0200 ERROR (vm/b1ddc1c9) [virt.vm] (vmId='b1ddc1c9-a4ab-4bb1-89b7-e8b810770f32') The vm start process failed (vm:915)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 844, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2737, in _run
    dom = self._connection.defineXML(domxml)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3585, in defineXML
    if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
libvirtError: unsupported configuration: Found duplicate drive address for disk with target name 'sda' controller='0' bus='0' target='0' unit='0'


Version-Release number of selected component (if applicable):
4.2.1_master

How reproducible:
100%

Steps to Reproduce:
1.Create a VM with non-bootable disk with disk name 'b'
2.Run the VM
3.Power-off the VM
4.Add new non-bootable disk to the VM with name 'a'
5.Run the VM

Actual results:
The VM failed to run


Expected results:
VM should run

Additional info:
Engine and VDSM logs attached

Comment 1 Michal Skrivanek 2017-12-29 06:40:17 UTC
It's up to you how do you want the disk to be orderd. Do you want it to show up after the first drive? Or shift the original one?

Comment 2 Eyal Shenitzky 2017-12-31 05:45:54 UTC
It doesn't really important, the most important thing is that we will be able to run the VM.
If I am not wrong, I think it will be more 'right' to show the newly added disk after the first drive.

Comment 3 Michal Skrivanek 2018-01-02 11:24:39 UTC
this is really your call, Tal. AFAIK it's sorted alphabetically now, but if there is no reason for that i suppose it can be easily removed.

Comment 4 Tal Nisan 2018-01-02 15:44:57 UTC
I don't think the order really matters, what matters is that the VM will run..

Comment 5 Michal Skrivanek 2018-01-03 12:55:38 UTC
if you do not mind that disks names will shift inside guest (i.e. former /dev/sda will suddenly become /dev/sdb and the newly added disk will be /dev/sda instead) then it's fine with me

Comment 6 Allon Mureinik 2018-01-08 13:24:30 UTC
(In reply to Michal Skrivanek from comment #5)
> if you do not mind that disks names will shift inside guest (i.e. former
> /dev/sda will suddenly become /dev/sdb and the newly added disk will be
> /dev/sda instead) then it's fine with me

I don't think that changing the /dev letters "under the guest's feet" is a good idea. I think that new disks should be add **after** all the disks with the stable addresses.

Comment 7 Michal Skrivanek 2018-01-19 13:17:04 UTC
*** Bug 1535907 has been marked as a duplicate of this bug. ***

Comment 8 Michal Skrivanek 2018-01-19 13:22:35 UTC
Arik, the sorting was there before but I guess there might be a difference since now the anme is generated in writeDisks() and I see it's for managed devces only..so maybe managed/unamanged is wrong
What do you think?

Comment 9 Arik 2018-01-21 16:13:09 UTC
(In reply to Michal Skrivanek from comment #8)
I think it could have happened before 4.2 as well: the engine used to provide VDSM with an index only for the bootable disk, VDSM used to generate indices for the other disks (vm.py#normalizeDrivesIndices) and disk names were determined according to those indices. From what I can tell, VDSM iterated the disks in the order they were received from the engine and the engine iterates the disks in that exact order while assigning disk names now with the engine-XML.

I would expect libvirt/qemu to examine the requested addresses first and then allocate only addresses that are free. But we can apply the solution that Allon suggested in comment 6 if we want a workaround on our side.

Comment 10 Tal Nisan 2018-02-26 15:56:18 UTC
OK, correct steps to reproduce after finally managing to do it:

1. Create a VM with two VirtIO-SCSI disks, one bootable
2. Run the VM (A crucial step)
3. Stop the VM
4. Add another VirtIO-SCSI disk with a name in lower alphabetical order (name it 'a', that should do
5. Attempt to run the VM again

Comment 11 Kevin Alon Goldblatt 2018-03-14 22:07:01 UTC
Verified with the following code:
--------------------------------------
ovirt-engine-4.2.2.2-0.1.el7.noarch
vdsm-4.20.20-1.el7ev.x86_64

Verified with the following scenario:
-------------------------------------
1. Create a VM with two VirtIO-SCSI disks, one bootable
2. Run the VM (A crucial step)
3. Stop the VM
4. Add another VirtIO-SCSI disk with a name in lower alphabetical order (name it 'a', that should do
5. Attempt to run the VM again - VM runs fine

Moving to VERIFIED

Comment 12 Sandro Bonazzola 2018-03-29 11:12:41 UTC
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.