Created attachment 1426616 [details] logs Description of problem: vdsm fail to start with libvirtError: XML error: Multiple 'scsi' controllers with index '0'. Scenario - Run VM on a host and reboot the host, when host is up, try to run this VM on this host, result - failed with: 2018-04-25 15:31:48,147+0300 ERROR (vm/f8abc451) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') The vm start process failed (vm:943) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2869, in _run dom = self._connection.defineXML(domxml) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3676, in defineXML if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self) libvirtError: XML error: Multiple 'scsi' controllers with index '0' 2018-04-25 15:31:48,147+0300 INFO (vm/f8abc451) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') Changed state to Down: XML error: Multiple 'scsi' controllers with index '0' (code=1) (vm:1683) 2018-04-25 15:31:48,165+0300 INFO (vm/f8abc451) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') Stopping connection (guestagent:438) Version-Release number of selected component (if applicable): vdsm-4.20.26-1.el7ev.x86_64 How reproducible: Seems to be 100% with the described scenario above Steps to Reproduce: 1. Run VM, 2. Reboot the host 3. When host is up, try to run the VM again on this host Actual results: Failed with - libvirtError: XML error: Multiple 'scsi' controllers with index '0' Expected results: Should run
dupe of bug 1568399 with logs? Michael, how old is this VM? Whenwws it created/run for the first time? Does the same happen for a newly created VM? The host restart shouldn’t be significant, it should behave the same when you just stop and start VM
(In reply to Michal Skrivanek from comment #1) > dupe of bug 1568399 with logs? > > Michael, how old is this VM? Whenwws it created/run for the first time? Does > the same happen for a newly created VM? The host restart shouldn’t be > significant, it should behave the same when you just stop and start VM Hi Michal, Indeed sounds like dup of BZ 1568399, it failed with the same error. The vm is pretty old, created long time ago, my 4.2 env(this specific one) is running upgrades since the first d/s qe build, so i guess since then. Can't say exactly when first run. Where i can see that? Now this VM can't longer run and fail with this error all the time, on all hosts in the cluster. I have the env available for you guys to take a look if you want. New VMs can start as expected.
(In reply to Michael Burman from comment #2) > (In reply to Michal Skrivanek from comment #1) > > dupe of bug 1568399 with logs? > > > > Michael, how old is this VM? Whenwws it created/run for the first time? Does > > the same happen for a newly created VM? The host restart shouldn’t be > > significant, it should behave the same when you just stop and start VM > > Hi Michal, > Indeed sounds like dup of BZ 1568399, it failed with the same error. > The vm is pretty old, created long time ago, my 4.2 env(this specific one) > is running upgrades since the first d/s qe build, so i guess since then. > Can't say exactly when first run. Where i can see that? > Now this VM can't longer run and fail with this error all the time, on all > hosts in the cluster. I have the env available for you guys to take a look > if you want. > New VMs can start as expected. then it's safe to say it is the same issue as bug 1543833. All VMs created in some 4.2.2 builds will suffer from the same, we only have amanual workaround as mentioned there - edit,switch virtio scsi/virtio-block and back In bug 1568399 we're waiting for logs, but so far I assume it's also the same issue and doesn't need any further change. If you indeed do not see it happening for newer VMs or in VMs upgraded from 4.1 to 4.2.3+ I believe we can close it
(In reply to Michal Skrivanek from comment #3) > (In reply to Michael Burman from comment #2) > > (In reply to Michal Skrivanek from comment #1) > > > dupe of bug 1568399 with logs? > > > > > > Michael, how old is this VM? Whenwws it created/run for the first time? Does > > > the same happen for a newly created VM? The host restart shouldn’t be > > > significant, it should behave the same when you just stop and start VM > > > > Hi Michal, > > Indeed sounds like dup of BZ 1568399, it failed with the same error. > > The vm is pretty old, created long time ago, my 4.2 env(this specific one) > > is running upgrades since the first d/s qe build, so i guess since then. > > Can't say exactly when first run. Where i can see that? > > Now this VM can't longer run and fail with this error all the time, on all > > hosts in the cluster. I have the env available for you guys to take a look > > if you want. > > New VMs can start as expected. > > then it's safe to say it is the same issue as bug 1543833. All VMs created > in some 4.2.2 builds will suffer from the same, we only have amanual > workaround as mentioned there - edit,switch virtio scsi/virtio-block and back > In bug 1568399 we're waiting for logs, but so far I assume it's also the > same issue and doesn't need any further change. > > If you indeed do not see it happening for newer VMs or in VMs upgraded from > 4.1 to 4.2.3+ I believe we can close it Michal, the work around you suggested doesn't work for me. The VM can't start no matter what with the same error.
According to attached logs, I see that the following error occurred: "managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='64de71e2-c745-4758-8fe4-259d854e682f', vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb'}', device='virtio-scsi', type='CONTROLLER', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}'" That implies that the problematic VM run with 2 SCSI controllers due to previous invalid upgrade process. Those 2 SCSI controllers are: 1. device name='scsi', managed=False, plugged=True 2. device name=virtio-scsi, managed=True, Plugged=False As long as you didn't edit this VM, you could run this VM without noticing this issue. But once you edited it, the un-plugged controller became plugged. So now you tried to run this VM with 2 plugged SCSI controllers that have the same index and therefore the VM failed to start: 1. device name='scsi', managed=False, pluggable=True 2. device name=virtio-scsi, managed=True, Pluggable=True As a workaround, for solving it, please try to run this VM via Run Once for only one time. The first Run Once will fail but since the un-managed devices are removed then next runs (regular or Run once) will succeed and this VM will be cleaned from this un-managed controller.
(In reply to Sharon Gratch from comment #5) > According to attached logs, I see that the following error occurred: > "managed non pluggable device was removed unexpectedly from libvirt: > 'VmDevice:{id='VmDeviceId:{deviceId='64de71e2-c745-4758-8fe4-259d854e682f', > vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb'}', device='virtio-scsi', > type='CONTROLLER', specParams='[]', address='', managed='true', > plugged='false', readOnly='false', deviceAlias='', customProperties='[]', > snapshotId='null', logicalName='null', hostDevice='null'}'" > > That implies that the problematic VM run with 2 SCSI controllers due to > previous invalid upgrade process. > Those 2 SCSI controllers are: > 1. device name='scsi', managed=False, plugged=True > 2. device name=virtio-scsi, managed=True, Plugged=False > > As long as you didn't edit this VM, you could run this VM without noticing > this issue. But once you edited it, the un-plugged controller became > plugged. So now you tried to run this VM with 2 plugged SCSI controllers > that have the same index and therefore the VM failed to start: > 1. device name='scsi', managed=False, pluggable=True > 2. device name=virtio-scsi, managed=True, Pluggable=True > > As a workaround, for solving it, please try to run this VM via Run Once for > only one time. The first Run Once will fail but since the un-managed devices > are removed then next runs (regular or Run once) will succeed and this VM > will be cleaned from this un-managed controller. Thank yo Sharon, it helped, VM can run now)
So in reply to Michal Skrivanek from comment #3: > then it's safe to say it is the same issue as bug 1543833. All VMs created > in some 4.2.2 builds will suffer from the same, we only have amanual > workaround as mentioned there - edit,switch virtio scsi/virtio-block and back > In bug 1568399 we're waiting for logs, but so far I assume it's also the > same issue and doesn't need any further change. > > If you indeed do not see it happening for newer VMs or in VMs upgraded from > 4.1 to 4.2.3+ I believe we can close it Michael, can we close this bug?
(In reply to Sharon Gratch from comment #7) > So in reply to Michal Skrivanek from comment #3: > > then it's safe to say it is the same issue as bug 1543833. All VMs created > > in some 4.2.2 builds will suffer from the same, we only have amanual > > workaround as mentioned there - edit,switch virtio scsi/virtio-block and back > > In bug 1568399 we're waiting for logs, but so far I assume it's also the > > same issue and doesn't need any further change. > > > > If you indeed do not see it happening for newer VMs or in VMs upgraded from > > 4.1 to 4.2.3+ I believe we can close it > > Michael, can we close this bug? From my side(qe side) no, it's a real bug, i don't feel ok with this. But if you(dev) think we should/need close it(based on Michal's comment#7) i won't fight about it). If deciding to close, then the resolution of close should be everything except NOTABUG. Thanks,
the invalid upgrade triggers the same problem as in bug 1570349 *** This bug has been marked as a duplicate of bug 1570349 ***