Bug 1775248

possibly same cause as bug 1770697. Shmuel, once fixed please verify this one is gone too

OK.

Created attachment 1664313 [details]
libvirt logs when simulating failure with debugging

I simulated this issue. The wanrings are not related to the error. The warnings occur because the aliases of the device are not GUID values. This is a different issue.

The error occurs because because the usb device returned from the VDSM via the dumpXmls Command is missing. The usb device is marked as managed, but not pluggable which causes the error in the engine:

2020-02-19 18:07:31,084+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-13) [] VM 'd2ddc775-9a4e-4897-b816-e87634fb6c68' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='c215e54f-3872-4e5f-9554-8111eb1f3469', vmId='d2ddc775-9a4e-4897-b816-e87634fb6c68'}', device='usb', type='CONTROLLER', specParams='[index=0, model=qemu-xhci]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='usb', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}'

Even though the usb device appeared in the domainXml sent from the engine previously:

2020-02-19 18:07:31,026+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (ForkJoinPool-1-worker-13) [] FINISH, DumpXmlsVDSCommand, return: {d2ddc775-9a4e-4897-b816-e87634fb6c68=<domain type='kvm' id='1'>
  <name>VMLinux</name>
  <uuid>d2ddc775-9a4e-4897-b816-e87634fb6c68</uuid>



 <controller type='usb' index='0' model='qemu-xhci' ports='8'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
 </controller>


}, log id: 654af7f4


There are errors in the libvirt that seem to be related to usb monitoring:


2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 : Found property key 'DRIVER' value 'usb' for device with sysname '2-1'

2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 : Found property key 'DRIVER' value '<null>' for device with sysname 'usbmon2'
2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 : Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname 'usbmon2'
2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceType:1191 : Could not determine device type for device with sysfs name 'usbmon2'
2020-02-19 16:06:54.295+0000: 31413: debug : udevAddOneDevice:1381 : Discarding device -1 0x7fd62406cf60 /sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2
2020-02-19 16:06:54.295+0000: 31413: debug : udevProcessDeviceListEntry:1404 : Failed to create node device for udev device '/sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2'


and


2020-02-19 16:07:17.166+0000: 31383: debug : virNodeDeviceDispose:716 : release dev 0x7fd67c0043f0 usb_1_0_1_0


2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceProperty:146 : Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname 'usbmon0'

2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceType:1191 : Could not determine device type for device with sysfs name 'usbmon0'



2020-02-19 16:06:54.343+0000: 31413: debug : udevAddOneDevice:1381 : Discarding device -1 0x7fd6240d6000 /sys/devices/virtual/usbmon/usbmon0

2020-02-19 16:06:54.343+0000: 31413: debug : udevProcessDeviceListEntry:1404 : Failed to create node device for udev device '/sys/devices/virtual/usbmon/usbmon0'

and libvirt appears to be aware that the usb device is registered:

2020-02-19 16:07:05.967+0000: 31383: debug : virConnectGetDomainCapabilities:11393 : conn=0x7fd668004110, ret=<domainCapabilities>
  <path>/usr/libexec/qemu-kvm</path>
  <domain>kvm</domain>
  <machine>pc-i440fx-rhel7.6.0</machine>


     <enum name='bus'>
        <value>ide</value>
        <value>fdc</value>
        <value>scsi</value>
        <value>virtio</value>
        <value>usb</value>
        <value>sata</value>
      </enum>

   <hostdev supported='yes'>
      <enum name='subsysType'>
        <value>usb</value>
        <value>pci</value>
        <value>scsi</value>
      </enum>

(In reply to Steven Rosenberg from comment #3)
> Created attachment 1664313 [details]
> libvirt logs when simulating failure with debugging
> 
> I simulated this issue. The wanrings are not related to the error. The
> warnings occur because the aliases of the device are not GUID values. This
> is a different issue.
> 
> The error occurs because because the usb device returned from the VDSM via
> the dumpXmls Command is missing. The usb device is marked as managed, but
> not pluggable which causes the error in the engine:
> 
> 2020-02-19 18:07:31,084+02 ERROR
> [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring]
> (ForkJoinPool-1-worker-13) [] VM 'd2ddc775-9a4e-4897-b816-e87634fb6c68'
> managed non pluggable device was removed unexpectedly from libvirt:
> 'VmDevice:{id='VmDeviceId:{deviceId='c215e54f-3872-4e5f-9554-8111eb1f3469',
> vmId='d2ddc775-9a4e-4897-b816-e87634fb6c68'}', device='usb',
> type='CONTROLLER', specParams='[index=0, model=qemu-xhci]', address='',
> managed='true', plugged='false', readOnly='false', deviceAlias='usb',
> customProperties='[]', snapshotId='null', logicalName='null',
> hostDevice='null'}'
> 
> Even though the usb device appeared in the domainXml sent from the engine
> previously:
> 
> 2020-02-19 18:07:31,026+02 INFO 
> [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand]
> (ForkJoinPool-1-worker-13) [] FINISH, DumpXmlsVDSCommand, return:
> {d2ddc775-9a4e-4897-b816-e87634fb6c68=<domain type='kvm' id='1'>
>   <name>VMLinux</name>
>   <uuid>d2ddc775-9a4e-4897-b816-e87634fb6c68</uuid>
> 
> 
> 
>  <controller type='usb' index='0' model='qemu-xhci' ports='8'>
>       <alias name='usb'/>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>  </controller>
> 
> 
> }, log id: 654af7f4
> 
> 
> There are errors in the libvirt that seem to be related to usb monitoring:
> 
> 
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'DRIVER' value 'usb' for device with sysname '2-1'
> 
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'DRIVER' value '<null>' for device with sysname 'usbmon2'
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname
> 'usbmon2'
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceType:1191 : Could
> not determine device type for device with sysfs name 'usbmon2'
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevAddOneDevice:1381 :
> Discarding device -1 0x7fd62406cf60
> /sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevProcessDeviceListEntry:1404
> : Failed to create node device for udev device
> '/sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2'
> 
> 
> and
> 
> 
> 2020-02-19 16:07:17.166+0000: 31383: debug : virNodeDeviceDispose:716 :
> release dev 0x7fd67c0043f0 usb_1_0_1_0
> 
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname
> 'usbmon0'
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceType:1191 : Could
> not determine device type for device with sysfs name 'usbmon0'
> 
> 
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevAddOneDevice:1381 :
> Discarding device -1 0x7fd6240d6000 /sys/devices/virtual/usbmon/usbmon0
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevProcessDeviceListEntry:1404
> : Failed to create node device for udev device
> '/sys/devices/virtual/usbmon/usbmon0'
> 
> and libvirt appears to be aware that the usb device is registered:
> 
> 2020-02-19 16:07:05.967+0000: 31383: debug :
> virConnectGetDomainCapabilities:11393 : conn=0x7fd668004110,
> ret=<domainCapabilities>
>   <path>/usr/libexec/qemu-kvm</path>
>   <domain>kvm</domain>
>   <machine>pc-i440fx-rhel7.6.0</machine>
> 
> 
>      <enum name='bus'>
>         <value>ide</value>
>         <value>fdc</value>
>         <value>scsi</value>
>         <value>virtio</value>
>         <value>usb</value>
>         <value>sata</value>
>       </enum>
> 
>    <hostdev supported='yes'>
>       <enum name='subsysType'>
>         <value>usb</value>
>         <value>pci</value>
>         <value>scsi</value>
>       </enum>

Please advise accordingly.

(In reply to Steven Rosenberg from comment #3)
> Created attachment 1664313 [details]
> libvirt logs when simulating failure with debugging
> 

I'm not exactly sure how can I help here.

> I simulated this issue. The wanrings are not related to the error. The
> warnings occur because the aliases of the device are not GUID values. This
> is a different issue.
> 
> The error occurs because because the usb device returned from the VDSM via
> the dumpXmls Command is missing. The usb device is marked as managed, but
> not pluggable which causes the error in the engine:
> 
> 2020-02-19 18:07:31,084+02 ERROR
> [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring]
> (ForkJoinPool-1-worker-13) [] VM 'd2ddc775-9a4e-4897-b816-e87634fb6c68'
> managed non pluggable device was removed unexpectedly from libvirt:
> 'VmDevice:{id='VmDeviceId:{deviceId='c215e54f-3872-4e5f-9554-8111eb1f3469',
> vmId='d2ddc775-9a4e-4897-b816-e87634fb6c68'}', device='usb',
> type='CONTROLLER', specParams='[index=0, model=qemu-xhci]', address='',
> managed='true', plugged='false', readOnly='false', deviceAlias='usb',
> customProperties='[]', snapshotId='null', logicalName='null',
> hostDevice='null'}'
> 
> Even though the usb device appeared in the domainXml sent from the engine
> previously:
> 
> 2020-02-19 18:07:31,026+02 INFO 
> [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand]
> (ForkJoinPool-1-worker-13) [] FINISH, DumpXmlsVDSCommand, return:
> {d2ddc775-9a4e-4897-b816-e87634fb6c68=<domain type='kvm' id='1'>
>   <name>VMLinux</name>
>   <uuid>d2ddc775-9a4e-4897-b816-e87634fb6c68</uuid>
> 
> 
> 
>  <controller type='usb' index='0' model='qemu-xhci' ports='8'>
>       <alias name='usb'/>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>  </controller>

This defines an USB controller for the guest (which itself is NOT an USB device but a PCI device - look at the address type='pci').

> 
> 
> }, log id: 654af7f4
> 
> 
> There are errors in the libvirt that seem to be related to usb monitoring:
> 
> 
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'DRIVER' value 'usb' for device with sysname '2-1'
> 
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'DRIVER' value '<null>' for device with sysname 'usbmon2'
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname
> 'usbmon2'
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceType:1191 : Could
> not determine device type for device with sysfs name 'usbmon2'
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevAddOneDevice:1381 :
> Discarding device -1 0x7fd62406cf60
> /sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2
> 2020-02-19 16:06:54.295+0000: 31413: debug : udevProcessDeviceListEntry:1404
> : Failed to create node device for udev device
> '/sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2'

This is a host telling us that it was unable to get all the details for a USB device that appeared. It has nothing to do with the guest.

> 
> 
> and
> 
> 
> 2020-02-19 16:07:17.166+0000: 31383: debug : virNodeDeviceDispose:716 :
> release dev 0x7fd67c0043f0 usb_1_0_1_0
> 
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceProperty:146 :
> Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname
> 'usbmon0'
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceType:1191 : Could
> not determine device type for device with sysfs name 'usbmon0'
> 
> 
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevAddOneDevice:1381 :
> Discarding device -1 0x7fd6240d6000 /sys/devices/virtual/usbmon/usbmon0
> 
> 2020-02-19 16:06:54.343+0000: 31413: debug : udevProcessDeviceListEntry:1404
> : Failed to create node device for udev device
> '/sys/devices/virtual/usbmon/usbmon0'
> 

Again, this is host part. It says nothing about guest.

> and libvirt appears to be aware that the usb device is registered:
> 
> 2020-02-19 16:07:05.967+0000: 31383: debug :
> virConnectGetDomainCapabilities:11393 : conn=0x7fd668004110,
> ret=<domainCapabilities>
>   <path>/usr/libexec/qemu-kvm</path>
>   <domain>kvm</domain>
>   <machine>pc-i440fx-rhel7.6.0</machine>
> 
> 
>      <enum name='bus'>
>         <value>ide</value>
>         <value>fdc</value>
>         <value>scsi</value>
>         <value>virtio</value>
>         <value>usb</value>
>         <value>sata</value>
>       </enum>
> 
>    <hostdev supported='yes'>
>       <enum name='subsysType'>
>         <value>usb</value>
>         <value>pci</value>
>         <value>scsi</value>
>       </enum>

And lastly, these "usb" strings you find here do not mean what you perhaps think they mean. The domain capabilities XML shows what buses and <hostdev/> types can be used (as in are supported by given qemu binary) when constructing domain XML. The plain fact that "usb" is there has nothing to do with host USB. I can have a file, representing a disk and plug it into guest's USB bus. And there is no host USB involved, is it?

As much as I would like to help, I don't see libvirt misbehaving, sorry.

(In reply to Michal Privoznik from comment #5)
> (In reply to Steven Rosenberg from comment #3)
> > Created attachment 1664313 [details]
> > libvirt logs when simulating failure with debugging
> > 
> 
> I'm not exactly sure how can I help here.
> 
> > I simulated this issue. The wanrings are not related to the error. The
> > warnings occur because the aliases of the device are not GUID values. This
> > is a different issue.
> > 
> > The error occurs because because the usb device returned from the VDSM via
> > the dumpXmls Command is missing. The usb device is marked as managed, but
> > not pluggable which causes the error in the engine:
> > 
> > 2020-02-19 18:07:31,084+02 ERROR
> > [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring]
> > (ForkJoinPool-1-worker-13) [] VM 'd2ddc775-9a4e-4897-b816-e87634fb6c68'
> > managed non pluggable device was removed unexpectedly from libvirt:
> > 'VmDevice:{id='VmDeviceId:{deviceId='c215e54f-3872-4e5f-9554-8111eb1f3469',
> > vmId='d2ddc775-9a4e-4897-b816-e87634fb6c68'}', device='usb',
> > type='CONTROLLER', specParams='[index=0, model=qemu-xhci]', address='',
> > managed='true', plugged='false', readOnly='false', deviceAlias='usb',
> > customProperties='[]', snapshotId='null', logicalName='null',
> > hostDevice='null'}'
> > 
> > Even though the usb device appeared in the domainXml sent from the engine
> > previously:
> > 
> > 2020-02-19 18:07:31,026+02 INFO 
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand]
> > (ForkJoinPool-1-worker-13) [] FINISH, DumpXmlsVDSCommand, return:
> > {d2ddc775-9a4e-4897-b816-e87634fb6c68=<domain type='kvm' id='1'>
> >   <name>VMLinux</name>
> >   <uuid>d2ddc775-9a4e-4897-b816-e87634fb6c68</uuid>
> > 
> > 
> > 
> >  <controller type='usb' index='0' model='qemu-xhci' ports='8'>
> >       <alias name='usb'/>
> >       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > function='0x0'/>
> >  </controller>
> 
> This defines an USB controller for the guest (which itself is NOT an USB
> device but a PCI device - look at the address type='pci').
> 
> > 
> > 
> > }, log id: 654af7f4
> > 
> > 
> > There are errors in the libvirt that seem to be related to usb monitoring:
> > 
> > 
> > 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> > Found property key 'DRIVER' value 'usb' for device with sysname '2-1'
> > 
> > 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> > Found property key 'DRIVER' value '<null>' for device with sysname 'usbmon2'
> > 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceProperty:146 :
> > Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname
> > 'usbmon2'
> > 2020-02-19 16:06:54.295+0000: 31413: debug : udevGetDeviceType:1191 : Could
> > not determine device type for device with sysfs name 'usbmon2'
> > 2020-02-19 16:06:54.295+0000: 31413: debug : udevAddOneDevice:1381 :
> > Discarding device -1 0x7fd62406cf60
> > /sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2
> > 2020-02-19 16:06:54.295+0000: 31413: debug : udevProcessDeviceListEntry:1404
> > : Failed to create node device for udev device
> > '/sys/devices/pci0000:00/0000:00:1d.0/usbmon/usbmon2'
> 
> This is a host telling us that it was unable to get all the details for a
> USB device that appeared. It has nothing to do with the guest.
> 
> > 
> > 
> > and
> > 
> > 
> > 2020-02-19 16:07:17.166+0000: 31383: debug : virNodeDeviceDispose:716 :
> > release dev 0x7fd67c0043f0 usb_1_0_1_0
> > 
> > 
> > 2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceProperty:146 :
> > Found property key 'SUBSYSTEM' value 'usbmon' for device with sysname
> > 'usbmon0'
> > 
> > 2020-02-19 16:06:54.343+0000: 31413: debug : udevGetDeviceType:1191 : Could
> > not determine device type for device with sysfs name 'usbmon0'
> > 
> > 
> > 
> > 2020-02-19 16:06:54.343+0000: 31413: debug : udevAddOneDevice:1381 :
> > Discarding device -1 0x7fd6240d6000 /sys/devices/virtual/usbmon/usbmon0
> > 
> > 2020-02-19 16:06:54.343+0000: 31413: debug : udevProcessDeviceListEntry:1404
> > : Failed to create node device for udev device
> > '/sys/devices/virtual/usbmon/usbmon0'
> > 
> 
> Again, this is host part. It says nothing about guest.
> 
> > and libvirt appears to be aware that the usb device is registered:
> > 
> > 2020-02-19 16:07:05.967+0000: 31383: debug :
> > virConnectGetDomainCapabilities:11393 : conn=0x7fd668004110,
> > ret=<domainCapabilities>
> >   <path>/usr/libexec/qemu-kvm</path>
> >   <domain>kvm</domain>
> >   <machine>pc-i440fx-rhel7.6.0</machine>
> > 
> > 
> >      <enum name='bus'>
> >         <value>ide</value>
> >         <value>fdc</value>
> >         <value>scsi</value>
> >         <value>virtio</value>
> >         <value>usb</value>
> >         <value>sata</value>
> >       </enum>
> > 
> >    <hostdev supported='yes'>
> >       <enum name='subsysType'>
> >         <value>usb</value>
> >         <value>pci</value>
> >         <value>scsi</value>
> >       </enum>
> 
> And lastly, these "usb" strings you find here do not mean what you perhaps
> think they mean. The domain capabilities XML shows what buses and <hostdev/>
> types can be used (as in are supported by given qemu binary) when
> constructing domain XML. The plain fact that "usb" is there has nothing to
> do with host USB. I can have a file, representing a disk and plug it into
> guest's USB bus. And there is no host USB involved, is it?
> 
> As much as I would like to help, I don't see libvirt misbehaving, sorry.

OK, thank you for explaining the errors. But why doesn't the DumpXmls Response message include the usb device which is the source of the error in the engine log?

(In reply to Steven Rosenberg from comment #6)
> OK, thank you for explaining the errors. But why doesn't the DumpXmls
> Response message include the usb device which is the source of the error in
> the engine log?

I'm not sure I understand your question. What dumpxml are you talking about here? Is it domain XML (aka 'virsh dumpxml') or something else?

Is the device you are talking about attached to a domain and it doesn't show in domain's XML? I know nothing about ovirt, so I have no idea what is happening here. And from the libvirt logs, there is only one problem that opening domain log file returned "Device or resource busy", but that is not what we are trying to solve here, is it?

(In reply to Michal Privoznik from comment #7)
> (In reply to Steven Rosenberg from comment #6)
> > OK, thank you for explaining the errors. But why doesn't the DumpXmls
> > Response message include the usb device which is the source of the error in
> > the engine log?
> 
> I'm not sure I understand your question. What dumpxml are you talking about
> here? Is it domain XML (aka 'virsh dumpxml') or something else?
> 
> Is the device you are talking about attached to a domain and it doesn't show
> in domain's XML? I know nothing about ovirt, so I have no idea what is
> happening here. And from the libvirt logs, there is only one problem that
> opening domain log file returned "Device or resource busy", but that is not
> what we are trying to solve here, is it?


Upon further review, this seems to be an issue related to the libvirt / qemu alias handling of the USB controller device [1]

Some details on the engine are in order to synchronize:

The engine does send the domainXml with the usb controller and the dumpxml response (domainxml) does return a usb controller device. But unlike other devices which contain the GUID of the device within the alias section of the domain xml so that the device can be correlated to the device information in the engine's database, the USB Controller Devices' alias is set to usb when the dumpxml response is returned in the device's section:

virsh -r dumpxml VM3

...

    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
    </controller>

...

Unlike say the cdrom device which contains the GUID after the ua- prefix:

...

   <disk type='file' device='cdrom'>
      <driver name='qemu' error_policy='report'/>
      <source startupPolicy='optional'/>
      <target dev='sdc' bus='sata'/>
      <readonly/>
      <alias name='ua-293d5fde-ee99-4ead-b193-0f2f4f73b993'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>

...

This is why the engine does not find the usb controller device in the domain xml received from the dumpxml response and ends up creating a new usb device, gives the error and adds the "new" device to the database. So there are actually two USB devices in the engine database when the VM is running. When the VM shuts down, the extra USB device is removed from the database.

When loading the VM, the alias is not added to the domain xml:

...

 <controller type="usb" model="piix3-uhci" index="0"/>

...

This is because the engine code purposely prevents adding the alias to the control and according to the comment [2]: "USB controllers must not be set with user-aliases (bz #1552127)", referencing [1]

One just needs to read the comment [2] to see that it references [1].

According to [1] it seems the alias was changed from:

<alias name='ua-ble'/>

to 

<alias name='ua-usb2'/>

But it is because the alias does not contain the GUID within the USB Controller Device section that we receive the error, and yes we can see also that the warnings for devices with aliases channel0 through channel2 and pci.1 through pci.8 are also because we do not set the alias within the ua- <GUID> format.

It is also not clear when the ua- prefix was removed from the response message.

Just for clarification, when parsing the alias of the USB controller, the parsing returns a GUID of 0s, unlike the channel and pci values. Which is why there is no warning for usb, only the error.

Therefore, Please clarify why the libvirt is not handling the devices consistently, specifically why the libvirt does not allow us to add an alias to the USB Controller, which if it was the ua- <GUID>, the error would not occur. If the libvirt cannot support the GUID in the alias, maybe the usb error should just be a warning, although the duplicate database entries are not very elegant.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1552127
[2] https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/builder/vminfo/LibvirtVmXmlBuilder.java#L1885-L1888

(In reply to Steven Rosenberg from comment #8)
> (In reply to Michal Privoznik from comment #7)
> > (In reply to Steven Rosenberg from comment #6)
> > > OK, thank you for explaining the errors. But why doesn't the DumpXmls
> > > Response message include the usb device which is the source of the error in
> > > the engine log?
> > 
> > I'm not sure I understand your question. What dumpxml are you talking about
> > here? Is it domain XML (aka 'virsh dumpxml') or something else?
> > 
> > Is the device you are talking about attached to a domain and it doesn't show
> > in domain's XML? I know nothing about ovirt, so I have no idea what is
> > happening here. And from the libvirt logs, there is only one problem that
> > opening domain log file returned "Device or resource busy", but that is not
> > what we are trying to solve here, is it?
> 
> 
> Upon further review, this seems to be an issue related to the libvirt / qemu
> alias handling of the USB controller device [1]
> 
> Some details on the engine are in order to synchronize:
> 
> The engine does send the domainXml with the usb controller and the dumpxml
> response (domainxml) does return a usb controller device. But unlike other
> devices which contain the GUID of the device within the alias section of the
> domain xml so that the device can be correlated to the device information in
> the engine's database, the USB Controller Devices' alias is set to usb when
> the dumpxml response is returned in the device's section:

Well there you go. If you don't pass any user alias, libvirt fills in its default. It can't do anything else, can it?

> 
> virsh -r dumpxml VM3
> 
> ...
> 
>     <controller type='usb' index='0' model='piix3-uhci'>
>       <alias name='usb'/>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x01'
> function='0x0'/>
>     </controller>
> 
> ...
> 

This is expected, if you don't set any alias then libvirt creates its own.

> Unlike say the cdrom device which contains the GUID after the ua- prefix:
> 
> ...
> 
>    <disk type='file' device='cdrom'>
>       <driver name='qemu' error_policy='report'/>
>       <source startupPolicy='optional'/>
>       <target dev='sdc' bus='sata'/>
>       <readonly/>
>       <alias name='ua-293d5fde-ee99-4ead-b193-0f2f4f73b993'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='2'/>
>     </disk>
> 
> ...
> 
> This is why the engine does not find the usb controller device in the domain
> xml received from the dumpxml response and ends up creating a new usb
> device, gives the error and adds the "new" device to the database. So there
> are actually two USB devices in the engine database when the VM is running.
> When the VM shuts down, the extra USB device is removed from the database.
> 
> When loading the VM, the alias is not added to the domain xml:
> 
> ...
> 
>  <controller type="usb" model="piix3-uhci" index="0"/>
> 
> ...
> 
> This is because the engine code purposely prevents adding the alias to the
> control and according to the comment [2]: "USB controllers must not be set
> with user-aliases (bz #1552127)", referencing [1]

This is a clear bug in oVirt code. The transcript of [1] shows that while I was thinking of forbidding but in the end a different approach was taken and user aliases are supported for all kinds of devices (as can be seen from QE verification, e.g. bug 1552127#20).

> 
> One just needs to read the comment [2] to see that it references [1].
> 
> According to [1] it seems the alias was changed from:
> 
> <alias name='ua-ble'/>
> 
> to 
> 
> <alias name='ua-usb2'/>
> 
> But it is because the alias does not contain the GUID within the USB
> Controller Device section that we receive the error, and yes we can see also
> that the warnings for devices with aliases channel0 through channel2 and
> pci.1 through pci.8 are also because we do not set the alias within the ua-
> <GUID> format.
> 
> It is also not clear when the ua- prefix was removed from the response
> message.

I'm not sure what you mean by removed. IIUC, the alias was never specified, so it couldn't have been removed.

> 
> Just for clarification, when parsing the alias of the USB controller, the
> parsing returns a GUID of 0s, unlike the channel and pci values. Which is
> why there is no warning for usb, only the error.
> 
> Therefore, Please clarify why the libvirt is not handling the devices
> consistently, specifically why the libvirt does not allow us to add an alias
> to the USB Controller, which if it was the ua- <GUID>, the error would not
> occur. If the libvirt cannot support the GUID in the alias, maybe the usb
> error should just be a warning, although the duplicate database entries are
> not very elegant.
> 

Libvirt does allow consistent handling of devices. User alias can be specified for USB controllers too.

Hope this helps.

(In reply to Michal Privoznik from comment #9)
> (In reply to Steven Rosenberg from comment #8)
> > (In reply to Michal Privoznik from comment #7)
> > > (In reply to Steven Rosenberg from comment #6)
> > > > OK, thank you for explaining the errors. But why doesn't the DumpXmls
> > > > Response message include the usb device which is the source of the error in
> > > > the engine log?
> > > 
> > > I'm not sure I understand your question. What dumpxml are you talking about
> > > here? Is it domain XML (aka 'virsh dumpxml') or something else?
> > > 
> > > Is the device you are talking about attached to a domain and it doesn't show
> > > in domain's XML? I know nothing about ovirt, so I have no idea what is
> > > happening here. And from the libvirt logs, there is only one problem that
> > > opening domain log file returned "Device or resource busy", but that is not
> > > what we are trying to solve here, is it?
> > 
> > 
> > Upon further review, this seems to be an issue related to the libvirt / qemu
> > alias handling of the USB controller device [1]
> > 
> > Some details on the engine are in order to synchronize:
> > 
> > The engine does send the domainXml with the usb controller and the dumpxml
> > response (domainxml) does return a usb controller device. But unlike other
> > devices which contain the GUID of the device within the alias section of the
> > domain xml so that the device can be correlated to the device information in
> > the engine's database, the USB Controller Devices' alias is set to usb when
> > the dumpxml response is returned in the device's section:
> 
> Well there you go. If you don't pass any user alias, libvirt fills in its
> default. It can't do anything else, can it?
> 
> > 
> > virsh -r dumpxml VM3
> > 
> > ...
> > 
> >     <controller type='usb' index='0' model='piix3-uhci'>
> >       <alias name='usb'/>
> >       <address type='pci' domain='0x0000' bus='0x04' slot='0x01'
> > function='0x0'/>
> >     </controller>
> > 
> > ...
> > 
> 
> This is expected, if you don't set any alias then libvirt creates its own.
> 
> > Unlike say the cdrom device which contains the GUID after the ua- prefix:
> > 
> > ...
> > 
> >    <disk type='file' device='cdrom'>
> >       <driver name='qemu' error_policy='report'/>
> >       <source startupPolicy='optional'/>
> >       <target dev='sdc' bus='sata'/>
> >       <readonly/>
> >       <alias name='ua-293d5fde-ee99-4ead-b193-0f2f4f73b993'/>
> >       <address type='drive' controller='0' bus='0' target='0' unit='2'/>
> >     </disk>
> > 
> > ...
> > 
> > This is why the engine does not find the usb controller device in the domain
> > xml received from the dumpxml response and ends up creating a new usb
> > device, gives the error and adds the "new" device to the database. So there
> > are actually two USB devices in the engine database when the VM is running.
> > When the VM shuts down, the extra USB device is removed from the database.
> > 
> > When loading the VM, the alias is not added to the domain xml:
> > 
> > ...
> > 
> >  <controller type="usb" model="piix3-uhci" index="0"/>
> > 
> > ...
> > 
> > This is because the engine code purposely prevents adding the alias to the
> > control and according to the comment [2]: "USB controllers must not be set
> > with user-aliases (bz #1552127)", referencing [1]
> 
> This is a clear bug in oVirt code. The transcript of [1] shows that while I
> was thinking of forbidding but in the end a different approach was taken and
> user aliases are supported for all kinds of devices (as can be seen from QE
> verification, e.g. bug 1552127#20).
> 
> > 
> > One just needs to read the comment [2] to see that it references [1].
> > 
> > According to [1] it seems the alias was changed from:
> > 
> > <alias name='ua-ble'/>
> > 
> > to 
> > 
> > <alias name='ua-usb2'/>
> > 
> > But it is because the alias does not contain the GUID within the USB
> > Controller Device section that we receive the error, and yes we can see also
> > that the warnings for devices with aliases channel0 through channel2 and
> > pci.1 through pci.8 are also because we do not set the alias within the ua-
> > <GUID> format.
> > 
> > It is also not clear when the ua- prefix was removed from the response
> > message.
> 
> I'm not sure what you mean by removed. IIUC, the alias was never specified,
> so it couldn't have been removed.
> 
> > 
> > Just for clarification, when parsing the alias of the USB controller, the
> > parsing returns a GUID of 0s, unlike the channel and pci values. Which is
> > why there is no warning for usb, only the error.
> > 
> > Therefore, Please clarify why the libvirt is not handling the devices
> > consistently, specifically why the libvirt does not allow us to add an alias
> > to the USB Controller, which if it was the ua- <GUID>, the error would not
> > occur. If the libvirt cannot support the GUID in the alias, maybe the usb
> > error should just be a warning, although the duplicate database entries are
> > not very elegant.
> > 
> 
> Libvirt does allow consistent handling of devices. User alias can be
> specified for USB controllers too.
> 
> Hope this helps.

Yes it does help. Thank you for the response.

I see that the warnings are generated by non managed devices. From the warnings and my investigation, I see three types of devices that are not managed that are received from the dumpxml response message - the domain xml the engine receives. 

They are

1. Channel devices (as per the warnings channel0 through channel2)
2. pci controller devices (as per the warnings pci.1 through pci.8)
3. ide device

Are there other non managed devices that the engine could receive within said message?

(In reply to Steven Rosenberg from comment #10)
>
> I see that the warnings are generated by non managed devices. From the
> warnings and my investigation, I see three types of devices that are not
> managed that are received from the dumpxml response message - the domain xml
> the engine receives. 
> 
> They are
> 
> 1. Channel devices (as per the warnings channel0 through channel2)
> 2. pci controller devices (as per the warnings pci.1 through pci.8)
> 3. ide device
> 
> Are there other non managed devices that the engine could receive within
> said message?

What do you mean by managed devices? Managed by whom, libvirt?
Libvirt will put some devices by default in the domain XML, pci controllers, serial port and membaloon. IDE device is not added automatically.

(In reply to Michal Privoznik from comment #11)
> (In reply to Steven Rosenberg from comment #10)
> >
> > I see that the warnings are generated by non managed devices. From the
> > warnings and my investigation, I see three types of devices that are not
> > managed that are received from the dumpxml response message - the domain xml
> > the engine receives. 
> > 
> > They are
> > 
> > 1. Channel devices (as per the warnings channel0 through channel2)
> > 2. pci controller devices (as per the warnings pci.1 through pci.8)
> > 3. ide device
> > 
> > Are there other non managed devices that the engine could receive within
> > said message?
> 
> What do you mean by managed devices? Managed by whom, libvirt?
> Libvirt will put some devices by default in the domain XML, pci controllers,
> serial port and membaloon. IDE device is not added automatically.

Managed by the engine. To clarify, devices that libvirt sends in response to a dumpxml request that the engine did not send within its domain xml such as the pci controllers.

I see that serial ports, ide and memballoons are sent from the engine within its the domain xml. What I see is that Channel and pci controllers are not sent by the engine, but received by libvirt. I was asking if there are any other devices like that.

(In reply to Steven Rosenberg from comment #12)
> Managed by the engine. To clarify, devices that libvirt sends in response to
> a dumpxml request that the engine did not send within its domain xml such as
> the pci controllers.
> 
> I see that serial ports, ide and memballoons are sent from the engine within
> its the domain xml. What I see is that Channel and pci controllers are not
> sent by the engine, but received by libvirt. I was asking if there are any
> other devices like that.

Looking into the code, the following devices may be added automatically:

various bus controllers (SCSI, FDC, IDE, SATA, ...),
USB controller,
PCI root controller,
PCIe root controller,
memballoon,
USB input (mouse and keyboard),
panic device (only on s390 and ppc),
console/serial port,
video

whether they are added or not depends on passed domain XML. And also, this set might change in the future.

*** Bug 1788018 has been marked as a duplicate of this bug. ***

verify on rhv-03 with version :
        rhv-release-4.4.1-10-001.noarch
        vdsm-4.40.22-1.el8ev.x86_64

no errors  which mention on the ticket appears on the engine log while powering on VMs (40 VMs) 
tail -F /var/log/ovirt-engine/engine.log |grep  "WARN\|ERROR\|Received unexpected user-alias"

This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.