Please attach libvirt config for both 7.2 and 7.3

Gerd, this is the error being received on the guest:

#################################
internalError: No free USB ports

file "/usr/lib64/python2.7/site-packages/libvirt.py", line 536, in attachDevice
if ret == -1: raise libvirtError ('virDomainAttachDevice() failed, dom=self')

libvirtError: internal error: No free USB ports
##################################

I cannot provide the libvirt config, but all options in the files are commented out. So it is a default config. 

-mahoney

> #################################
> internalError: No free USB ports
> 
> file "/usr/lib64/python2.7/site-packages/libvirt.py", line 536, in
> attachDevice
> if ret == -1: raise libvirtError ('virDomainAttachDevice() failed, dom=self')
> 
> libvirtError: internal error: No free USB ports
> ##################################

That looks like libvirt not being able to handle the domain configuration after upgrading.

> I cannot provide the libvirt config, but all options in the files are
> commented out. So it is a default config. 

Oh, I mean the libvirt xml config (i.e. the output of "virsh dumpxml $domain").

I suspect what causes problems here is that libvirt starts managing usb ports in recent versions.  Old libvirt versions leave that to qemu.  qemu will then automatically add a usb hub to avoid running out of usb ports.  You can see that with "lsusb -t" (in the guest).  Problem with that approach is that it can break live migration in case usb devices end up in different ports on source and target host (can happen if you hotplug usb devices).

New libvirt versions manage the usb ports, and in that case usb hubs must be explicitly added to the libvirt domain config.  Appearently libvirt doesn't handle that case automatically after upgrading.

Try removing usb devices until you have 5 only, then add a usb hub, then re-add the devices.  That should get things going how they did on 7.2

libvirt xml syntax looks like this for a hub:

    <hub type='usb'>
      <address type='usb' bus='0' port='6'/>
    </hub>

plug usb devices into the hub this way:

    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='6.1'/>
    </input>

port='6.1' specifies the usb port, "6" is the root port (where the hub is plugged in) and "1" means the first port of the hub.

Alternative approach #1: Add a second usb2 host controller, so you have 12 root ports.

Alternative approach #2: Use a xhci (usb3 host controller) instead, which allows to configure the number of root ports (4 by default, up to 15 are possible):

    <controller type='usb' index='0' model='nec-xhci' ports='8'/>

Moving bug to libvirt.

Upstream patch:
https://www.redhat.com/archives/libvir-list/2017-January/msg01101.html

Pushed upstream as:
commit 3ac97c2dedd2a4715d06cf506be3727df32e79eb
Author:     Ján Tomko <jtomko>
CommitDate: 2017-01-31 13:09:08 +0100

    qemu: Add enough USB hubs to accomodate all devices
    
    Commit 815d98a started auto-adding one hub if there are more USB devices
    than available USB ports.
    
    This was a strange choice, since there might be even more devices.
    Before USB address allocation was implemented in libvirt, QEMU
    automatically added a new USB hub if the old one was full.
    
    Adjust the logic to try adding as many hubs as will be needed
    to plug in all the specified devices.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1410188

git describe: v3.0.0-95-g3ac97c2

Version :
libvirt-3.2.0-9.el7.x86_64
qemu-kvm-rhev-2.9.0-9.el7.x86_64

Steps :
Define a vm with multiple usb devices (more than 6)
1. Prepare vm1 xml as following :
......
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/r74.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/usb/usba.img'/>
      <target dev='sda' bus='usb'/>
    </disk>
......
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/usb/usbe.img'/>
      <target dev='sde' bus='usb'/>
    </disk>
......
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>

2. Define and start vm1 
# virsh define vm1.xml 
Domain vm1 defined from vm1.xml

#  virsh start vm1
Domain vm1 started

3. Check domain xml, it will add a hub automatically.
......
    <hub type='usb'>
      <address type='usb' bus='0' port='1'/>
    </hub>
......

Also check using virsh edit to add multiple usb devices without address,
save and start vm successfully.

Hi,
I want to cold plug multiple USB disks then start guest. 
It seems that usb hub will not be added automatically for cold plug. 
I wonder if adding usb hubs automatically only works well for define or edit VMs.

1. Define a guest without USB disks.

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     vm1                            shut off

# virsh domblklist vm1
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/r74.qcow2

# virsh dumpxml vm1 | grep usb -A 3
......
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
--
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>

2. Cold plug multiple USB disks 
# for i in {a..h}; do virsh attach-disk vm1 /var/lib/libvirt/images/usb/usb$i.img --targetbus usb sd$i --config; done 
Disk attached successfully

Disk attached successfully

Disk attached successfully

Disk attached successfully

error: Failed to attach disk
error: internal error: No free USB ports

error: Failed to attach disk
error: internal error: No free USB ports
......

The code does not plug in another hub when you fill up the last free port with a non-hub device. This also happens on hotplug. And I believe this deserves a separate bug.

Thanks a lot for your info.
I filed another Bug 1461633 to track issue in comment 10.
And move this bug to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846