Bug 1076708 - Upon assigning addresses to new virtio-serial ports, libvirt can over-allocate
Summary: Upon assigning addresses to new virtio-serial ports, libvirt can over-allocate
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1042505
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-14 21:05 UTC by Jonathan Lebon
Modified: 2015-11-19 05:45 UTC (History)
8 users (show)

Fixed In Version: libvirt-1.2.15-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of: 1042505
Environment:
Last Closed: 2015-11-19 05:45:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Jonathan Lebon 2014-03-14 21:05:19 UTC
+++ This bug was initially created as a clone of Bug #1042505 +++

Version: libvirt-1.1.1-27.el7

I noticed this issue is still present in the latest libvirt RHEL7 build. Any chance this could be fixed? I know it's low priority, but it's a nice 'polishing' bug.

Note that it's not only an issue in the default case, which is a bit unlikely (i.e. virtio-serial controller index 0 with more than 31 ports... don't think a lot of people need that many ports). It is a larger issue for virtio-serial controllers with e.g. maxport=4.

I've posted patches for this issue we could potentially work from here:
https://www.redhat.com/archives/libvir-list/2014-January/msg00187.html

Thanks,

Jonathan

+++ Original description +++

Good afternoon,

While experimenting with virtio-serial ports (using SystemTap's stapvirt
utility), we noticed something interesting. When stapvirt adds a
virtio-serial port, it simply adds the following XML

  <channel type='unix'>
    <source mode='bind' path='/my/path/to/sock.sock'/>
    <target type='virtio' name='org.systemtap.stapsh.0'/>
  </channel>

and relies on libvirt to get assigned to a virtio-serial controller
rather than creating an <address> element itself. So the resulting XML
might look like this:

  <channel type='unix'>
    <source mode='bind' path='/my/path/to/sock.sock'/>
    <target type='virtio' name='org.systemtap.stapsh.0'/>
    <address type='virtio-serial' controller='0' bus='0' port='1'/>
  </channel>

When creating another port, libvirt simply looks at previous
virtio-serial ports and adds one to the port number.

There are a few issues however:

1. libvirt will always use controller='0' and bus='0' even if there is
already a controller on a different index. This is not a big deal in
itself, since we can make a reasonable assumption that a controller
index 0 will exist before index 1, except that

2. libvirt does not respect the maximum number of ports available on a
controller. Even if the controller is declared with ports='4', libvirt
will gladly create e.g. 10 ports on that controller. This of course will
cause qemu to complain, e.g.:

error: Failed to start domain TestVM
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device
virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.systemtap.stapsh.3:
virtio-serial-bus: Out-of-range port id specified, max. allowed: 3
qemu-system-x86_64: -device
virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.systemtap.stapsh.3:
Device 'virtserialport' could not be initialized

This means that by default, libvirt will implicitly create a controller with index 0, then continuously add ports to that same controller well passed the limit of 31.

Expected:

libvirt respects the 'ports' attribute of the virtio-serial controller (assume 31 if missing) and creates a new controller if all the controllers are filled.

More generally, upon assigning an address, look at all the virtio-serial controllers available (including those to be inferred) and check if there are ports available on any of them. If not, then create a new controller (e.g. set controller to max_controller_index+1 and let it get implicitly created later during the definition process).


Thanks

Comment 4 Ján Tomko 2015-03-03 14:47:27 UTC
Upstream patches:
https://www.redhat.com/archives/libvir-list/2015-March/msg00071.html

Comment 6 Ján Tomko 2015-03-17 11:42:29 UTC
Another version of the patches:
https://www.redhat.com/archives/libvir-list/2015-March/msg00821.html

Comment 8 Ján Tomko 2015-04-02 13:08:37 UTC
Fixed upstream by:
commit 5903378834bafb031407ab02ce37dcc9ec782d1f
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2015-04-02 15:00:13 +0200

    Allocate virtio-serial addresses when starting a domain

git describe: v1.2.14-30-g5903378

commit 371ea92f058a0df3ba55c0b398f892653cbfca3             
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2015-04-02 15:00:13 +0200

    Auto add virtio-serial controllers
  
git describe: v1.2.14-33-g1371ea9

Comment 10 Hu Jianwei 2015-05-07 06:08:12 UTC
There are some big changes for virtio char devices, I do some sanity testing for those changes.

Version:
libvirt-1.2.15-1.el7.x86_64
qemu-kvm-rhev-2.3.0-1.el7.x86_64

1. libvirt can find out a free port for the new char device using "virsh edit"
[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>

Add below 3 line xml to domain:
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.1'/>
    </channel>

[root@localhost ~]# virsh edit r71
Domain r71 XML configuration edited.

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

2. Automatically generated port number for multi char devices 
add below lines into domain
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
    </channel>

[root@localhost ~]# virsh edit r71
Domain r71 XML configuration edited.

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B3
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>
[root@localhost ~]#

3. Hot-plug/unplug char device for live domain
a. add a char with port 4
[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
[root@localhost ~]# cat channel.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
</channel>

[root@localhost ~]# virsh attach-device r71 channel.xml
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>

b. add a char without port, search a free port from in 0~30
[root@localhost ~]# cat ch.xml 
<channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
    </channel>

[root@localhost ~]# virsh attach-device r71 ch.xml
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel4'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>

c. detach a char device, its port is 2
[root@localhost ~]# cat de.xml 
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
[root@localhost ~]# virsh detach-device r71 de.xml
Device detached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel4'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>

 d. attach a char dev without port, the port 2 can be used again
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.44'/>
</channel>

[root@localhost ~]# 
[root@localhost ~]# virsh attach-device r71 ch1.xml
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel4'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.44'/>
      <alias name='channel5'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

4. Checked some exception output:
a. duplicated port
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
</channel>
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: virtio serial port 4 on controller 0 is already occupied

b. with a non-existing controller
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='1' bus='0' port='4'/>
</channel>
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: virtio serial controller 1 is missing

c. exceed the maximum port
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='31'/>
</channel>
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: virtio serial controller 0 does not have port 31

d. when no free port on virtio serial controller 
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: Unable to find a free virtio-serial port

c. libvirt does not support to attach a controller to live domain, only SCSI controller can do that by now.
[root@localhost ~]# cat con.xml 
    <controller type='virtio-serial' index='1'>
    </controller>
[root@localhost ~]# virsh attach-device r71 con.xml
error: Failed to attach device from con.xml
error: Operation not supported: 'virtio-serial' controller cannot be hot plugged.

But I found a small issue, with "--config" option libvirt still can accept duplicated port for virtio channel devcies, but based on your patches, the domain can not boot up normally, need we make a small patch to fix this?
Thanks.

[root@localhost ~]# virsh attach-device r71 channel.xml --config
Device attached successfully
[root@localhost ~]# vim channel.xml 
[root@localhost ~]# 
[root@localhost ~]# virsh attach-device r71 channel.xml --config
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.1'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.4'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
[root@localhost ~]# virsh start r71
error: Failed to start domain r71
error: XML error: virtio serial port 4 on controller 0 is already occupied


BTW, any suggestion for my above testing results? Is it enough for the bug?

Comment 11 Ján Tomko 2015-05-07 11:47:04 UTC
The testing seems sufficient to me.

I have sent a patch for the --config issue. It probably deserves a separate bug.
https://www.redhat.com/archives/libvir-list/2015-May/msg00184.html

Comment 12 Hu Jianwei 2015-05-11 01:38:11 UTC
(In reply to Ján Tomko from comment #11)
> The testing seems sufficient to me.
> 

Thanks for your review, I'll verify the bug after do more regression testing.

> I have sent a patch for the --config issue. It probably deserves a separate
> bug.
> https://www.redhat.com/archives/libvir-list/2015-May/msg00184.html

A new bug for the coldplug issue:
Bug 1220195 - Assign virtio-serial addresses after coldplugging a device

Thanks.

Comment 13 Hu Jianwei 2015-06-04 00:53:30 UTC
According to comment 10 and 11, moved to Verified.

Comment 15 errata-xmlrpc 2015-11-19 05:45:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html


Note You need to log in before you can comment on or make changes to this bug.