1076708 – Upon assigning addresses to new virtio-serial ports, libvirt can over-allocate

Bug 1076708 - Upon assigning addresses to new virtio-serial ports, libvirt can over-allocate

Summary: Upon assigning addresses to new virtio-serial ports, libvirt can over-allocate

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Ján Tomko
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:	1042505
Blocks:
TreeView+	depends on / blocked

Reported:	2014-03-14 21:05 UTC by Jonathan Lebon
Modified:	2015-11-19 05:45 UTC (History)
CC List:	8 users (show)
Fixed In Version:	libvirt-1.2.15-1.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:	1042505
Environment:
Last Closed:	2015-11-19 05:45:29 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:2202	0	normal	SHIPPED_LIVE	libvirt bug fix and enhancement update	2015-11-19 08:17:58 UTC

Description Jonathan Lebon 2014-03-14 21:05:19 UTC

+++ This bug was initially created as a clone of Bug #1042505 +++

Version: libvirt-1.1.1-27.el7

I noticed this issue is still present in the latest libvirt RHEL7 build. Any chance this could be fixed? I know it's low priority, but it's a nice 'polishing' bug.

Note that it's not only an issue in the default case, which is a bit unlikely (i.e. virtio-serial controller index 0 with more than 31 ports... don't think a lot of people need that many ports). It is a larger issue for virtio-serial controllers with e.g. maxport=4.

I've posted patches for this issue we could potentially work from here:
https://www.redhat.com/archives/libvir-list/2014-January/msg00187.html

Thanks,

Jonathan

+++ Original description +++

Good afternoon,

While experimenting with virtio-serial ports (using SystemTap's stapvirt
utility), we noticed something interesting. When stapvirt adds a
virtio-serial port, it simply adds the following XML

  <channel type='unix'>
    <source mode='bind' path='/my/path/to/sock.sock'/>
    <target type='virtio' name='org.systemtap.stapsh.0'/>
  </channel>

and relies on libvirt to get assigned to a virtio-serial controller
rather than creating an <address> element itself. So the resulting XML
might look like this:

  <channel type='unix'>
    <source mode='bind' path='/my/path/to/sock.sock'/>
    <target type='virtio' name='org.systemtap.stapsh.0'/>
    <address type='virtio-serial' controller='0' bus='0' port='1'/>
  </channel>

When creating another port, libvirt simply looks at previous
virtio-serial ports and adds one to the port number.

There are a few issues however:

1. libvirt will always use controller='0' and bus='0' even if there is
already a controller on a different index. This is not a big deal in
itself, since we can make a reasonable assumption that a controller
index 0 will exist before index 1, except that

2. libvirt does not respect the maximum number of ports available on a
controller. Even if the controller is declared with ports='4', libvirt
will gladly create e.g. 10 ports on that controller. This of course will
cause qemu to complain, e.g.:

error: Failed to start domain TestVM
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device
virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.systemtap.stapsh.3:
virtio-serial-bus: Out-of-range port id specified, max. allowed: 3
qemu-system-x86_64: -device
virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.systemtap.stapsh.3:
Device 'virtserialport' could not be initialized

This means that by default, libvirt will implicitly create a controller with index 0, then continuously add ports to that same controller well passed the limit of 31.

Expected:

libvirt respects the 'ports' attribute of the virtio-serial controller (assume 31 if missing) and creates a new controller if all the controllers are filled.

More generally, upon assigning an address, look at all the virtio-serial controllers available (including those to be inferred) and check if there are ports available on any of them. If not, then create a new controller (e.g. set controller to max_controller_index+1 and let it get implicitly created later during the definition process).


Thanks

Comment 4 Ján Tomko 2015-03-03 14:47:27 UTC

Upstream patches:
https://www.redhat.com/archives/libvir-list/2015-March/msg00071.html

Comment 6 Ján Tomko 2015-03-17 11:42:29 UTC

Another version of the patches:
https://www.redhat.com/archives/libvir-list/2015-March/msg00821.html

Comment 7 Ján Tomko 2015-03-24 17:16:42 UTC

v3: https://www.redhat.com/archives/libvir-list/2015-March/msg01233.html

Comment 8 Ján Tomko 2015-04-02 13:08:37 UTC

Fixed upstream by:
commit 5903378834bafb031407ab02ce37dcc9ec782d1f
Author:     Ján Tomko <jtomko>
CommitDate: 2015-04-02 15:00:13 +0200

    Allocate virtio-serial addresses when starting a domain

git describe: v1.2.14-30-g5903378

commit 371ea92f058a0df3ba55c0b398f892653cbfca3             
Author:     Ján Tomko <jtomko>
CommitDate: 2015-04-02 15:00:13 +0200

    Auto add virtio-serial controllers
  
git describe: v1.2.14-33-g1371ea9

Comment 10 Hu Jianwei 2015-05-07 06:08:12 UTC

There are some big changes for virtio char devices, I do some sanity testing for those changes.

Version:
libvirt-1.2.15-1.el7.x86_64
qemu-kvm-rhev-2.3.0-1.el7.x86_64

1. libvirt can find out a free port for the new char device using "virsh edit"
[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>

Add below 3 line xml to domain:
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.1'/>
    </channel>

[root@localhost ~]# virsh edit r71
Domain r71 XML configuration edited.

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

2. Automatically generated port number for multi char devices 
add below lines into domain
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
    </channel>

[root@localhost ~]# virsh edit r71
Domain r71 XML configuration edited.

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B3
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>
[root@localhost ~]#

3. Hot-plug/unplug char device for live domain
a. add a char with port 4
[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
[root@localhost ~]# cat channel.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
</channel>

[root@localhost ~]# virsh attach-device r71 channel.xml
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>

b. add a char without port, search a free port from in 0~30
[root@localhost ~]# cat ch.xml 
<channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
    </channel>

[root@localhost ~]# virsh attach-device r71 ch.xml
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel4'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>

c. detach a char device, its port is 2
[root@localhost ~]# cat de.xml 
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
[root@localhost ~]# virsh detach-device r71 de.xml
Device detached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel4'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>

 d. attach a char dev without port, the port 2 can be used again
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.44'/>
</channel>

[root@localhost ~]# 
[root@localhost ~]# virsh attach-device r71 ch1.xml
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/9'/>
      <target type='virtio' name='org.linux-kvm.port.1' state='disconnected'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/10'/>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <alias name='channel3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel4'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>
    <channel type='pty'>
      <source path='/dev/pts/8'/>
      <target type='virtio' name='org.linux-kvm.port.44'/>
      <alias name='channel5'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

4. Checked some exception output:
a. duplicated port
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
</channel>
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: virtio serial port 4 on controller 0 is already occupied

b. with a non-existing controller
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='1' bus='0' port='4'/>
</channel>
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: virtio serial controller 1 is missing

c. exceed the maximum port
[root@localhost ~]# cat ch1.xml 
<channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='31'/>
</channel>
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: virtio serial controller 0 does not have port 31

d. when no free port on virtio serial controller 
[root@localhost ~]# virsh attach-device r71 ch1.xml
error: Failed to attach device from ch1.xml
error: XML error: Unable to find a free virtio-serial port

c. libvirt does not support to attach a controller to live domain, only SCSI controller can do that by now.
[root@localhost ~]# cat con.xml 
    <controller type='virtio-serial' index='1'>
    </controller>
[root@localhost ~]# virsh attach-device r71 con.xml
error: Failed to attach device from con.xml
error: Operation not supported: 'virtio-serial' controller cannot be hot plugged.

But I found a small issue, with "--config" option libvirt still can accept duplicated port for virtio channel devcies, but based on your patches, the domain can not boot up normally, need we make a small patch to fix this?
Thanks.

[root@localhost ~]# virsh attach-device r71 channel.xml --config
Device attached successfully
[root@localhost ~]# vim channel.xml 
[root@localhost ~]# 
[root@localhost ~]# virsh attach-device r71 channel.xml --config
Device attached successfully

[root@localhost ~]# virsh dumpxml r71 | grep "</channel>" -B5
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.1'/>
      <address type='virtio-serial' controller='0' bus='0' port='8'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.3'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.4'/>
      <address type='virtio-serial' controller='0' bus='0' port='4'/>
    </channel>
[root@localhost ~]# virsh start r71
error: Failed to start domain r71
error: XML error: virtio serial port 4 on controller 0 is already occupied


BTW, any suggestion for my above testing results? Is it enough for the bug?

Comment 11 Ján Tomko 2015-05-07 11:47:04 UTC

The testing seems sufficient to me.

I have sent a patch for the --config issue. It probably deserves a separate bug.
https://www.redhat.com/archives/libvir-list/2015-May/msg00184.html

Comment 12 Hu Jianwei 2015-05-11 01:38:11 UTC

(In reply to Ján Tomko from comment #11)
> The testing seems sufficient to me.
> 

Thanks for your review, I'll verify the bug after do more regression testing.

> I have sent a patch for the --config issue. It probably deserves a separate
> bug.
> https://www.redhat.com/archives/libvir-list/2015-May/msg00184.html

A new bug for the coldplug issue:
Bug 1220195 - Assign virtio-serial addresses after coldplugging a device

Thanks.

Comment 13 Hu Jianwei 2015-06-04 00:53:30 UTC

According to comment 10 and 11, moved to Verified.

Comment 15 errata-xmlrpc 2015-11-19 05:45:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html

Note You need to log in before you can comment on or make changes to this bug.