Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1366108

Summary: libvirt use wrong qmp command to hotplug a vhostuser network
Product: Red Hat Enterprise Linux 7 Reporter: Luyao Huang <lhuang>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: chhu
Severity: medium Docs Contact:
Priority: high    
Version: 7.3CC: chhu, dyuan, jdenemar, jsuchane, juzhang, pezhang, rbalakri, snagar, xuzhang, yalzhang
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.5.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1392032 (view as bug list) Environment:
Last Closed: 2017-08-01 17:11:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1394140    
Bug Blocks: 1366505, 1392032    

Description Luyao Huang 2016-08-11 05:18:28 UTC 
Description of problem:
libvirt use wrong command to hotplug a vhostuser network

Version-Release number of selected component (if applicable):
libvirt-2.0.0-5.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.
# cat vhost.xml
    <interface type='vhostuser'>
      <mac address='52:54:00:bb:25:74'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
    </interface>

2.
# virsh attach-device r7-raw vhost.xml 
Device attached successfully

3. check guest xml:

# virsh dumpxml r7-raw
...
    <interface type='vhostuser'>
      <mac address='52:54:00:bb:25:74'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>

4. check libvirt used qmp cmd line:

147.242 > 0x7fb7e8004500 {"execute":"netdev_add","arguments":{"type":"user","id":"hostnet2"},"id":"libvirt-40"}
147.244 < 0x7fb7e8004500 {"return": {}, "id": "libvirt-40"}
147.245 > 0x7fb7e8004500 {"execute":"device_add","arguments":{"driver":"virtio-net-pci","netdev":"hostnet2","id":"net2","mac":"52:54:00:bb:25:74","bus":"pci.0","addr":"0xa"},"id":"libvirt-41"}
147.276 < 0x7fb7e8004500 {"return": {}, "id": "libvirt-41"}


Actual results:

libvirt use wrong command to hotplug a vhostuser network

Expected results:

libvirt need attach a chardev which backend is socket and then use netdev_add vhost-user net device, at last use device_add to add virtio-net-pci

hmp will like this:

(qemu) chardev-add backend=socket,id=charnet2,path=/var/run/openvswitch/vhost-user2
(qemu) netdev_add vhost-user,id=hostnet2,chardev=charnet2
(qemu) device_add virtio-net-pci,netdev=hostnet2,id=net2

if libvirt do not want to support it right now, libvirt should report an error instead of attach a wrong device.

Additional info:

this problem already been mentioned on this mail:

https://www.redhat.com/archives/libvir-list/2015-January/msg00361.html
Comment 1 Michal Privoznik 2016-08-16 16:36:21 UTC 
Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2016-August/msg00806.html
Comment 3 Michal Privoznik 2016-10-14 05:37:14 UTC 
Moving to POST:

commit ff89d5cbcf266e780b4ff93960969b3a6c2e9a01
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Aug 15 18:01:55 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 11:45:01 2016 +0800

    qemu_hotplug: Support interface type of vhost-user hotplug
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1366108
    
    There are couple of things that needs to be done in order to
    allow vhost-user hotplug. Firstly, vhost-user requires a chardev
    which is connected to vhost-user bridge and through which qemu
    communicates with the bridge (no acutal guest traffic is sent
    through there, just some metadata). In order to generate proper
    chardev alias, we must assign device alias way sooner.
    
    Then, because we are plugging the chardev first, we need to do
    the proper undo if something fails - that is remove netdev too.
    We don't want anything to be left over in case attach fails at
    some point.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit e1844d85cbdcb3ed8199ae8ae5274ffcc4968aa5
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Aug 16 12:55:02 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 11:45:01 2016 +0800

    qemuBuildHostNetStr: Support VIR_DOMAIN_NET_TYPE_VHOSTUSER
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1366505
    
    So far, this function lacked support for
    VIR_DOMAIN_NET_TYPE_VHOSTUSER leaving callers to hack around the
    problem by constructing the command line on their own. This is
    not ideal as it blocks hot plug support.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit b093e8522483c689e6ffaf7bdef98991f862c88b
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Aug 16 12:21:49 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 11:45:01 2016 +0800

    qemuBuildVhostuserCommandLine: Unify -netdev creation
    
    Currently, what we do for vhost-user network is generate the
    following part of command line:
    
    -netdev type=vhost-user,id=hostnet0,chardev=charnet0
    
    There's no need for 'type=' it is the default. Drop it.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 0c61cf315859a74326da93bbb6f04a08509ef91e
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Aug 15 16:15:44 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 11:44:53 2016 +0800

    qemuBuildVhostuserCommandLine: Reuse qemuBuildChrChardevStr
    
    There's no need to reinvent the wheel here. We already have a
    function to format virDomainChrSourceDefPtr. It's called
    qemuBuildChrChardevStr(). Use that instead of some dummy
    virBufferAsprintf().
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 336d4a71feb1cd68df13e1812e86971c61553522
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Aug 16 13:06:46 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuBuildChrChardevStr: Introduce @nowait argument
    
    This alone makes not much sense. But the aim is to reuse this
    function in qemuBuildVhostuserCommandLine() where 'nowait' is not
    supported for vhost-user devices.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 1dcbef8a0f4bdfb25bf6b8cac08afd1a589c1ccd
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Aug 16 08:48:04 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuBuildHostNetStr: Explicitly enumerate net types
    
    We tend to prevent using 'default' in switches. And it is for a
    good reason - control may end up in paths we wouldn't want for
    new values. In this specific case, if qemuBuildHostNetStr is
    called over VIR_DOMAIN_NET_TYPE_VHOSTUSER it would produce
    meaningless output. Fortunately, there no such call yet.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit c266b6044003ac2f10f4cd15e58164d629508ccb
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Sep 26 11:53:58 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuDomainAttachNetDevice: Explicitly list allowed types for hotplug
    
    Instead of blindly claim support for hot-plugging of every
    interface type out there we should copy approach we have for
    device types: white listing supported types and explicitly error
    out on unsupported ones.
    For instance, trying to hotplug vhostuser interface results in
    nothing usable from guest currently. vhostuser typed interfaces
    require additional work on our side.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 5b65d772dd909d9b1ded07a86304229674ebf4ac
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Sep 26 11:41:36 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuDomainAttachNetDevice: Move hostdev handling a bit further
    
    The idea is to have function that does some checking at its
    beginning and then have one big switch for all the interface
    types it supports.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 0bce012d7f0bbf68c1b42a93d7d1c4e085e2695e
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Sep 26 11:05:00 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuBuildInterfaceCommandLine: Move from if-else forest to switch
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 4a74ccdb92fd2103392b177c38846b685e141840
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Sep 26 10:32:44 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuBuildInterfaceCommandLine: Move vhostuser handling a bit further
    
    The idea is to have function that does some checking of the
    arguments at its beginning and then have one big switch for all
    the interface types it supports. Each one of them generating the
    corresponding part of the command line.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit ec7f612a56ed8d3a2e83101b5cffcea3058fd30a
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Sep 26 09:54:26 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    qemuBuildInterfaceCommandLine: Move hostdev handling a bit further
    
    The idea is to have function that does some checking of the
    arguments at its beginning and then have one big switch for all
    the interface types it supports. Each one of them generating the
    corresponding part of the command line.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 507032d98d390245047f69474d0671d6b0b12651
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Sep 23 17:04:53 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    virDomainNetGetActualType: Return type is virDomainNetType
    
    This function for some weird reason returns integer instead of
    virDomainNetType type. It is important to return the correct type
    so that we know what values we can expect.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 94a10de26b1fa999dbb67cfeea8e3339205d9db6
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Sep 26 11:06:43 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 14 10:15:51 2016 +0800

    virDomainNetDefParseXML: Realign
    
    There are couple of formatting issues. No functional change
    though.
    
    Signed-off-by: Michal Privoznik <mprivozn>


v2.3.0-130-gff89d5c
Comment 7 chhu 2016-11-11 07:55:59 UTC 
Git latest upstream libvirt with all the patches listed in comment3, 
tested with packages:

qemu-kvm-rhev-2.6.0-27.el7.x86_64
kernel: 3.10.0-514.el7.x86_64

Hit Bug 1394140 when hot-plug a vhostuser network:
Bug 1394140 - qemu gets SIGSEGV when hot-plug a vhostuser network
Comment 10 chhu 2017-03-07 03:40:01 UTC 
Verified with packages:
libvirt-3.1.0-1.el7.x86_64
qemu-kvm-rhev-2.8.0-5.el7.x86_64
kernel-3.10.0-585.el7.x86_64

Test steps:
1. Setup the openvswtich dpdk environment, start an guest without vhostuser network.
# virsh start vm1
Domain vm1 started
# virsh dumpxml vm1|grep interface -A 5

2. Open another terminal to setup a systemtap of qemu monitor command:
# stap qemu-monitor.stp 
  0.000 begin

3. Attach a vhostuser network successfully.
# cat vhostuser1.xml 
   <interface type='vhostuser'>
      <mac address='52:54:00:93:61:dd'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

# virsh attach-device vm1 vhostuser1.xml 
Device attached successfully

# virsh dumpxml vm1|grep interface -A 5
    <interface type='vhostuser'>
      <mac address='52:54:00:93:61:dd'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

4. Check the systemtap of qemu monitor command output: libvirt attach a chardev which backend is socket and then use netdev_add vhost-user net device, then use device_add to add virtio-net-pci

# stap qemu-monitor.stp
  0.000 begin
 30.841 > 0x7f3e0c002460 {"execute":"chardev-add","arguments":{"id":"charnet0","backend":{"type":"socket","data":{"addr":{"type":"unix","data":{"path":"/var/run/openvswitch/vhost-user1"}},"wait":false,"server":false}}},"id":"libvirt-14"}
 30.844 < 0x7f3e0c002460 {"return": {}, "id": "libvirt-14"}
 30.844 > 0x7f3e0c002460 {"execute":"netdev_add","arguments":{"type":"vhost-user","chardev":"charnet0","id":"hostnet0"},"id":"libvirt-15"}
 30.846 < 0x7f3e0c002460 {"return": {}, "id": "libvirt-15"}
 30.846 > 0x7f3e0c002460 {"execute":"device_add","arguments":{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:93:61:dd","bus":"pci.0","addr":"0x3"},"id":"libvirt-16"}
 30.855 < 0x7f3e0c002460 {"return": {}, "id": "libvirt-16"}

5. Attach the second vhostuser network and detach it, check the systemtap of qemu monitor command output: PASS

# virsh attach-device vm1 vhostuser2.xml 
Device attached successfully

# virsh dumpxml vm1|grep interface -A 5
    <interface type='vhostuser'>
      <mac address='52:54:00:93:61:dd'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:32:43:51:dd'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# stap qemu-monitor.stp
  0.000 begin
167.477 > 0x7f3e0c005780 {"execute":"chardev-add","arguments":{"id":"charnet1","backend":{"type":"socket","data":{"addr":{"type":"unix","data":{"path":"/var/run/openvswitch/vhost-user2"}},"wait":false,"server":false}}},"id":"libvirt-23"}
167.479 < 0x7f3e0c005780 {"return": {}, "id": "libvirt-23"}
167.480 > 0x7f3e0c005780 {"execute":"netdev_add","arguments":{"type":"vhost-user","chardev":"charnet1","id":"hostnet1"},"id":"libvirt-24"}
167.481 < 0x7f3e0c005780 {"return": {}, "id": "libvirt-24"}
167.481 > 0x7f3e0c005780 {"execute":"device_add","arguments":{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"52:54:32:43:51:dd","bus":"pci.0","addr":"0x9"},"id":"libvirt-25"}
167.494 < 0x7f3e0c005780 {"return": {}, "id": "libvirt-25"}

Login to the guest:
# lspci |grep Ethernet
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:09.0 Ethernet controller: Red Hat, Inc Virtio network device

# virsh detach-device vm1 vhostuser2.xml 
Device detached successfully

# virsh dumpxml vm1| grep interface -A 5
    <interface type='vhostuser'>
      <mac address='52:54:00:93:61:dd'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

406.771 > 0x7f3e0c005780 {"execute":"device_del","arguments":{"id":"net1"},"id":"libvirt-27"}
406.772 < 0x7f3e0c005780 {"return": {}, "id": "libvirt-27"}
406.825 ! 0x7f3e0c005780 {"timestamp": {"seconds": 1488857477, "microseconds": 218248}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}
406.825 ! 0x7f3e0c005780 {"timestamp": {"seconds": 1488857477, "microseconds": 218337}, "event": "DEVICE_DELETED", "data": {"device": "net1", "path": "/machine/peripheral/net1"}}
406.825 > 0x7f3e0c005780 {"execute":"netdev_del","arguments":{"id":"hostnet1"},"id":"libvirt-28"}
406.828 < 0x7f3e0c005780 {"return": {}, "id": "libvirt-28"}
406.828 > 0x7f3e0c005780 {"execute":"chardev-remove","arguments":{"id":"charnet1"},"id":"libvirt-29"}
406.828 < 0x7f3e0c005780 {"return": {}, "id": "libvirt-29"}

Login to the guest:
# lspci|grep Ethernet
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
Comment 11 errata-xmlrpc 2017-08-01 17:11:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 12 errata-xmlrpc 2017-08-01 23:53:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846