Bug 1677200 - Fails to hotplug/unplug guestfwd character device
Summary: Fails to hotplug/unplug guestfwd character device
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 8.0
Assignee: Marc-Andre Lureau
QA Contact: liunana
URL:
Whiteboard:
Depends On:
Blocks: 1591636 1624204
TreeView+ depends on / blocked
 
Reported: 2019-02-14 09:18 UTC by Jaroslav Suchanek
Modified: 2019-11-06 07:13 UTC (History)
14 users (show)

Fixed In Version: qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1591636
Environment:
Last Closed: 2019-11-06 07:12:59 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:13:39 UTC

Description Jaroslav Suchanek 2019-02-14 09:18:33 UTC
Commit e47f81b617684c4546af286d307b69014a83538a is needed for qemu-kvm as stated by Michal below.

+++ This bug was initially created as a clone of Bug #1591636 +++

Description of problem:
Fails to hotplug/unplug guestfwd character device

Version-Release number of selected component (if applicable):
libvirt-4.4.0-2.virtcov.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Start a guest

2.Attach guestfwd to guest:
# cat guestfwd.xml 
    <channel type='unix'>
      <source mode='bind' path='/tmp/guestfwd'/>
      <target type='guestfwd' address='10.0.2.1' port='4600'/>
    </channel>

# virsh attach-device 1 guestfwd.xml 
error: Failed to attach device from guestfwd.xml
error: internal error: unable to execute QEMU command 'device_add': 'user' is not a valid device model name


3.Coldplug guestfwd to guest and restart guest

4.Hotunplug guestfwd:
# virsh detach-device 1 guestfwd.xml 
error: Failed to detach device from guestfwd.xml
error: internal error: unable to execute QEMU command 'device_del': Device 'channel3' not found


Actual results:
Hotplug/unplug guestfwd fails

Expected results:
Hotplug/unplug guestfwd succeeds

Additional info:
Log:
2018-06-15 07:26:49.954+0000: 26130: info : qemuMonitorIOWrite:535 : QEMU_MONITOR_IO_WRITE: mon=0x7fc67402c030 buf={"execute":"device_add","arguments":{"driver":"user","guestfwd":"tcp:10.0.2.1:4600-chardev:charchannel3","id":"user-channel3"},"id":"libvirt-22"}

2018-06-15 07:28:13.555+0000: 26136: debug : qemuMonitorJSONCheckError:383 : unable to execute QEMU command {"execute":"device_del","arguments":{"id":"channel3"},"id":"libvirt-21"}: {"id":"libvirt-21","error":{"class":"DeviceNotFound","desc":"Device 'channel3' not found"}}


--- Additional comment from Michal Privoznik on 2019-02-14 09:35:17 CET ---

I've just merged patches upstream:

17ddfd420a qemuhotplugtest: Test guestfwd attach and detach
112f3a8d0f qemu_hotplug: Detach guestfwd using netdev_del
903315dc8f qemu_hotplug: Attach guestfwd using netdev_add
18b8f67745 qemuL: Drop "user-" prefix for guestfwd netdev
f538f5ed3a qemu: Use @tmpChr in qemuDomainDetachChrDevice to build device string

v5.0.0-387-g17ddfd420a

However, as mentioned here:

https://www.redhat.com/archives/libvir-list/2019-February/msg00798.html

it is important to use new enough qemu (at the time I'm writing these lines there's no qemu released that has the fix and thus only qemu from git works). The qemu commit that fixes qemu behaviour is:

commit e47f81b617684c4546af286d307b69014a83538a
Merge: ff372bb5c4 ee261c0233
Author:     Peter Maydell <peter.maydell>
AuthorDate: Thu Feb 7 18:53:25 2019 +0000
Commit:     Peter Maydell <peter.maydell>
CommitDate: Thu Feb 7 18:53:25 2019 +0000

    Merge remote-tracking branch 'remotes/thibault/tags/samuel-thibault' into staging
    
    More work towards libslirp

Comment 1 Li Xiaohui 2019-02-15 09:51:11 UTC
Hi all, 
I have tried to reproduce this bug via libvirt on rhel8, and reproduce successfully:
1.hotplug device to guest
[root@ibm-x3250m6-07 qemu-sh]# virsh attach-device verify-bug guestfwd.xml
error: Failed to attach device from guestfwd.xml
error: internal error: unable to execute QEMU command 'device_add': 'user' is not a valid device model name

2.boot guest with guestfwd, and unhotplug device
[root@ibm-x3250m6-07 qemu-sh]# virsh detach-device verify-bug guestfwd.xml
error: Failed to detach device from guestfwd.xml
error: internal error: unable to execute QEMU command 'device_del': Device 'channel0' not found

Notes:when boot guest with guestfwd, get qemu command like followings:
-chardev socket,id=charchannel0,fd=31,server,nowait -netdev user,guestfwd=tcp:10.0.2.1:4600-chardev:charchannel0,id=user-channel0


And tried to test the bug via qemu, boot guest with above guestfwd qemu commands(-chardev socket,... -netdev user,...), found qemu don't support "fd" parameter for chardev socket:
qemu-kvm: -chardev socket,id=charchannel0,fd=29,server,nowait: File descriptor '31' is not a socket
Why qemu couldn't boot guest with fd for chardev socket? Is this a normal action for qemu? Or maybe I do some mistake in qemu commands? If so, welcome to correct me.

Could someone give me some useful advise since we don't test such case in qemu before?

Comment 2 Michal Privoznik 2019-02-15 10:19:46 UTC
(In reply to Li Xiaohui from comment #1)
> Hi all, 
> I have tried to reproduce this bug via libvirt on rhel8, and reproduce
> successfully:
> 1.hotplug device to guest
> [root@ibm-x3250m6-07 qemu-sh]# virsh attach-device verify-bug guestfwd.xml
> error: Failed to attach device from guestfwd.xml
> error: internal error: unable to execute QEMU command 'device_add': 'user'
> is not a valid device model name
> 
> 2.boot guest with guestfwd, and unhotplug device
> [root@ibm-x3250m6-07 qemu-sh]# virsh detach-device verify-bug guestfwd.xml
> error: Failed to detach device from guestfwd.xml
> error: internal error: unable to execute QEMU command 'device_del': Device
> 'channel0' not found
> 
> Notes:when boot guest with guestfwd, get qemu command like followings:
> -chardev socket,id=charchannel0,fd=31,server,nowait -netdev
> user,guestfwd=tcp:10.0.2.1:4600-chardev:charchannel0,id=user-channel0
> 
> 
> And tried to test the bug via qemu, boot guest with above guestfwd qemu
> commands(-chardev socket,... -netdev user,...), found qemu don't support
> "fd" parameter for chardev socket:
> qemu-kvm: -chardev socket,id=charchannel0,fd=29,server,nowait: File
> descriptor '31' is not a socket
> Why qemu couldn't boot guest with fd for chardev socket? Is this a normal
> action for qemu? Or maybe I do some mistake in qemu commands? If so, welcome
> to correct me.

The file descriptor passing won't work if you run qemu by hand. You will have to replace fd= with path=. For instance:

-chardev socket,id=charchannel1,path=/tmp/guestfwd,server,nowait \
-netdev user,guestfwd=tcp:10.0.2.1:4600-chardev:charchannel1,id=channel1 \

This bus is then about qemu being unable to detach charchannel1 after channel1 is detached. In my testing I was getting "Chardev 'charchannel1' is busy" error message. With the referenced commit, I no longer see the error and the charchannel1 is detached automatically with netdev_del of channel1.

Comment 4 Li Xiaohui 2019-02-18 02:56:26 UTC
(In reply to Michal Privoznik from comment #2)
> (In reply to Li Xiaohui from comment #1)
> > Hi all, 
> > I have tried to reproduce this bug via libvirt on rhel8, and reproduce
> > successfully:
> > 1.hotplug device to guest
> > [root@ibm-x3250m6-07 qemu-sh]# virsh attach-device verify-bug guestfwd.xml
> > error: Failed to attach device from guestfwd.xml
> > error: internal error: unable to execute QEMU command 'device_add': 'user'
> > is not a valid device model name
> > 
> > 2.boot guest with guestfwd, and unhotplug device
> > [root@ibm-x3250m6-07 qemu-sh]# virsh detach-device verify-bug guestfwd.xml
> > error: Failed to detach device from guestfwd.xml
> > error: internal error: unable to execute QEMU command 'device_del': Device
> > 'channel0' not found
> > 
> > Notes:when boot guest with guestfwd, get qemu command like followings:
> > -chardev socket,id=charchannel0,fd=31,server,nowait -netdev
> > user,guestfwd=tcp:10.0.2.1:4600-chardev:charchannel0,id=user-channel0
> > 
> > 
> > And tried to test the bug via qemu, boot guest with above guestfwd qemu
> > commands(-chardev socket,... -netdev user,...), found qemu don't support
> > "fd" parameter for chardev socket:
> > qemu-kvm: -chardev socket,id=charchannel0,fd=29,server,nowait: File
> > descriptor '31' is not a socket
> > Why qemu couldn't boot guest with fd for chardev socket? Is this a normal
> > action for qemu? Or maybe I do some mistake in qemu commands? If so, welcome
> > to correct me.
> 
> The file descriptor passing won't work if you run qemu by hand. You will
> have to replace fd= with path=. For instance:
> 
> -chardev socket,id=charchannel1,path=/tmp/guestfwd,server,nowait \
> -netdev user,guestfwd=tcp:10.0.2.1:4600-chardev:charchannel1,id=channel1 \
> 
> This bus is then about qemu being unable to detach charchannel1 after
> channel1 is detached. In my testing I was getting "Chardev 'charchannel1' is
> busy" error message. With the referenced commit, I no longer see the error
> and the charchannel1 is detached automatically with netdev_del of channel1.

Thank you, Michal, I test again using your given comands:
1.boot guest and hotplug guestfwd successfully:
(1)in qmp
{"execute":"chardev-add","arguments":{"id":"charchannel1","backend":{"type":"socket","data":{"addr":{"type":"unix","data":{"path": "/tmp/guestfwd"}}}}}}
{"return": {}}
{"timestamp": {"seconds": 1550457853, "microseconds": 804802}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "netdev1", "path": "/machine/peripheral/netdev1/virtio-backend"}}
{"execute":"netdev_add","arguments":{"type":"user","id":"channel1","guestfwd":"tcp:10.0.2.1:4600-chardev:charchannel1"}}
{"return": {}}
(2)in hmp, check added devices:
(qemu) info chardev
...
charchannel1: filename=disconnected:unix:/tmp/guestfwd,server 
(qemu) info network 
netdev1: index=0,type=nic,model=virtio-net-pci,macaddr=02:b4:56:24:82:1c
 \ net1: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
channel1: index=0,type=user,net=10.0.2.0,restrict=off

2.boot guest with guestfwd and unhotplug it, get same result with yours:
(qemu) netdev_del channel1
(qemu) char
chardev-add         chardev-change      chardev-remove      chardev-send-break  
(qemu) chardev-remove 
charchannel1     compat_monitor0  compat_monitor1  
(qemu) chardev-remove charchannel1 
Chardev 'charchannel1' is busy

Waiting for fix version to verify this bug. Thanks all.

Best Regards,
Li Xiaohui

Comment 6 liunana 2019-05-22 07:51:09 UTC
Hi, 

I test this bug with qemu4.0, steps are as follows:

Test Environments
 host:
   qemu-kvm-4.0.0-1.module+el8.1.0+3225+a8268fde.x86_64
   libvirt-4.5.0-24.module+el8.1.0+3205+41ff0a42.x86_64
   kernel-4.18.0-91.el8.x86_64
 guest:
   kernel-4.18.0-87.el8.x86_64


1. boot a guest boot guest and hotplug guestfwd successfully.
Steps:

   a-. hotplug chardev
    #{"execute":"chardev-add","arguments":{"id":"charchannel1","backend":{"type":"socket","data":{"addr":{"type":"unix","data":{"path": "/tmp/guestfwd"}}}}}}
     #{"return": {}}
  b-. check from hmp
    (qemu) info chardev
     compat_monitor1: filename=disconnected:tcp:0.0.0.0:4444,server
     compat_monitor0: filename=stdio
     charchannel1: filename=disconnected:unix:/tmp/guestfwd,server
  c-. hotplug guestfwd
     #{"execute":"netdev_add","arguments":{"type":"user","id":"channel1","guestfwd":"tcp:10.0.2.1:4600-chardev:charchannel1"}}
     #{"return": {}}
  d-. check
    (qemu) info network 
     idhRa7sf: index=0,type=nic,model=virtio-net-pci,macaddr=70:5a:0f:38:cd:a7
     \ idNIlYmb: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
     channel1: index=0,type=user,net=10.0.2.0,restrict=off
  

2. boot guest with guestfwd and unhotplug it successfully.
Steps:

   a-. boot guest with commands:
      -chardev socket,id=charchannel1,path=/tmp/guestfwd,server,nowait \
      -netdev user,guestfwd=tcp:10.0.2.1:4600-chardev:charchannel1,id=channel1 \
   
   b-. can find chardev "charchannel1" and guestfwd "channel1" by hmp
 
   c-. hot-unplug guestfwd
      (qemu) netdev_del channel1 

   d-. check by hmp, we can find hot-unplug guestfwd and chardev successfully.
     (qemu) info network 
      idhRa7sf: index=0,type=nic,model=virtio-net-pci,macaddr=70:5a:0f:38:cd:a7
      \ idNIlYmb: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

     (qemu) info chardev
       compat_monitor1: filename=disconnected:tcp:0.0.0.0:4444,server
       compat_monitor0: filename=stdio
 
The charchannel1 is detached automatically with netdev_del of channel1.
 
Besides, I meet one issue when boot with guestfwd, qemu show one warning info "(qemu) qemu-kvm: warning: netdev channel1 has no peer", seems guestfwd need a option to connect when we usee it, am I right?

Please help check this test and problem, thanks a lot!




Best regards 
Liu Nana

Comment 8 liunana 2019-06-21 02:23:05 UTC
Hi


Could you please help to check Comment 6? If it is ok we can change this bug's status from ON_QA to VERIFIED.
Thanks


Best regards
Liu Nana

Comment 9 Marc-Andre Lureau 2019-06-21 11:24:03 UTC
(In reply to liunana from comment #8)
> Hi
> 
> 
> Could you please help to check Comment 6? If it is ok we can change this
> bug's status from ON_QA to VERIFIED.
> Thanks

comment 6 is correctly showing hotplug/unplug of guestfwd/chardev.

Regarding "Besides, I meet one issue when boot with guestfwd, qemu show one warning info "(qemu) qemu-kvm: warning: netdev channel1 has no peer", seems guestfwd need a option to connect when we usee it, am I right?",

last I looked at it, the way guest forwarding is implemented in libvirt is quite broken. In in this hotplug case, adding a netdev channel alone doesn't help if it's not linked to an interface. Qemu doesn't know how to change forwarding rules dynamically.

Since this is unsupported and broken feature anyway, but the original bug is fixed, I think we should close this bug.

Comment 10 liunana 2019-06-26 06:26:18 UTC
Hi,

Since the original bug is fixed I think we can change this status to VERIFIED as for this bug.
I will Verified this bug according to Comment 6 ~ Comment 9, any problem please tell me,thanks.


Best regards
Liu Nana

Comment 12 errata-xmlrpc 2019-11-06 07:12:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.