Bug 1027181

Summary: Guest can't receive any character transmitted from host after hot unplugging virtserialport then hot plugging again
Product: Red Hat Enterprise Linux 6 Reporter: FuXiangChun <xfu>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.5CC: amit.shah, areis, bcao, bsarathy, ghammer, hhuang, huding, jen, juzhang, knoel, lijin, lmiksik, mazhang, michen, mkenneth, mrezanin, qzhang, rbalakri, rhod, virt-maint, xfu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.443.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1038914 (view as bug list) Environment:
Last Closed: 2014-10-14 06:54:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1038914, 1056252    

Description FuXiangChun 2013-11-06 10:36:04 UTC
Description of problem:
Keep port open in guest, guest could receive characters sent from host via virtserialport, but hot unplug the port then hot plug it on same nr and same chardev, guest wouldn't receive any character any more, except:
1. set up connection from guest by echo aaa > /dev/vportXX
2. re-connect host by nc -U /path/to/unix-socket
3. re-send characters from host

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.412.el6.x86_64
2.6.32-416.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot rhel6.5 guest with this command line
/usr/libexec/qemu-kvm -M pc -enable-kvm -m 4G -smp 4,sockets=1,cores=4,threads=1 -name test -rtc base=localtime,clock=host,driftfix=slew -k en-us -boot menu=on -spice disable-ticketing,port=5930 -vga qxl -monitor stdio -drive file=/home/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:22:15:27:54:9d,bus=pci.0,addr=0x9   -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0 -chardev socket,id=charchannel0,path=/opt/chardev-testing.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm

2.transfer data from host to guest
2. On host:
# hexdump -C /dev/sda | nc -U /opt/chardev-testing.com.redhat.rhevm.vdsm

3.In guest, receive the data from host by:
# cat /dev/vport0p

4.hot unplug the virtserialport:
(qemu) device_del channel0

5. hot plug virtserialport:
(qemu) device_add virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm.com

6. cat /dev/vport0p1 in guest

Actual results:

after step6, guest cann't receive any data from host.


Expected results:
guest can receive data after step 6.


Additional info:

Comment 1 FuXiangChun 2013-11-06 10:43:35 UTC
since all hit this issue from qemu-kvm-0.12.1.2-2.412.el6.x86_64 to qemu-kvm-0.12.1.2-2.416.el6.x86_64.  but before qemu-kvm-0.12.1.2-2.412.el6.x86_64 version are good(don't hit this issue). so set regression to keywords

Comment 2 Amit Shah 2013-11-06 10:45:29 UTC
One question: what was the behaviour with qemu-kvm-0.12.1.2-2.367.el6 i.e. before bug 909059 was fixed?

If it was the same behaviour, it won't be a regression between releases, and can be explained by another bug.

Comment 3 Amit Shah 2013-11-06 10:46:58 UTC
For the record, the following was included in -411..-412:

0e741f93da41b39a6d5b4b24cf0e843bd7a31c48 os-posix: block SIGUSR2 in os_setup_early_signal_handling()
a9b3aa794a4e8acd8646db12ffc7897d7469ed9f char: remove watch callback on chardev detach from frontend
d5d63d84edf86a0312d56013dff7e94ad5f3daf4 char: use common function to disable callbacks on chardev close
fdc2f959b4c2370865a73f9df8a0dc4e2c26d31a char: move backends' io watch tag to CharDriverState

Comment 4 FuXiangChun 2013-11-06 10:51:41 UTC
Retest this issue with qemu-kvm-0.12.1.2-2.367.el6.x86_64.

After step6 in comments 0. guest can continue to receive data from host. so this qemu-kvm is good.

Comment 9 Gal Hammer 2013-11-13 15:28:20 UTC
(In reply to Amit Shah from comment #3)
> For the record, the following was included in -411..-412:
> 
> 0e741f93da41b39a6d5b4b24cf0e843bd7a31c48 os-posix: block SIGUSR2 in
> os_setup_early_signal_handling()
> a9b3aa794a4e8acd8646db12ffc7897d7469ed9f char: remove watch callback on
> chardev detach from frontend

It looks like this one cause the problem. The vport device removal is follow by the removal of the fd watch (qdev_free >> qemu_chr_add_handlers). The fd is not reconnected after the vport returns.

> d5d63d84edf86a0312d56013dff7e94ad5f3daf4 char: use common function to
> disable callbacks on chardev close
> fdc2f959b4c2370865a73f9df8a0dc4e2c26d31a char: move backends' io watch tag
> to CharDriverState

Comment 12 Gal Hammer 2013-11-25 08:14:59 UTC
I've submitted an upstream patch via qemu-devel.

Comment 16 huiqingding 2014-08-05 07:27:22 UTC
Reproduce this bug using the following version:
qemu-kvm-1.5.3-62.el7.x86_64
kernel-3.10.0-140.el7.x86_64

Steps to Reproduce:
1. boot a guest
#/usr/libexec/qemu-kvm -M pc -enable-kvm -m 4G -smp 4,sockets=1,cores=4,threads=1 -name test -rtc base=localtime,clock=host,driftfix=slew -k en-us -boot menu=on -spice disable-ticketing,port=5930 -vga qxl -monitor stdio -drive file=gluster://10.66.11.129/gv0/rhel7_1.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:22:15:27:54:9d,bus=pci.0,addr=0x9   -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0 -chardev socket,id=charchannel0,path=/opt/chardev-testing.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -qmp tcp:0:4445,server,nowait

2.transfer data from host to guest
On host:
# hexdump -C /dev/sda | nc -U /opt/chardev-testing.com.redhat.rhevm.vdsm

3.In guest, receive the data from host by:
# cat /dev/vport0p

4.hot unplug the virtserialport:
{"execute":"device_del","arguments":{"id":"channel0"}}

5. hot plug virtserialport:
{"execute":"device_add","arguments":{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"com.redhat.rhevm.vdsm"}}

6. cat /dev/vport0p1 in guest

Actual results:

after step6, guest cann't receive any data from host.

Comment 17 huiqingding 2014-08-05 07:34:46 UTC
Test this bug on an intel host using the following version:
qemu-kvm-1.5.3-66.el7.x86_64
kernel-3.10.0-140.el7.x86_64

Steps to Reproduce:
1. boot a RHEL7.1 guest
#/usr/libexec/qemu-kvm -M pc -enable-kvm -m 4G -smp 4,sockets=1,cores=4,threads=1 -name test -rtc base=localtime,clock=host,driftfix=slew -k en-us -boot menu=on -spice disable-ticketing,port=5930 -vga qxl -monitor stdio -drive file=gluster://10.66.11.129/gv0/rhel7_1.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:22:15:27:54:9d,bus=pci.0,addr=0x9   -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0 -chardev socket,id=charchannel0,path=/opt/chardev-testing.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -qmp tcp:0:4445,server,nowait

2.transfer data from host to guest
On host:
# hexdump -C /dev/sda | nc -U /opt/chardev-testing.com.redhat.rhevm.vdsm

3.In guest, receive the data from host by:
# cat /dev/vport0p1

4.hot unplug the virtserialport:
{"execute":"device_del","arguments":{"id":"channel0"}}

5. hot plug virtserialport:
{"execute":"device_add","arguments":{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"com.redhat.rhevm.vdsm"}}

6. cat /dev/vport0p1 in guest

Results:

after step6, guest can receive data from host.

Addtional test:
also test HMP command as comment 0, after step6, guest can recieve data from host.

Comment 18 huiqingding 2014-08-05 08:05:48 UTC
Sorry, please ignore comment 16 and comment 17, it is the test result of rhel7.1.

Comment 30 Qunfang Zhang 2014-09-01 01:49:01 UTC
We will verify the scratch build asap and then provide qa_ack+ after passed.

Comment 31 mazhang 2014-09-01 05:29:00 UTC
1. qemu-kvm-0.12.1.2-2.431.el6 and qemu-kvm-0.12.1.2-2.440.el6 can not reproduce this bug, qemu-kvm-0.12.1.2-2.415.el6 hit this problem.
2. Use the scratch build which pasted on comment#26 test this bug, the problem has gone.

Comment 32 mazhang 2014-09-01 05:52:34 UTC
Host:
kernel-2.6.32-497.el6.x86_64
qemu-img-0.12.1.2-2.441.el6.bz1027181.x86_64
qemu-kvm-0.12.1.2-2.441.el6.bz1027181.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.441.el6.bz1027181.x86_64
qemu-kvm-tools-0.12.1.2-2.441.el6.bz1027181.x86_64

Guest:
kernel-2.6.32-497.el6.x86_64

Steps:
1. Boot guest:
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1,maxcpus=160 \
-enable-kvm \
-name rhel6.6 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-monitor unix:/tmp/monitor2,server,nowait \
-vga qxl \
-spice port=5900,disable-ticketing \
-usb \
-device usb-tablet,id=tablet0 \
-device virtio-scsi-pci,id=si0 \
-drive file=/home/RHEL-Server-6.6-64.qcow2,if=none,media=disk,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native \
-device scsi-hd,drive=drive-scsi-disk,bus=si0.0,id=scsi-disk0,bootindex=0 \
-device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0 \
-chardev socket,id=charchannel0,path=/opt/chardev-testing.com.redhat.rhevm.vdsm,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm \

2.Transfer data from host to guest
host:
#hexdump -C /dev/sda | nc -U /opt/chardev-testing.com.redhat.rhevm.vdsm

3.Read data from serial port in guest
#cat /dev/vport0p1

4.hot unplug the virtserialport and stop transfer data.
(qemu) device_del channel0

5. hot plug virtserialport:
(qemu) device_add virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm.com

6. Repeat step2~3.

Result:
Get data from guest serial port after hotplug virtio serial port.

Comment 33 Qunfang Zhang 2014-09-01 06:02:08 UTC
Function test result will be updated after finish.

Comment 34 Miroslav Rezanina 2014-09-01 13:49:11 UTC
Fix included in qemu-kvm-0.12.1.2-2.443.el6

Comment 37 mazhang 2014-09-03 02:14:37 UTC
Verify this bug on qemu-kvm-0.12.1.2-2.443.el6.

Host:
qemu-kvm-tools-0.12.1.2-2.443.el6.x86_64
qemu-img-0.12.1.2-2.443.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.443.el6.x86_64
qemu-kvm-0.12.1.2-2.443.el6.x86_64
kernel-2.6.32-500.el6.x86_64

Guest:
RHEL-6.6-64
kernel-2.6.32-500.el6.x86_64
Win7-64
virtio-win-prewhql-0.1-90

Steps:
1. boot guest.
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1,maxcpus=160 \
-enable-kvm \
-name rhel6.6 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-monitor unix:/tmp/monitor2,server,nowait \
-vga qxl \
-spice port=5900,disable-ticketing \
-usb \
-device usb-tablet,id=tablet0 \
-device virtio-scsi-pci,id=si0 \
-drive file=/home/win7-64.qcow2,if=none,media=disk,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native \
-device scsi-hd,drive=drive-scsi-disk,bus=si0.0,id=scsi-disk0,bootindex=0 \
-device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0 \
-chardev socket,id=charchannel0,path=/tmp/helloworld2,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm \

2.Transfer data from host to guest
host:
#hexdump -C /dev/sda | nc -U /tmp/helloworld2

3.Read data from serial port in guest
#cat /dev/vport0p1

4.hot unplug the virtserialport and stop transfer data.
(qemu) device_del channel0

5. hot plug virtserialport:
(qemu) device_add virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm

6. Repeat step2~3.

Result:
Get data from guest serial port after hotplug virtio serial port.

Comment 39 errata-xmlrpc 2014-10-14 06:54:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html

Comment 40 Gal Hammer 2015-01-14 09:12:56 UTC
*** Bug 1025660 has been marked as a duplicate of this bug. ***