Bug 992900 - Remove pending watches after virtserialport unplug
Remove pending watches after virtserialport unplug
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Amit Shah
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-05 03:01 EDT by Amit Shah
Modified: 2014-06-17 23:33 EDT (History)
7 users (show)

See Also:
Fixed In Version: qemu-kvm-1.5.2-3.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 09:08:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Amit Shah 2013-08-05 03:01:33 EDT
Description of problem:

Backport upstream commit 203439ce0a832e36b276f10892846bd91ee836eb

This commit ensures unplug of virtserialports removes pending watches.  Watches are callbacks that get registered with the chardev layer for flow control.

To reproduce and test, these steps might show the problem:
1. Start a listening process on the host, but don't read from the host, e.g. using a python program
2. send lots of data from the guest, till the guest can't send anymore due to queues being full.
3. hot-unplug the port (but not the chardev)

After step 3, qemu will try to reference structures in the port that was unplugged.


commit 203439ce0a832e36b276f10892846bd91ee836eb
Author: Andreas Färber
Date:   Thu Aug 1 01:28:46 2013 +0200

    virtio-console: Use exitfn for virtserialport, too
    
    virtconsole and virtserialport are identical in every other aspect
    except for the distinguishing VirtIOSerialPortClass::is_console field.
Comment 1 Min Deng 2013-08-06 22:23:43 EDT
Hi Amit,
   Could you please help to check if I reproduce the bug correctly,thanks in advance.
Boot up guest with CLI
Build info
qemu-kvm-1.5.1-2.el7.x86_64
kernel-3.10.0-1.el7.x86_64
Steps,
1./usr/libexec/qemu-kvm -cpu Opteron_G3 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -name usb-device -uuid `uuidgen` -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0 -chardev file,id=channel1,path=/home/helloworld1.txt -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1,nr=1 -drive file=rhel64-new.raw,if=none,id=drive-system-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,serial=QEMU-DISK1 -device ide-hd,bus=ide.0,unit=0,drive=drive-system-disk,id=system-disk,bootindex=2 -netdev tap,sndbuf=0,id=hostnet0,script=/etc/qemu-ifup,downscript=no -device e1000,netdev=hostnet0,mac=00:15:65:01:3a:20 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -vnc :1 -monitor stdio 
2.python
[root@dhcp-10-127 home]# python
Python 2.7.5 (default, May 28 2013, 19:05:42) 
[GCC 4.8.0 20130510 (Red Hat 4.8.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open ('/home/helloworld1.txt','rw')
3.send data from guest to host 
  I choose a file about 2G 
  cat file > /dev/vport0p1
4.Hot-unplug the port
  the port from monitor
  device_del port1
5.cat file > /dev/vport0p1 in guest again
6.on host
>>> f.read()
After step 6,I will get the following message
e|2\xa6\x04\x0b\x8a\xb0"\x97"\xc7GZ\xd9K\xd9\xdb7\x18\xe0\x9ew\x15\x04aN\xfdeY\xd8\xa2\x976\xea=\x84\x02r\x19\xa8\x1d\xb7j\xf6\xc3%\xff\xa1\xf6!\x00\xbaR\xb0b\x9c\x8f\x7f\xf2\xb6\x8d\xf4\xd7}u\xc8\x01\x952\x03\xe5u\xbf<\xa0\xa2\xbeJ\xe4\xfb\xa0\xde5J\xd2Ge{\xdc,\xfe\x06:\xc3\x0c\xe03\xe4\xdc\xec\x0c\xb8\x8e\xdb\x1d\x1fo\xc2g\x00R-f\xa6\xee\xfc\xa9\x10\x87\xfe\x7f\xf4\x13\xce\\~d\x99W\xb7\xa6\xb4\xd5\x11\xb1\xddW(\x82\x0e"\xea\xd8V\xdfm\x9f\\cb\xfe\xb9\xca\x04\x0c\xa3\xba\xac\x01<\x17\t\xd9R\x125\xe9e\x9da\rX\xef[\x84\x03\x19\xb0\xb4\xed\x11\x16\xf8\x9b\xc7:\xf7\xb7>ZaT\xbf\x15\xd4\x99\x98.\x06\xac\\\xf2\x86j\xb0\x7f\xe6c\x19R\xaf\xb0\x97\xf9\x9esdp\xd8\x9c\xe2Rk\xc0\x90\xdc-,X\x80\x995\xc6\xfd=\x19\xcf\x04\n\xf4\xe23B\xeav\'pMB$@t\x92\xf5\x1fsf\x15\xde\x0e3mO\xff\xf1/\x11B\xc3^NJ\x0cir\xf8\x8c\xe1.Bo\x07RZ1\x1b?\x96\xba\x9d"%\xe2\xf4B\xc9Y\xac\x95\xd3qx\x04+1.>\xd2\xc1w\xfd\x1f|\xba\xa7"\t\xbd\x08s\x10\x9a\xac\x9c<4]P\xe7\x86=\x7f|\xbfw\x92\x00[\xd8\x96R\x13s\xc7\x^Ze9\x01x\x11\xd1\xdcn\x99i\xdb!\xbd\xfc[\x03%\x1f\t\xe6\xa0ei<)?<l\xc7\xb0\xc6\x8c\x9c)L+5~\xb5)\xe5\xb4[zD\xf2\xca\xfa\xff\xb3\x1fu}\xa7x\xc9Y_h\xe6}\xa9?\xc8\x1c\x9cl[?)\x0f\xdekr\xe5\xc1\xb4Q$<oK\x91F\x9b\x1b\x1d\x10sY\xbd\xbe\xeeT\xa3L\xca\xeae\x8b\xb6\x11\xbb\xd1<2_\xb7\xb2t\r\xa5\xce7\x1c\x17\xfe\xdb\xd6\x18\xaf\x87 \xf7\n\x11B\x05\xab\x11tV\xfb\xf8L\x99\xd8\t\xa6\xf2\x1b\x14T\xd93e[h\x7f\xbf\xb7r&\xfb\xed\xc3U/\x91\xb2\x8c\x0f:Xh\x96=\x87M\xa1\xfde:H\xc1\xdc\xd7\x8akd\xdb9\xd6\x85\xdc\xf4\xc4\xcb4-\xb6\xe3\xcb\xa37)X\xfe\xce\x93\xa9\xce\x7f$| \x01\x00\tcN\xe2\x0f&\x1e\xbecvk\x839\x04\n\x974\x82\x98\x10@\xfa\x13\r\xa8\x14\xda",A\x8do@1\xaa\x02\xa95\xf2\xe3\x87\xb1\xa6k\x1aAn*"7\x1a\xfe\x97\xf6\no\x98\xc7\xb8\xf35\x9c\x86\x82`\xf7\xff\xbe\xc2A\xbc\xce\xc5\x92\xf9\xbe\xa0\x97\x95M1\x9cyx\x9e\x106\x9f7\x95\x8d@\xd9!\x1dc\xb9\xea\xf9\xc4-\xeew\xf7\xd8\x97\x13l9Ml\xc6\xd9&z\x8e\xe3\xe4\x1e)\xac\xcb\x00\x93S\xdc\x94\xf4s\x97\x9c!_[\x162<$\xb6\xd0\x81v\x05r\x1c \x7f\xe6\x97ff\xea4=}kw\xff\x8c\xb746\x84\x92;\xe6

Best Regards,
Min Deng
Comment 2 Min Deng 2013-08-07 02:23:37 EDT
Per Amit,adjust the above steps 
Remove step 5
Update step 6 
       while True:
             f.read()
Comment 3 Miroslav Rezanina 2013-08-09 07:43:33 EDT
Fix included in qemu-kvm-1.5.2-3.el7
Comment 4 mazhang 2014-01-06 01:38:54 EST
Hi Amit,

I can reproduce this bug with qemu-kvm-1.5.1-2.el7.x86_64 as comment 1 steps.
But after update to qemu-kvm-1.5.3-30.el7.x86_64, test result the same as older.
so could you provide some suggestions for verify this bug?

Thanks,
Mazhang.
Comment 6 juzhang 2014-01-13 02:19:41 EST
Hi Amit,

Could you have a look comment4 and add your comment?

Best Regards,
Junyi
Comment 7 Amit Shah 2014-01-13 03:31:42 EST
(In reply to mazhang from comment #4)
> Hi Amit,
> 
> I can reproduce this bug with qemu-kvm-1.5.1-2.el7.x86_64 as comment 1 steps.
> But after update to qemu-kvm-1.5.3-30.el7.x86_64, test result the same as
> older.
> so could you provide some suggestions for verify this bug?

Can you say how did you reproduce?  What exactly happens?

Also, note that you shouldn't read from the host socket; just an open call, without any reads.
Comment 8 mazhang 2014-01-13 22:02:17 EST
Hi Amit,

Here is my steps for reproduce.

Host:
qemu-kvm-1.5.1-2.el7.x86_64
kernel-3.10.0-66.el7.x86_64

Guest:
kernel-3.10.0-64.el7.x86_64

1. Start qemu-kvm with following command line:
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1,maxcpus=16 \
-enable-kvm \
-name rhel6-64 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:6666,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-monitor unix:/tmp/guest-sock,server,nowait \
-device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0 \
-chardev file,id=channel1,path=/home/helloworld1.txt \
-device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1,nr=1 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmp,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control  \
-drive file=/home/rhel7-64.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2c \
-vga qxl \
-spice port=5900,disable-ticketing \

2. Start a listening process on the host
(host)
[root@localhost ~]# python
Python 2.7.5 (default, Nov  6 2013, 23:28:41) 
[GCC 4.8.2 20131020 (Red Hat 4.8.2-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open ('/home/helloworld1.txt','rw')
>>> 

3. Send lots of data from the guest.
(guest)
[root@unused ~]# ll win7.iso 
-rw-r--r--. 1 root root 2563039232 Jan  6 07:16 win7.iso
[root@unused ~]# ll /dev/vport0p1 
crw-------. 1 root root 249, 1 Jan 14  2014 /dev/vport0p1
[root@unused ~]# cat win7.iso >/dev/vport0p1 
[root@unused ~]# echo $?
0


4. Hot unplug port1 after step 3 finished.

5. Read data from host.
(host)
>>> f.read()
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x01\x02\x00\x93\x00\x00\x00i9\xe0\x00\x05\x04\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xad5\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00B\x1a\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00*Microsoft CDIMAGE UDF\x00\x06\x00\x00\x00\x00\x00\x00\x00u\x02\x00\x00\x00\x00\x00\x008\x00\x00\x00\x08\x00\x00\x00\x06\x01\x02\x007\x00\x00\x00,\xf1\x08\x00\x05\x04\x00\x008\x00\x00\x008\x00\x00\x00\x05\x00\x00\x00\x01\x00\x00\x00 \x00\x00\x00\x0c\x00\x00\x00\x01\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00B\x1a\x00\x00\x0e\x8a\x12\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\

Lots of data output in step 5, was my steps correct? any suggestion will be appreciate.

Thanks,
Mazhang.
Comment 9 Amit Shah 2014-01-14 01:11:39 EST
Try this:

* use a unix or tcp socket for the chardev on the host

1. just open the chardev on the host (you'll have to use the 'socket' call in python; sample program at the end of this message[1])
2. send the file from the guest
3. hot-unplug port from host
4. read data from port in the python program (the while loop)

The data that the python program outputs would be valid data -- ideally you should see an error -- an abort or crash in qemu.

[1]
import socket

sock = socket.socket(socket.AF_UNIX)
sock.connect("/tmp/foo")

while 1:
	sock.recv(4096)
Comment 10 mazhang 2014-01-14 02:03:05 EST
(In reply to Amit Shah from comment #9)
> Try this:
> 
> * use a unix or tcp socket for the chardev on the host
> 
> 1. just open the chardev on the host (you'll have to use the 'socket' call
> in python; sample program at the end of this message[1])
> 2. send the file from the guest
> 3. hot-unplug port from host
> 4. read data from port in the python program (the while loop)
> 
> The data that the python program outputs would be valid data -- ideally you
> should see an error -- an abort or crash in qemu.
> 
> [1]
> import socket
> 
> sock = socket.socket(socket.AF_UNIX)
> sock.connect("/tmp/foo")
> 
> while 1:
> 	sock.recv(4096)

Try reproduce this bug with qemu-kvm-1.5.1-2.el7.x86_64.

1 Start qemu-kvm with virtio serial port.
...
-device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0 \
-chardev socket,id=channel1,path=/tmp/foo,server,nowait \
-device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 \

2 Open chardev in host
(host)
[root@localhost ~]# python
Python 2.7.5 (default, Nov  6 2013, 23:28:41) 
[GCC 4.8.2 20131020 (Red Hat 4.8.2-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import socket
>>> sock = socket.socket(socket.AF_UNIX)
>>> sock.connect("/tmp/foo")

3 Send a big file from guest to host.
(guest)
[root@unused ~]# cat win7.iso >/dev/vport0p1 

This command can not be finished in this time.

4 After transfer data about 2 minutes hot unplug port.

5 Read data in python.
>>> while 1:
>>>     sock.recv(4096)

Result:
Got lots of data after steps 5, qemu-kvm not abort or crash.
Comment 11 Amit Shah 2014-01-16 02:44:51 EST
That's the best recipe I have which can cause problems.  Looks like it doesn't reproduce, but the fix is obviously correct by looking at the code.  I'm sorry I can't provide any more information on how to observe badness.
Comment 12 mazhang 2014-01-16 03:11:55 EST
(In reply to Amit Shah from comment #11)
> That's the best recipe I have which can cause problems.  Looks like it
> doesn't reproduce, but the fix is obviously correct by looking at the code. 
> I'm sorry I can't provide any more information on how to observe badness.

Hi Amit,

Really appreciate for your help, I'll re-try it, and discuss with my manager how to verify this bug when he back.

Thanks,
Mazhang.
Comment 13 juzhang 2014-01-20 03:14:53 EST
Hi Mazhang,

Please have a try by using latest version and update the result in the bz.

Best Regards,
Junyi
Comment 14 mazhang 2014-01-21 22:28:46 EST
1. Downgrade to qemu-kvm-1.3.0-3.el7.x86_64 still not reproduce this problem.


2. Update to latest qemu-kvm and test this bug.
Host:
qemu-kvm-common-1.5.3-38.el7.x86_64
qemu-kvm-debuginfo-1.5.3-38.el7.x86_64
ipxe-roms-qemu-20130517-1.gitc4bce43.el7.noarch
qemu-img-1.5.3-38.el7.x86_64
qemu-kvm-tools-1.5.3-38.el7.x86_64
qemu-kvm-1.5.3-38.el7.x86_64
kernel-3.10.0-69.el7.x86_64

Guest:
kernel-3.10.0-64.el7.x86_64

Steps:
Same as comment#10.

Result:
After 5 times try, qemu-kvm works well, not abort or crash.
Comment 16 Ludek Smid 2014-06-13 09:08:43 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.