Bug 992900
| Summary: | Remove pending watches after virtserialport unplug | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Amit Shah <amit.shah> |
| Component: | qemu-kvm | Assignee: | Amit Shah <amit.shah> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.0 | CC: | acathrow, amit.shah, hhuang, juzhang, mazhang, mdeng, virt-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-1.5.2-3.el7 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-06-13 13:08:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi Amit, Could you please help to check if I reproduce the bug correctly,thanks in advance. Boot up guest with CLI Build info qemu-kvm-1.5.1-2.el7.x86_64 kernel-3.10.0-1.el7.x86_64 Steps, 1./usr/libexec/qemu-kvm -cpu Opteron_G3 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -name usb-device -uuid `uuidgen` -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0 -chardev file,id=channel1,path=/home/helloworld1.txt -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1,nr=1 -drive file=rhel64-new.raw,if=none,id=drive-system-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,serial=QEMU-DISK1 -device ide-hd,bus=ide.0,unit=0,drive=drive-system-disk,id=system-disk,bootindex=2 -netdev tap,sndbuf=0,id=hostnet0,script=/etc/qemu-ifup,downscript=no -device e1000,netdev=hostnet0,mac=00:15:65:01:3a:20 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -vnc :1 -monitor stdio 2.python [root@dhcp-10-127 home]# python Python 2.7.5 (default, May 28 2013, 19:05:42) [GCC 4.8.0 20130510 (Red Hat 4.8.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> f = open ('/home/helloworld1.txt','rw') 3.send data from guest to host I choose a file about 2G cat file > /dev/vport0p1 4.Hot-unplug the port the port from monitor device_del port1 5.cat file > /dev/vport0p1 in guest again 6.on host >>> f.read() After step 6,I will get the following message e|2\xa6\x04\x0b\x8a\xb0"\x97"\xc7GZ\xd9K\xd9\xdb7\x18\xe0\x9ew\x15\x04aN\xfdeY\xd8\xa2\x976\xea=\x84\x02r\x19\xa8\x1d\xb7j\xf6\xc3%\xff\xa1\xf6!\x00\xbaR\xb0b\x9c\x8f\x7f\xf2\xb6\x8d\xf4\xd7}u\xc8\x01\x952\x03\xe5u\xbf<\xa0\xa2\xbeJ\xe4\xfb\xa0\xde5J\xd2Ge{\xdc,\xfe\x06:\xc3\x0c\xe03\xe4\xdc\xec\x0c\xb8\x8e\xdb\x1d\x1fo\xc2g\x00R-f\xa6\xee\xfc\xa9\x10\x87\xfe\x7f\xf4\x13\xce\\~d\x99W\xb7\xa6\xb4\xd5\x11\xb1\xddW(\x82\x0e"\xea\xd8V\xdfm\x9f\\cb\xfe\xb9\xca\x04\x0c\xa3\xba\xac\x01<\x17\t\xd9R\x125\xe9e\x9da\rX\xef[\x84\x03\x19\xb0\xb4\xed\x11\x16\xf8\x9b\xc7:\xf7\xb7>ZaT\xbf\x15\xd4\x99\x98.\x06\xac\\\xf2\x86j\xb0\x7f\xe6c\x19R\xaf\xb0\x97\xf9\x9esdp\xd8\x9c\xe2Rk\xc0\x90\xdc-,X\x80\x995\xc6\xfd=\x19\xcf\x04\n\xf4\xe23B\xeav\'pMB$@t\x92\xf5\x1fsf\x15\xde\x0e3mO\xff\xf1/\x11B\xc3^NJ\x0cir\xf8\x8c\xe1.Bo\x07RZ1\x1b?\x96\xba\x9d"%\xe2\xf4B\xc9Y\xac\x95\xd3qx\x04+1.>\xd2\xc1w\xfd\x1f|\xba\xa7"\t\xbd\x08s\x10\x9a\xac\x9c<4]P\xe7\x86=\x7f|\xbfw\x92\x00[\xd8\x96R\x13s\xc7\x^Ze9\x01x\x11\xd1\xdcn\x99i\xdb!\xbd\xfc[\x03%\x1f\t\xe6\xa0ei<)?<l\xc7\xb0\xc6\x8c\x9c)L+5~\xb5)\xe5\xb4[zD\xf2\xca\xfa\xff\xb3\x1fu}\xa7x\xc9Y_h\xe6}\xa9?\xc8\x1c\x9cl[?)\x0f\xdekr\xe5\xc1\xb4Q$<oK\x91F\x9b\x1b\x1d\x10sY\xbd\xbe\xeeT\xa3L\xca\xeae\x8b\xb6\x11\xbb\xd1<2_\xb7\xb2t\r\xa5\xce7\x1c\x17\xfe\xdb\xd6\x18\xaf\x87 \xf7\n\x11B\x05\xab\x11tV\xfb\xf8L\x99\xd8\t\xa6\xf2\x1b\x14T\xd93e[h\x7f\xbf\xb7r&\xfb\xed\xc3U/\x91\xb2\x8c\x0f:Xh\x96=\x87M\xa1\xfde:H\xc1\xdc\xd7\x8akd\xdb9\xd6\x85\xdc\xf4\xc4\xcb4-\xb6\xe3\xcb\xa37)X\xfe\xce\x93\xa9\xce\x7f$| \x01\x00\tcN\xe2\x0f&\x1e\xbecvk\x839\x04\n\x974\x82\x98\x10@\xfa\x13\r\xa8\x14\xda",A\x8do@1\xaa\x02\xa95\xf2\xe3\x87\xb1\xa6k\x1aAn*"7\x1a\xfe\x97\xf6\no\x98\xc7\xb8\xf35\x9c\x86\x82`\xf7\xff\xbe\xc2A\xbc\xce\xc5\x92\xf9\xbe\xa0\x97\x95M1\x9cyx\x9e\x106\x9f7\x95\x8d@\xd9!\x1dc\xb9\xea\xf9\xc4-\xeew\xf7\xd8\x97\x13l9Ml\xc6\xd9&z\x8e\xe3\xe4\x1e)\xac\xcb\x00\x93S\xdc\x94\xf4s\x97\x9c!_[\x162<$\xb6\xd0\x81v\x05r\x1c \x7f\xe6\x97ff\xea4=}kw\xff\x8c\xb746\x84\x92;\xe6 Best Regards, Min Deng Per Amit,adjust the above steps
Remove step 5
Update step 6
while True:
f.read()
Fix included in qemu-kvm-1.5.2-3.el7 Hi Amit, I can reproduce this bug with qemu-kvm-1.5.1-2.el7.x86_64 as comment 1 steps. But after update to qemu-kvm-1.5.3-30.el7.x86_64, test result the same as older. so could you provide some suggestions for verify this bug? Thanks, Mazhang. Hi Amit, Could you have a look comment4 and add your comment? Best Regards, Junyi (In reply to mazhang from comment #4) > Hi Amit, > > I can reproduce this bug with qemu-kvm-1.5.1-2.el7.x86_64 as comment 1 steps. > But after update to qemu-kvm-1.5.3-30.el7.x86_64, test result the same as > older. > so could you provide some suggestions for verify this bug? Can you say how did you reproduce? What exactly happens? Also, note that you shouldn't read from the host socket; just an open call, without any reads. Hi Amit, Here is my steps for reproduce. Host: qemu-kvm-1.5.1-2.el7.x86_64 kernel-3.10.0-66.el7.x86_64 Guest: kernel-3.10.0-64.el7.x86_64 1. Start qemu-kvm with following command line: /usr/libexec/qemu-kvm \ -M pc \ -cpu SandyBridge \ -m 4G \ -smp 4,sockets=2,cores=2,threads=1,maxcpus=16 \ -enable-kvm \ -name rhel6-64 \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \ -k en-us \ -rtc base=localtime,clock=host,driftfix=slew \ -nodefaults \ -monitor stdio \ -qmp tcp:0:6666,server,nowait \ -boot menu=on \ -bios /usr/share/seabios/bios.bin \ -monitor unix:/tmp/guest-sock,server,nowait \ -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0 \ -chardev file,id=channel1,path=/home/helloworld1.txt \ -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1,nr=1 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmp,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -drive file=/home/rhel7-64.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop,aio=threads \ -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2c \ -vga qxl \ -spice port=5900,disable-ticketing \ 2. Start a listening process on the host (host) [root@localhost ~]# python Python 2.7.5 (default, Nov 6 2013, 23:28:41) [GCC 4.8.2 20131020 (Red Hat 4.8.2-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> f = open ('/home/helloworld1.txt','rw') >>> 3. Send lots of data from the guest. (guest) [root@unused ~]# ll win7.iso -rw-r--r--. 1 root root 2563039232 Jan 6 07:16 win7.iso [root@unused ~]# ll /dev/vport0p1 crw-------. 1 root root 249, 1 Jan 14 2014 /dev/vport0p1 [root@unused ~]# cat win7.iso >/dev/vport0p1 [root@unused ~]# echo $? 0 4. Hot unplug port1 after step 3 finished. 5. Read data from host. (host) >>> f.read() 0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x01\x02\x00\x93\x00\x00\x00i9\xe0\x00\x05\x04\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xad5\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00B\x1a\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00*Microsoft CDIMAGE UDF\x00\x06\x00\x00\x00\x00\x00\x00\x00u\x02\x00\x00\x00\x00\x00\x008\x00\x00\x00\x08\x00\x00\x00\x06\x01\x02\x007\x00\x00\x00,\xf1\x08\x00\x05\x04\x00\x008\x00\x00\x008\x00\x00\x00\x05\x00\x00\x00\x01\x00\x00\x00 \x00\x00\x00\x0c\x00\x00\x00\x01\x00\x00\x00 \x1e\xda\x07\x0b\x14\x12\x11\x02\x00\x00\x00B\x1a\x00\x00\x0e\x8a\x12\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ Lots of data output in step 5, was my steps correct? any suggestion will be appreciate. Thanks, Mazhang. Try this:
* use a unix or tcp socket for the chardev on the host
1. just open the chardev on the host (you'll have to use the 'socket' call in python; sample program at the end of this message[1])
2. send the file from the guest
3. hot-unplug port from host
4. read data from port in the python program (the while loop)
The data that the python program outputs would be valid data -- ideally you should see an error -- an abort or crash in qemu.
[1]
import socket
sock = socket.socket(socket.AF_UNIX)
sock.connect("/tmp/foo")
while 1:
sock.recv(4096)
(In reply to Amit Shah from comment #9) > Try this: > > * use a unix or tcp socket for the chardev on the host > > 1. just open the chardev on the host (you'll have to use the 'socket' call > in python; sample program at the end of this message[1]) > 2. send the file from the guest > 3. hot-unplug port from host > 4. read data from port in the python program (the while loop) > > The data that the python program outputs would be valid data -- ideally you > should see an error -- an abort or crash in qemu. > > [1] > import socket > > sock = socket.socket(socket.AF_UNIX) > sock.connect("/tmp/foo") > > while 1: > sock.recv(4096) Try reproduce this bug with qemu-kvm-1.5.1-2.el7.x86_64. 1 Start qemu-kvm with virtio serial port. ... -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0 \ -chardev socket,id=channel1,path=/tmp/foo,server,nowait \ -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 \ 2 Open chardev in host (host) [root@localhost ~]# python Python 2.7.5 (default, Nov 6 2013, 23:28:41) [GCC 4.8.2 20131020 (Red Hat 4.8.2-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> import socket >>> sock = socket.socket(socket.AF_UNIX) >>> sock.connect("/tmp/foo") 3 Send a big file from guest to host. (guest) [root@unused ~]# cat win7.iso >/dev/vport0p1 This command can not be finished in this time. 4 After transfer data about 2 minutes hot unplug port. 5 Read data in python. >>> while 1: >>> sock.recv(4096) Result: Got lots of data after steps 5, qemu-kvm not abort or crash. That's the best recipe I have which can cause problems. Looks like it doesn't reproduce, but the fix is obviously correct by looking at the code. I'm sorry I can't provide any more information on how to observe badness. (In reply to Amit Shah from comment #11) > That's the best recipe I have which can cause problems. Looks like it > doesn't reproduce, but the fix is obviously correct by looking at the code. > I'm sorry I can't provide any more information on how to observe badness. Hi Amit, Really appreciate for your help, I'll re-try it, and discuss with my manager how to verify this bug when he back. Thanks, Mazhang. Hi Mazhang, Please have a try by using latest version and update the result in the bz. Best Regards, Junyi 1. Downgrade to qemu-kvm-1.3.0-3.el7.x86_64 still not reproduce this problem. 2. Update to latest qemu-kvm and test this bug. Host: qemu-kvm-common-1.5.3-38.el7.x86_64 qemu-kvm-debuginfo-1.5.3-38.el7.x86_64 ipxe-roms-qemu-20130517-1.gitc4bce43.el7.noarch qemu-img-1.5.3-38.el7.x86_64 qemu-kvm-tools-1.5.3-38.el7.x86_64 qemu-kvm-1.5.3-38.el7.x86_64 kernel-3.10.0-69.el7.x86_64 Guest: kernel-3.10.0-64.el7.x86_64 Steps: Same as comment#10. Result: After 5 times try, qemu-kvm works well, not abort or crash. This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request. |
Description of problem: Backport upstream commit 203439ce0a832e36b276f10892846bd91ee836eb This commit ensures unplug of virtserialports removes pending watches. Watches are callbacks that get registered with the chardev layer for flow control. To reproduce and test, these steps might show the problem: 1. Start a listening process on the host, but don't read from the host, e.g. using a python program 2. send lots of data from the guest, till the guest can't send anymore due to queues being full. 3. hot-unplug the port (but not the chardev) After step 3, qemu will try to reference structures in the port that was unplugged. commit 203439ce0a832e36b276f10892846bd91ee836eb Author: Andreas Färber Date: Thu Aug 1 01:28:46 2013 +0200 virtio-console: Use exitfn for virtserialport, too virtconsole and virtserialport are identical in every other aspect except for the distinguishing VirtIOSerialPortClass::is_console field.