Bug 1043403

Summary: can not connect to the virtio serial port on host side
Product: Red Hat Enterprise Linux 7 Reporter: Mike Cao <bcao>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: bcao, hhuang, juzhang, knoel, lijin, mdeng, michen, qzhang, rbalakri, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-08 07:26:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
strace.txt none

Description Mike Cao 2013-12-16 08:49:50 UTC
Description of problem:


Version-Release number of selected component (if applicable):
3.10.0-50.el7.x86_64
qemu-kvm-rhev-1.5.3-19.el7.x86_64
virtio-win-1.6.7-2

How reproducible:
20%

Steps to Reproduce:
1.Start VM w/ virtio-serial-port
2./usr/libexec/qemu-kvm -M pc-i440fx-rhel7.0.0 -cpu SandyBridge -enable-kvm -m 2G -smp 2,cores=2 -name bcao_win-7-32-netkvm -uuid 884e673a-1b4a-4385-a522-b3cc35ef4e18 -rtc base=localtime,clock=host,driftfix=slew -drive file=win8-64.qcow2,if=none,media=disk,serial=aaabbbccc,werror=stop,rerror=stop,cache=none,format=qcow2,id=drive-disk0 -device ide-drive,bus=ide.0,unit=1,physical_block_size=4096,logical_block_size=512,drive=drive-disk0,id=disk0 -drive file=/root/en_windows_8_enterprise_x64_dvd_917522.iso,if=none,media=cdrom,id=aa -device ide-drive,id=aa1,drive=aa,bootindex=1 -drive file=/usr/share/virtio-win/virtio-win.iso,media=cdrom,if=none,id=bb -device ide-drive,id=bb1,drive=bb -netdev tap,vhost=on,id=netdev0 -device e1000,netdev=netdev0,id=nic1,mac=1a:46:0b:ca:bc:7b -vnc :1 -vga std -monitor stdio -usb -device usb-tablet,id=tablet0 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -monitor unix:/tmp/tt,server,nowait -device virtio-serial-pci,id=virtio-serial0,max_ports=16 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1
3.nc -U /tmp/helloworld1

Actual results:
# nc -U /tmp/helloworld1
Ncat: Invalid argument.

Expected results:
It should be connected successfully

Additional info:
not 100% reproducible

Comment 2 Amit Shah 2013-12-17 06:14:39 UTC
nc says it's an invalid argument; I tend to think it was a result of a typo in the socket name.  Since you can't reproduce it easily, I'm tempted to just close->notabug.  But I'll wait for a few more days for confirmation, or a recurrence of this bug.  Please keep an eye out for it, and test it a few more times.

Comment 3 Mike Cao 2013-12-17 06:18:40 UTC
(In reply to Amit Shah from comment #2)
> nc says it's an invalid argument; I tend to think it was a result of a typo
> in the socket name.  Since you can't reproduce it easily, I'm tempted to
> just close->notabug.  But I'll wait for a few more days for confirmation, or
> a recurrence of this bug.  Please keep an eye out for it, and test it a few
> more times.

It is not a typo.

After the bug occurs,I kill the VM ,and restart it w/ exactly same commandline ,then use nc -U to connect the exactly same socket ,the issue will gone 

It happened more or less 8 times during my vioserial functional test run ,so report this bug

Comment 4 Amit Shah 2013-12-17 07:02:28 UTC
OK.

We don't really know which syscall returns with -EINVAL in nc's invocation.  You could try strace'ing the nc instance next time.

BTW did qemu get enough time to initialize the socket?  i.e. are you issuing the nc command too quickly, perhaps even before qemu has had a change to init the chardev?  If you're doing this in a script or similar, please try adding a delay between qemu invocation and nc invocation.

Comment 5 Mike Cao 2013-12-17 08:00:47 UTC
(In reply to Amit Shah from comment #4)
> OK.
> 
> We don't really know which syscall returns with -EINVAL in nc's invocation. 
> You could try strace'ing the nc instance next time.
> 
Could you show me more detailed usage about it ?
> BTW did qemu get enough time to initialize the socket?  i.e. are you issuing
> the nc command too quickly, perhaps even before qemu has had a change to
> init the chardev?  If you're doing this in a script or similar, please try
> adding a delay between qemu invocation and nc invocation.

No ,I am doing it manually ,I a mmore I issue the nc command after qemu process start

Comment 6 Amit Shah 2014-01-07 08:03:32 UTC
(In reply to Mike Cao from comment #5)
> (In reply to Amit Shah from comment #4)
> > OK.
> > 
> > We don't really know which syscall returns with -EINVAL in nc's invocation. 
> > You could try strace'ing the nc instance next time.
> > 
> Could you show me more detailed usage about it ?

strace -o nc-strace -f -ff <regular nc cmdline>

Then attach all the nc-strace* files to the bug.

Comment 8 Mike Cao 2014-03-20 03:11:27 UTC
Reproduced it again with following steps ,but not 100% reproduce 

1.Transfering date from guest to host
in the host scripts like #for((;;)) ; do python serialhost-receive.py /tmp/075SRL201264VFE_port0 ;done
2.ctrl-c it 
3.ctrl-z it 

^Z
[1]+  Stopped                 python serial-host-receive.py /tmp/075SRL201264VFE_port0
[root@dhcp-9-181 guest-send]# killall -9 python
[root@dhcp-9-181 guest-send]# nc -U /tmp/075SRL201264VFE_port0
Ncat: Invalid argument.

Comment 9 Mike Cao 2014-03-20 03:11:50 UTC
3.10.0-113.el7.x86_64
qemu-kvm-rhev-1.5.3-53.el7.x86_64

Comment 11 Mike Cao 2014-07-15 09:57:25 UTC
ReTest this issue 
on 3.10.0-121.el7.x86_64
qemu-kvm-1.5.3-62.el7.x86_64

Steps same as comment #8,Still can reproduce this issue

Comment 12 Amit Shah 2014-07-15 10:09:01 UTC
Mike, a few questions:

What happens if you use udp sockets instead of unix or tcp on host chardev?

What do the python scripts do?

Did you manage to get a strace of the nc process as described in comment 6?

Comment 13 Mike Cao 2014-07-15 10:13:24 UTC
(In reply to Amit Shah from comment #12)
> Mike, a few questions:
> 
> What happens if you use udp sockets instead of unix or tcp on host chardev?

I will try to find the cmmandline to retry it 
> 
> What do the python scripts do?

For linux guests there is /dev/vport0p1 in the guest which can be used to read/write data successuflly while windows guest doesn't ,we use python scripts to transferring data from host to guest or guest to host 
> 
> Did you manage to get a strace of the nc process as described in comment 6?
Can you show me the exactly commands ?

Thanks,
Mike

Comment 14 Amit Shah 2014-07-15 10:18:38 UTC
(In reply to Mike Cao from comment #13)
> (In reply to Amit Shah from comment #12)
> > Mike, a few questions:
> > 
> > What happens if you use udp sockets instead of unix or tcp on host chardev?
> 
> I will try to find the cmmandline to retry it 
> > 
> > What do the python scripts do?
> 
> For linux guests there is /dev/vport0p1 in the guest which can be used to
> read/write data successuflly while windows guest doesn't ,we use python
> scripts to transferring data from host to guest or guest to host 

From comment 8, looks like you're using the scripts in the host?

Also, if you use ^Z, the process is only going to be frozen, and the sockets become unavailable for more connections.  That's the reason I'm asking you to try udp sockets.

> > Did you manage to get a strace of the nc process as described in comment 6?
> Can you show me the exactly commands ?

comment 6 has the cmdline; should be sufficient.

Comment 15 Mike Cao 2014-07-15 10:35:15 UTC
Created attachment 918116 [details]
strace.txt

Comment 16 Mike Cao 2014-07-21 07:51:42 UTC
(In reply to Amit Shah from comment #14)
> (In reply to Mike Cao from comment #13)
> > (In reply to Amit Shah from comment #12)
> > > Mike, a few questions:
> > > 
> > > What happens if you use udp sockets instead of unix or tcp on host chardev?
> > 
> > I will try to find the cmmandline to retry it 
> > > 
> > > What do the python scripts do?
> > 
> > For linux guests there is /dev/vport0p1 in the guest which can be used to
> > read/write data successuflly while windows guest doesn't ,we use python
> > scripts to transferring data from host to guest or guest to host 
> 
> From comment 8, looks like you're using the scripts in the host?
> 
> Also, if you use ^Z, the process is only going to be frozen, and the sockets
> become unavailable for more connections.  That's the reason I'm asking you
> to try udp sockets.

after ^Z , I use #fg to active it ,then close it

Comment 17 lijin 2014-09-01 08:02:00 UTC
test with virtio-win-prewhql-90 and qemu-kvm-0.12.1.2-2.441.el6.bz1027181.x86_64:

package info:
qemu-kvm-0.12.1.2-2.441.el6.bz1027181.x86_64
virtio-win-prewhql-90 
2.6.32-486.el6.x86_64

scenario 1:
steps:
1.boot guest with:
/usr/libexec/qemu-kvm -drive file=088SRLWIN864ODX,if=none,cache=none,media=disk,format=raw,id=drive-ide0-0-1 -device ide-drive,id=ide0-0-1,drive=drive-ide0-0-1,bootindex=0 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -spice disable-ticketing,port=5901 -vga qxl -global qxl-vga.revision=3 -usb -device usb-tablet -netdev tap,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:e2:71:29:46,bus=pci.0 -chardev file,path=/root/console.log,id=serial1 -monitor unix:/tmp/tt,server,nowait -device isa-serial,chardev=serial1,id=s1 -cpu Penryn,hv_relaxed,+sep -M rhel6.5.0 -smp 2,maxcpus=2,cores=2,threads=1,sockets=1 -m 2G -enable-kvm -cdrom /usr/share/virtio-win/virtio-win.iso -device virtio-serial-pci,id=virtio-serial0,max_ports=16 -chardev socket,path=/tmp/tt0,server,nowait,id=chardev0 -device virtserialport,chardev=chardev0,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port0,nr=1 -chardev socket,path=/tmp/tt1,server,nowait,id=chardev1 -device virtserialport,chardev=chardev1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1
2.nc -U /tmp/tt1
3.ctrl-c the nc
4.nc -U /tmp/tt1 again

Actual result:
during step2 and step4,nc -U /tmp/tt1 connected successfully

scenario 2:
steps:
1.boot guest the above command;
2.transfer data from guest to host:
eg: on the host # for ((;;)) ; do python serial-host-receive.py /tmp/tt1; done
    in the guest #for ((;;)) ; do python VirtIOChannel_guest_send.py com.redhat.rhevm.vdsm1;done
3.ctrl-c the script on host side
4.rerun the python script again to check  whether host side still can recieve data.

Actual result:
host can receive data during step2 and step4.

Comment 18 Amit Shah 2014-09-01 12:53:49 UTC
(In reply to lijin from comment #17)
> test with virtio-win-prewhql-90 and
> qemu-kvm-0.12.1.2-2.441.el6.bz1027181.x86_64:

This originally is a RHEL7 bug.  The fix in bug 1027181 is targeted for RHEL6.  RHEL7 already has the fix mentioned there.

Comment 20 Amit Shah 2015-09-08 07:26:17 UTC
Looks like a problem somewhere else.  If it still reproduces, please reopen.