Bug 820226 - cannot transmit string via serial port since -289
cannot transmit string via serial port since -289
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.3
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Amit Shah
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-09 08:29 EDT by Chao Yang
Modified: 2013-01-09 19:56 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-14 08:44:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs in /sys/kernel/debug (43.24 KB, application/x-gzip)
2012-05-10 23:40 EDT, Chao Yang
no flags Details

  None (edit)
Description Chao Yang 2012-05-09 08:29:51 EDT
Description of problem:
Transmit a string from guest to host via /dev/vport0p1, hit:
-bash: /dev/vport0p1: Too many open files

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.292.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. in guest:
# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files


Actual results:


Expected results:


Additional info:
This issue not happen on -288
CLI:
/usr/libexec/qemu-kvm -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -name test -uuid abb4c266-0b98-44e2-9280-5ed4c6ac904e -rtc base=utc,clock=host,driftfix=slew -boot menu=on -drive file=/home/RHEL6.3-20120502.3-Server-x86_64-copy.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0,scsi=off -netdev tap,id=net -device virtio-net-pci,netdev=net,id=net0,mac=64:31:5a:b3:c9:11 -usb -spice port=8000,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -balloon none -device virtio-serial-pci,id=virtio-serial0,max_ports=16 -chardev socket,id=charchannel0,path=/tmp/serial.socket,server,nowait -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0 -bios /usr/share/seabios/bios-pm.bin
Comment 1 Chao Yang 2012-05-09 08:34:45 EDT
This bug blocks verification of Bug 816099 on qemu-kvm -289 and later.
Comment 4 Amit Shah 2012-05-10 00:45:56 EDT
(In reply to comment #0)
> Description of problem:
> Transmit a string from guest to host via /dev/vport0p1, hit:
> -bash: /dev/vport0p1: Too many open files

This means some other process has the port open in the guest.  Ensure there's nothing else using the port.

Also start the guest with more ports and check if other ports also show similar behaviour.
Comment 5 Amit Shah 2012-05-10 06:03:54 EDT
Alon could reproduce this.

It doesn't reproduce upstream, so this is specific to the backport or other RHEL patches.
Comment 6 Alon Levy 2012-05-10 06:12:21 EDT
correction: I cannot reproduce it, but I'd like to suggest a possible cause.

what does "lsof /dev/vport0p1" show? if it shows something, then you will get the "Too many open files" error. This is consistent in 288 and 289, it's a feature, not a bug ;)

When I thought I reproduced I had spice-vdagent running and using the same port, killing it solved the problem.

Alon
Comment 7 Chao Yang 2012-05-10 08:39:55 EDT
Hi Amit, Alon,
 I tried much more times, and have some info to update you here:
on qemu-292: 
------------
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  2776) is running...
[root@localhost ~]# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files
[root@localhost ~]# lsof /dev/vport0p1 <---- it returns nothing
[root@localhost ~]# echo $?
1
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  2776) is running...
[root@localhost ~]# service spice-vdagentd restart
Stopping spice-vdagentd:                                   [  OK  ]
Starting spice-vdagentd:                                   [  OK  ]
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo $?
1
[root@localhost ~]# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files
[root@localhost ~]# 


on qemu-288:
------------
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  1716) is running...
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo $?
1
[root@localhost ~]# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files
[root@localhost ~]# service spice-vdagentd restart
Stopping spice-vdagentd:                                   [  OK  ]
Starting spice-vdagentd:                                   [  OK  ]
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  2590) is running...
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo aaa > /dev/vport0p1 
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo $?
1
[root@localhost ~]# echo aaa > /dev/vport0p1 
[root@localhost ~]# echo $?
0

At least, this is not a regression cause I am hitting this error both on -288
and -292 now. It was lucky for me that didn't find this error on -288 but on -292. I now suspect the spice-vgagentd(spice-vdagent-0.8.1-3.el6.x86_64,
2.6.32-268.el6.x86_64).
Comment 8 Chao Yang 2012-05-10 08:40:47 EDT
Removing "Regression".
Comment 9 Amit Shah 2012-05-10 09:07:57 EDT
Waiting for answers to questions in comment 5.
Comment 10 Amit Shah 2012-05-10 09:28:11 EDT
Please try out a couple more things.

1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different terminal.  Check if lsof shows the port is open.  Try this on a different port if you end up getting the same error on vport0p1.

2. Check and upload the values in debugfs: /sys/kernel/debug/virtio_console/*.  You'll have to mount debugfs first in /sys/kernel/debug/.
Comment 11 Chao Yang 2012-05-10 23:39:40 EDT
(In reply to comment #10)
> Please try out a couple more things.
> 
> 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> terminal.  Check if lsof shows the port is open. Try this on a different port
> if you end up getting the same error on vport0p1.
> 
cat /dev/vport0p1 shows the same error, lsof in a different terminal returns nothing.
cat /dev/vport0p2 does not the error, but lsof in a different terminal returns nothing too.
> 2. Check and upload the values in debugfs: /sys/kernel/debug/virtio_console/*. 
> You'll have to mount debugfs first in /sys/kernel/debug/.
Please see the attachment.
Comment 12 Chao Yang 2012-05-10 23:40:28 EDT
Created attachment 583710 [details]
logs in /sys/kernel/debug
Comment 13 Amit Shah 2012-05-11 02:06:50 EDT
(In reply to comment #11)
> > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > terminal.  Check if lsof shows the port is open. Try this on a different port
> > if you end up getting the same error on vport0p1.
> > 
> cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> nothing.


> cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> nothing too.

So this means some other process is indeed holding /dev/vport0p1 open.  Can't say which one it is, since lsof isn't showing the process name.

From the command line,

> -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0

Can you change the name to something else?  Can you change the name and try again?

> > 2. Check and upload the values in debugfs: /sys/kernel/debug
/virtio_console/*. 
> > You'll have to mount debugfs first in /sys/kernel/debug/.
> Please see the attachment.

All the files in the attachment are empty.  I just need contents of /sys/kernel/debug/virtio-ports/*, so just individually cat'ting them and pasting output here will be OK.
Comment 14 Chao Yang 2012-05-11 10:42:06 EDT
(In reply to comment #13)
> (In reply to comment #11)
> > > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > > terminal.  Check if lsof shows the port is open. Try this on a different port
> > > if you end up getting the same error on vport0p1.
> > > 
> > cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> > nothing.
> 
> 
> > cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> > nothing too.
> 
> So this means some other process is indeed holding /dev/vport0p1 open.  Can't
> say which one it is, since lsof isn't showing the process name.
> 
> From the command line,
> 
> > -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0
> 
> Can you change the name to something else?  Can you change the name and try
> again?
> 
I will update soon. 
> > > 2. Check and upload the values in debugfs: /sys/kernel/debug
> /virtio_console/*. 
> > > You'll have to mount debugfs first in /sys/kernel/debug/.
> > Please see the attachment.
> 
> All the files in the attachment are empty.  I just need contents of
> /sys/kernel/debug/virtio-ports/*, so just individually cat'ting them and
> pasting output here will be OK.
Sorry for my mistake. I thought they would be in the compressed file. I remember the sent/received data was 0 in virtio-ports/vport0p1, but not 0 in virtio/vport0p2. I will retest this and attach the files you need.
Comment 15 Chao Yang 2012-05-14 08:32:06 EDT
(In reply to comment #13)
> (In reply to comment #11)
> > > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > > terminal.  Check if lsof shows the port is open. Try this on a different port
> > > if you end up getting the same error on vport0p1.
> > > 
> > cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> > nothing.
> 
> 
> > cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> > nothing too.
> 
> So this means some other process is indeed holding /dev/vport0p1 open.  Can't
> say which one it is, since lsof isn't showing the process name.
> 
This time lsof says:
# lsof /dev/vport0p1 
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
spice-vda 1738 root    9u   CHR  249,1      0t0 10628 /dev/vport0p1
# lsof /dev/vport0p2
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
cat     2596 root    3r   CHR  249,2      0t0 10636 /dev/vport0p2


> From the command line,
> 
> > -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0
> 
> Can you change the name to something else?  Can you change the name and try
> again?
> 
I am not sure how changing the name to something else works, but from my test, spice-vdagentd is stopped after changing the name to something else. And cat /dev/vport0p1 didn't get the error.

> > > 2. Check and upload the values in debugfs: /sys/kernel/debug
> /virtio_console/*. 
> > > You'll have to mount debugfs first in /sys/kernel/debug/.
> > Please see the attachment.
> 
# cat vport0p1 
name: com.redhat.spice.0
guest_connected: 1
host_connected: 1
outvq_full: 0
bytes_sent: 72
bytes_received: 12
bytes_discarded: 0
is_console: no

# cat vport0p2
name: com.redhat.spice.1
guest_connected: 1
host_connected: 1
outvq_full: 0
bytes_sent: 0
bytes_received: 5
bytes_discarded: 0
is_console: no
console_vtermno: 1


Host nc cli:
# nc -U /tmp/serial.socket
gg


aaa

ccc
g^C

# nc -U /tmp/serial-1.socket 
bbbb


qemu-kvm cli:
# /usr/libexec/qemu-kvm -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -name test -uuid abb4c266-0b98-44e2-9280-5ed4c6ac904e -rtc base=utc,clock=host,driftfix=slew -boot menu=on -drive file=/home/rhel6.3-64.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0,scsi=off -netdev tap,id=net -device virtio-net-pci,netdev=net,id=net0,mac=64:31:5a:b3:c9:11 -usb -spice port=8000,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -balloon none -device virtio-serial-pci,id=virtio-serial0,max_ports=16 -chardev socket,id=charchannel0,path=/tmp/serial.socket,server,nowait -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/tmp/serial-1.socket,server,nowait -device virtserialport,bus=virtio-serial0.0,chardev=charchannel1,id=channel2,name=com.redhat.spice.1 -bios /usr/share/seabios/bios-pm.bin
Comment 16 Amit Shah 2012-05-14 08:44:19 EDT
(In reply to comment #15)
> (In reply to comment #13)
> > (In reply to comment #11)
> > > > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > > > terminal.  Check if lsof shows the port is open. Try this on a different port
> > > > if you end up getting the same error on vport0p1.
> > > > 
> > > cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> > > nothing.
> > 
> > 
> > > cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> > > nothing too.
> > 
> > So this means some other process is indeed holding /dev/vport0p1 open.  Can't
> > say which one it is, since lsof isn't showing the process name.
> > 
> This time lsof says:
> # lsof /dev/vport0p1 
> COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> spice-vda 1738 root    9u   CHR  249,1      0t0 10628 /dev/vport0p1
> # lsof /dev/vport0p2
> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> cat     2596 root    3r   CHR  249,2      0t0 10636 /dev/vport0p2

Aha, so there you have it.  spice-vdagent is holding the port open.  Hence, you get that error.

> > From the command line,
> > 
> > > -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0
> > 
> > Can you change the name to something else?  Can you change the name and try
> > again?
> > 
> I am not sure how changing the name to something else works, but from my test,
> spice-vdagentd is stopped after changing the name to something else. And cat
> /dev/vport0p1 didn't get the error.

spice-vdagent only opens /dev/virtio-ports/com.redhat.spice.0, not /dev/vport0p1 directly.  Once you change the name, spice thinks there's no port meant for it initiated, so it doesn't open any port.

> > > > 2. Check and upload the values in debugfs: /sys/kernel/debug
> > /virtio_console/*. 
> > > > You'll have to mount debugfs first in /sys/kernel/debug/.
> > > Please see the attachment.
> > 
> # cat vport0p1 
> name: com.redhat.spice.0
> guest_connected: 1

Since lsof showed the name of the process, this is redundant, but we can see from here that guest is connected, meaning some process has the port open.

So this isn't really a bug, and everything works as expected.

Note You need to log in before you can comment on or make changes to this bug.