Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 820226

Summary: cannot transmit string via serial port since -289
Product: Red Hat Enterprise Linux 6 Reporter: Chao Yang <chayang>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: acathrow, alevy, amit.shah, bsarathy, dyasny, juzhang, michen, mkenneth, qzhang, shuang, tburke, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-14 12:44:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs in /sys/kernel/debug none

Description Chao Yang 2012-05-09 12:29:51 UTC 
Description of problem:
Transmit a string from guest to host via /dev/vport0p1, hit:
-bash: /dev/vport0p1: Too many open files

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.292.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. in guest:
# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files


Actual results:


Expected results:


Additional info:
This issue not happen on -288
CLI:
/usr/libexec/qemu-kvm -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -name test -uuid abb4c266-0b98-44e2-9280-5ed4c6ac904e -rtc base=utc,clock=host,driftfix=slew -boot menu=on -drive file=/home/RHEL6.3-20120502.3-Server-x86_64-copy.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0,scsi=off -netdev tap,id=net -device virtio-net-pci,netdev=net,id=net0,mac=64:31:5a:b3:c9:11 -usb -spice port=8000,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -balloon none -device virtio-serial-pci,id=virtio-serial0,max_ports=16 -chardev socket,id=charchannel0,path=/tmp/serial.socket,server,nowait -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0 -bios /usr/share/seabios/bios-pm.bin
Comment 1 Chao Yang 2012-05-09 12:34:45 UTC 
This bug blocks verification of Bug 816099 on qemu-kvm -289 and later.
Comment 4 Amit Shah 2012-05-10 04:45:56 UTC 
(In reply to comment #0)
> Description of problem:
> Transmit a string from guest to host via /dev/vport0p1, hit:
> -bash: /dev/vport0p1: Too many open files

This means some other process has the port open in the guest.  Ensure there's nothing else using the port.

Also start the guest with more ports and check if other ports also show similar behaviour.
Comment 5 Amit Shah 2012-05-10 10:03:54 UTC 
Alon could reproduce this.

It doesn't reproduce upstream, so this is specific to the backport or other RHEL patches.
Comment 6 Alon Levy 2012-05-10 10:12:21 UTC 
correction: I cannot reproduce it, but I'd like to suggest a possible cause.

what does "lsof /dev/vport0p1" show? if it shows something, then you will get the "Too many open files" error. This is consistent in 288 and 289, it's a feature, not a bug ;)

When I thought I reproduced I had spice-vdagent running and using the same port, killing it solved the problem.

Alon
Comment 7 Chao Yang 2012-05-10 12:39:55 UTC 
Hi Amit, Alon,
 I tried much more times, and have some info to update you here:
on qemu-292: 
------------
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  2776) is running...
[root@localhost ~]# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files
[root@localhost ~]# lsof /dev/vport0p1 <---- it returns nothing
[root@localhost ~]# echo $?
1
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  2776) is running...
[root@localhost ~]# service spice-vdagentd restart
Stopping spice-vdagentd:                                   [  OK  ]
Starting spice-vdagentd:                                   [  OK  ]
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo $?
1
[root@localhost ~]# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files
[root@localhost ~]# 


on qemu-288:
------------
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  1716) is running...
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo $?
1
[root@localhost ~]# echo aaa > /dev/vport0p1 
-bash: /dev/vport0p1: Too many open files
[root@localhost ~]# service spice-vdagentd restart
Stopping spice-vdagentd:                                   [  OK  ]
Starting spice-vdagentd:                                   [  OK  ]
[root@localhost ~]# service spice-vdagentd status
spice-vdagentd (pid  2590) is running...
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo aaa > /dev/vport0p1 
[root@localhost ~]# lsof /dev/vport0p1 
[root@localhost ~]# echo $?
1
[root@localhost ~]# echo aaa > /dev/vport0p1 
[root@localhost ~]# echo $?
0

At least, this is not a regression cause I am hitting this error both on -288
and -292 now. It was lucky for me that didn't find this error on -288 but on -292. I now suspect the spice-vgagentd(spice-vdagent-0.8.1-3.el6.x86_64,
2.6.32-268.el6.x86_64).
Comment 8 Chao Yang 2012-05-10 12:40:47 UTC 
Removing "Regression".
Comment 9 Amit Shah 2012-05-10 13:07:57 UTC 
Waiting for answers to questions in comment 5.
Comment 10 Amit Shah 2012-05-10 13:28:11 UTC 
Please try out a couple more things.

1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different terminal.  Check if lsof shows the port is open.  Try this on a different port if you end up getting the same error on vport0p1.

2. Check and upload the values in debugfs: /sys/kernel/debug/virtio_console/*.  You'll have to mount debugfs first in /sys/kernel/debug/.
Comment 11 Chao Yang 2012-05-11 03:39:40 UTC 
(In reply to comment #10)
> Please try out a couple more things.
> 
> 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> terminal.  Check if lsof shows the port is open. Try this on a different port
> if you end up getting the same error on vport0p1.
> 
cat /dev/vport0p1 shows the same error, lsof in a different terminal returns nothing.
cat /dev/vport0p2 does not the error, but lsof in a different terminal returns nothing too.
> 2. Check and upload the values in debugfs: /sys/kernel/debug/virtio_console/*. 
> You'll have to mount debugfs first in /sys/kernel/debug/.
Please see the attachment.
Comment 12 Chao Yang 2012-05-11 03:40:28 UTC 
Created attachment 583710 [details]
logs in /sys/kernel/debug
Comment 13 Amit Shah 2012-05-11 06:06:50 UTC 
(In reply to comment #11)
> > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > terminal.  Check if lsof shows the port is open. Try this on a different port
> > if you end up getting the same error on vport0p1.
> > 
> cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> nothing.


> cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> nothing too.

So this means some other process is indeed holding /dev/vport0p1 open.  Can't say which one it is, since lsof isn't showing the process name.

From the command line,

> -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0

Can you change the name to something else?  Can you change the name and try again?

> > 2. Check and upload the values in debugfs: /sys/kernel/debug
/virtio_console/*. 
> > You'll have to mount debugfs first in /sys/kernel/debug/.
> Please see the attachment.

All the files in the attachment are empty.  I just need contents of /sys/kernel/debug/virtio-ports/*, so just individually cat'ting them and pasting output here will be OK.
Comment 14 Chao Yang 2012-05-11 14:42:06 UTC 
(In reply to comment #13)
> (In reply to comment #11)
> > > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > > terminal.  Check if lsof shows the port is open. Try this on a different port
> > > if you end up getting the same error on vport0p1.
> > > 
> > cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> > nothing.
> 
> 
> > cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> > nothing too.
> 
> So this means some other process is indeed holding /dev/vport0p1 open.  Can't
> say which one it is, since lsof isn't showing the process name.
> 
> From the command line,
> 
> > -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0
> 
> Can you change the name to something else?  Can you change the name and try
> again?
> 
I will update soon. 
> > > 2. Check and upload the values in debugfs: /sys/kernel/debug
> /virtio_console/*. 
> > > You'll have to mount debugfs first in /sys/kernel/debug/.
> > Please see the attachment.
> 
> All the files in the attachment are empty.  I just need contents of
> /sys/kernel/debug/virtio-ports/*, so just individually cat'ting them and
> pasting output here will be OK.
Sorry for my mistake. I thought they would be in the compressed file. I remember the sent/received data was 0 in virtio-ports/vport0p1, but not 0 in virtio/vport0p2. I will retest this and attach the files you need.
Comment 15 Chao Yang 2012-05-14 12:32:06 UTC 
(In reply to comment #13)
> (In reply to comment #11)
> > > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > > terminal.  Check if lsof shows the port is open. Try this on a different port
> > > if you end up getting the same error on vport0p1.
> > > 
> > cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> > nothing.
> 
> 
> > cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> > nothing too.
> 
> So this means some other process is indeed holding /dev/vport0p1 open.  Can't
> say which one it is, since lsof isn't showing the process name.
> 
This time lsof says:
# lsof /dev/vport0p1 
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
spice-vda 1738 root    9u   CHR  249,1      0t0 10628 /dev/vport0p1
# lsof /dev/vport0p2
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
cat     2596 root    3r   CHR  249,2      0t0 10636 /dev/vport0p2


> From the command line,
> 
> > -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0
> 
> Can you change the name to something else?  Can you change the name and try
> again?
> 
I am not sure how changing the name to something else works, but from my test, spice-vdagentd is stopped after changing the name to something else. And cat /dev/vport0p1 didn't get the error.

> > > 2. Check and upload the values in debugfs: /sys/kernel/debug
> /virtio_console/*. 
> > > You'll have to mount debugfs first in /sys/kernel/debug/.
> > Please see the attachment.
> 
# cat vport0p1 
name: com.redhat.spice.0
guest_connected: 1
host_connected: 1
outvq_full: 0
bytes_sent: 72
bytes_received: 12
bytes_discarded: 0
is_console: no

# cat vport0p2
name: com.redhat.spice.1
guest_connected: 1
host_connected: 1
outvq_full: 0
bytes_sent: 0
bytes_received: 5
bytes_discarded: 0
is_console: no
console_vtermno: 1


Host nc cli:
# nc -U /tmp/serial.socket
gg


aaa

ccc
g^C

# nc -U /tmp/serial-1.socket 
bbbb


qemu-kvm cli:
# /usr/libexec/qemu-kvm -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -name test -uuid abb4c266-0b98-44e2-9280-5ed4c6ac904e -rtc base=utc,clock=host,driftfix=slew -boot menu=on -drive file=/home/rhel6.3-64.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0,scsi=off -netdev tap,id=net -device virtio-net-pci,netdev=net,id=net0,mac=64:31:5a:b3:c9:11 -usb -spice port=8000,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -monitor stdio -balloon none -device virtio-serial-pci,id=virtio-serial0,max_ports=16 -chardev socket,id=charchannel0,path=/tmp/serial.socket,server,nowait -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/tmp/serial-1.socket,server,nowait -device virtserialport,bus=virtio-serial0.0,chardev=charchannel1,id=channel2,name=com.redhat.spice.1 -bios /usr/share/seabios/bios-pm.bin
Comment 16 Amit Shah 2012-05-14 12:44:19 UTC 
(In reply to comment #15)
> (In reply to comment #13)
> > (In reply to comment #11)
> > > > 1. Keep the port open (cat /dev/vport0p1) and then issue lsof in a different
> > > > terminal.  Check if lsof shows the port is open. Try this on a different port
> > > > if you end up getting the same error on vport0p1.
> > > > 
> > > cat /dev/vport0p1 shows the same error, lsof in a different terminal returns
> > > nothing.
> > 
> > 
> > > cat /dev/vport0p2 does not the error, but lsof in a different terminal returns
> > > nothing too.
> > 
> > So this means some other process is indeed holding /dev/vport0p1 open.  Can't
> > say which one it is, since lsof isn't showing the process name.
> > 
> This time lsof says:
> # lsof /dev/vport0p1 
> COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> spice-vda 1738 root    9u   CHR  249,1      0t0 10628 /dev/vport0p1
> # lsof /dev/vport0p2
> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> cat     2596 root    3r   CHR  249,2      0t0 10636 /dev/vport0p2

Aha, so there you have it.  spice-vdagent is holding the port open.  Hence, you get that error.

> > From the command line,
> > 
> > > -device virtserialport,bus=virtio-serial0.0,chardev=charchannel0,id=channel1,name=com.redhat.spice.0
> > 
> > Can you change the name to something else?  Can you change the name and try
> > again?
> > 
> I am not sure how changing the name to something else works, but from my test,
> spice-vdagentd is stopped after changing the name to something else. And cat
> /dev/vport0p1 didn't get the error.

spice-vdagent only opens /dev/virtio-ports/com.redhat.spice.0, not /dev/vport0p1 directly.  Once you change the name, spice thinks there's no port meant for it initiated, so it doesn't open any port.

> > > > 2. Check and upload the values in debugfs: /sys/kernel/debug
> > /virtio_console/*. 
> > > > You'll have to mount debugfs first in /sys/kernel/debug/.
> > > Please see the attachment.
> > 
> # cat vport0p1 
> name: com.redhat.spice.0
> guest_connected: 1

Since lsof showed the name of the process, this is redundant, but we can see from here that guest is connected, meaning some process has the port open.

So this isn't really a bug, and everything works as expected.