Bug 1468437 - [Power9]Boot up guest with multiple virtio-serial-pcis some errors pop up from hmp
[Power9]Boot up guest with multiple virtio-serial-pcis some errors pop up fro...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.4
All Unspecified
medium Severity medium
: rc
: ---
Assigned To: Laurent Vivier
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-07 01:42 EDT by Min Deng
Modified: 2017-07-11 09:23 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-11 02:24:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
The output requested by dev (2.32 MB, text/plain)
2017-07-10 04:18 EDT, Min Deng
no flags Details

  None (edit)
Description Min Deng 2017-07-07 01:42:51 EDT
Description of problem:
Boot up guest with multiple virtio-serial-pci some errors pop up from hmp
Version-Release number of selected component (if applicable):
Host,
kernel-4.11.0-10.el7a.ppc64le
qemu-kvm-2.9.0-16.el7a.ppc64le
Guest,(kernel-4.11.0-10.el7a.ppc64le)
RHEL-ALT-7.4-20170626.4-Server-ppc64le-dvd1.iso
How reproducible:
2 times
Steps to Reproduce:
1.boot up guest with the following cli,
  /usr/libexec/qemu-kvm -name mdeng -sandbox off -machine pseries-rhel7.4.0 -nodefaults -vga none -chardev socket,id=serial_id_serial0,path=/tmp/tt,server,nowait -device spapr-vty,reg=0x30000000,chardev=serial_id_serial0 -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=newfull.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:2b:2c:2d:2e:2f,id=id6b5tKj,vectors=4,netdev=idXB7qte,bus=pci.0,addr=0x5 -netdev tap,id=idXB7qte,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-down,id=hostnet1 -m 8192 -smp 8,cores=4,threads=1,sockets=2 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host -enable-kvm -monitor stdio -qmp tcp:0:4444,server,nowait -device virtio-serial-pci,id=virtio-serial0,max_ports=31 -chardev socket,id=channel0,path=/tmp/channel0,server,nowait -device virtserialport,chardev=channel0,name=org.linux-kvm.port.0,bus=virtio-serial0.0,id=port1 -device virtio-serial-pci,id=virtio-serial1,max_ports=31 -chardev socket,id=channel1,path=/tmp/channel1,server,nowait -device virtserialport,chardev=channel1,name=org.linux-kvm.port.1,bus=virtio-serial0.0,id=port2 -device virtio-serial-pci,id=virtio-serial1002,max_ports=31 -device virtio-serial-pci,id=virtio-serial3,max_ports=31 -device virtio-serial-pci,id=virtio-serial4,max_ports=31 -device virtio-serial-pci,id=virtio-serial5,max_ports=31 -device virtio-serial-pci,id=virtio-serial6,max_ports=31 -device virtio-serial-pci,id=virtio-serial7,max_ports=31 -device virtio-serial-pci,id=virtio-serial8,max_ports=31 -device virtio-serial-pci,id=virtio-serial9,max_ports=31 -device virtio-serial-pci,id=virtio-serial10,max_ports=31 -device virtio-serial-pci,id=virtio-serial11,max_ports=31 -device virtio-serial-pci,id=virtio-serial12,max_ports=31 -device virtio-serial-pci,id=virtio-serial2,max_ports=31 -device virtio-serial-pci,id=virtio-serial13,max_ports=31 -device virtio-serial-pci,id=virtio-serial14,max_ports=31 -device virtio-serial-pci,id=virtio-serial15,max_ports=31 -device virtio-serial-pci,id=virtio-serial16,max_ports=31 -device virtio-serial-pci,id=virtio-serial17,max_ports=31 -device virtio-serial-pci,id=virtio-serial18,max_ports=31 -device virtio-serial-pci,id=virtio-serial19,max_ports=31

2.do some actions 
  info qtree,
  info ***
  system_reset
3.

Actual results:
There was error message 
...
(qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

Expected results:
No error.

Additional info:
Comment 2 Min Deng 2017-07-07 01:56:50 EDT
QE also tried it on x86 platform but I didn't meet the issue so far.
Build info,
qemu-kvm-rhev-2.9.0-14.el7.x86_64
and
qemu-kvm-2.9.0-16.el7a.x86_64
Comment 3 Laurent Vivier 2017-07-07 07:49:59 EDT
According to the error message "Too many open files", you should check the result of "ulimit -n".

But it looks file descriptors are not closed correctly on reset, and re-opening them exceeds the limit.  When the problem appears, could you provide the result of "lsof"?
Comment 4 Min Deng 2017-07-10 04:18 EDT
Created attachment 1295726 [details]
The output requested by dev
Comment 5 Min Deng 2017-07-10 04:21:07 EDT
(In reply to Laurent Vivier from comment #3)
> According to the error message "Too many open files", you should check the
> result of "ulimit -n".
  It was 1024 on my host.
> But it looks file descriptors are not closed correctly on reset, and
> re-opening them exceeds the limit.  When the problem appears, could you
> provide the result of "lsof"?
  Please see attachment.
Comment 6 Laurent Vivier 2017-07-10 07:28:46 EDT
Thanks.

I'm able to reproduce the problem on the first boot, so it's not a cleanup problem, on a POWER8 host with RHEL7.4, so this is not specific to pegas.

I don't have the problem if I increase the ulimit of the number of open files.

I've reproduced the problem on x86 with qemu-kvm-rhev-2.9.0-9.el7.x86_64:

IMAGE=/var/lib/libvirt/images/rhel7.4.qcow2
SERIAL=""
for id in $(seq 0 19)  ; do
        SERIAL="$SERIAL -device virtio-serial-pci,id=virtio-serial$id,max_ports=31"
done

/usr/libexec/qemu-kvm -name laurent-vm -sandbox off -M pc \
        -nodefaults -m 2G \
        -enable-kvm \
        -chardev stdio,id=conmon,mux=on,signal=off \
        -mon conmon \
        -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
        -device virtio-serial-pci \
        -device virtconsole,chardev=conmon \
        -rtc base=localtime,clock=host \
        -smp 8,cores=4,threads=1,sockets=2 \
        -device virtio-net-pci,mac=9a:d4:d5:d6:d7:a9,id=idkdMjSW,netdev=hostnet0 \
        -netdev bridge,id=hostnet0,br=virbr0 \
        -device virtio-scsi-pci,id=scsi0  \
        -device scsi-hd,id=disk,bus=scsi0.0,bootindex=1,drive=drive-disk0 \
        -drive file=$IMAGE,format=qcow2,if=none,id=drive-disk0,werror=stop,rerror=stop \
        -device usb-kbd,id=input0 \
        -device usb-mouse,id=input1 \
        -device VGA -vnc :11\
        $SERIAL

I'm going to check in the source code how many eventfd should be opened in this case to see if the error is expected.
Comment 7 Laurent Vivier 2017-07-10 08:58:33 EDT
It seems we need (nb_virtio_serial_card * (2 + max_ports * 2)) eventfd, this explains why we overflow the limit of allowed open files.

Paolo, is this correct?
Comment 9 Paolo Bonzini 2017-07-10 17:55:46 EDT
Yes, this sounds right.
Comment 10 Laurent Vivier 2017-07-11 02:24:42 EDT
So I close the BZ as NOTABUG.

If you want to test this case, increase the limit with ulimit.
Comment 11 Laurent Vivier 2017-07-11 09:23:39 EDT
BTW, it is a duplicate of BZ1271060

Note You need to log in before you can comment on or make changes to this bug.