RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1436534 - It's very slow to boot guest with multiple virtio-serial-pci in specify host.
Summary: It's very slow to boot guest with multiple virtio-serial-pci in specify host.
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.4
Hardware: x86_64
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: pagupta
QA Contact: Qianqian Zhu
URL:
Whiteboard:
Depends On:
Blocks: 1473046
TreeView+ depends on / blocked
 
Reported: 2017-03-28 06:39 UTC by Peixiu Hou
Modified: 2018-12-13 08:22 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-13 07:57:00 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Peixiu Hou 2017-03-28 06:39:25 UTC 
Description of problem:
It's very slow to boot guest with multiple virtio-serial-pci in specify host. 

Version-Release number of selected component (if applicable):
kernel-3.10.0-603.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot up guest with 2 virtio-serial-pci
/usr/libexec/qemu-kvm \
-name 'win2016-serial-ovmf-pc' \
-nodefaults  \
-vga std \
-m 3G  \
-smp 4  \
-enable-kvm \
-usb -device usb-tablet \
-device ahci,id=ahci \
-drive file=rhel74-64-virtio.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ahci.0,drive=drive-ide0-0-0,id=ide0-0-0 \
-drive file=en_windows_server_2016_x64_dvd_9327751.iso,if=none,readonly=on,media=cdrom,format=raw,id=cdrom1 -device ide-drive,drive=cdrom1,bus=ahci.1,id=ide-cd1 \
-drive file=virtio-win-prewhql-133.iso,if=none,readonly=on,media=cdrom,format=raw,id=cdrom2 -device ide-drive,drive=cdrom2,bus=ahci.2,id=ide-cd2 \
-rtc base=localtime,clock=host,driftfix=slew  \
-boot order=cd,menu=on \
-vnc :1 \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:1235,server,nowait \
-netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 \
-device e1000,netdev=hostnet0,id=net0,mac=00:52:0a:5c:f1:1a \
 -device virtio-serial-pci,id=virtio-serial0,max_ports=511 -chardev socket,id=channel0,path=/tmp/helloworld0,server,nowait -device virtserialport,chardev=channel0,name=com.redhat.rhevm.vdsm0,bus=virtio-serial0.0,id=port0 -device virtio-serial-pci,id=virtio-serial1,max_ports=511 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial1.0,id=port1 

2.boot up guest with 5 virtio-serial-pci
 -device virtio-serial-pci,id=virtio-serial0,max_ports=511 -chardev socket,id=channel0,path=/tmp/helloworld0,server,nowait -device virtserialport,chardev=channel0,name=com.redhat.rhevm.vdsm0,bus=virtio-serial0.0,id=port0 -device virtio-serial-pci,id=virtio-serial1,max_ports=511 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm1,bus=virtio-serial1.0,id=port1 -device virtio-serial-pci,id=virtio-serial2,max_ports=511 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm2,bus=virtio-serial2.0,id=port2 -device virtio-serial-pci,id=virtio-serial3,max_ports=511 -chardev socket,id=channel3,path=/tmp/helloworld3,server,nowait -device virtserialport,chardev=channel3,name=com.redhat.rhevm.vdsm3,bus=virtio-serial3.0,id=port3 -device virtio-serial-pci,id=virtio-serial4,max_ports=511 -chardev socket,id=channel4,path=/tmp/helloworld4,server,nowait -device virtserialport,chardev=channel4,name=com.redhat.rhevm.vdsm4,bus=virtio-serial4.0,id=port4

3.boot up guest with 8 virtio-serial-pci
4.boot up guest with 10 virtio-serial-pci
5.boot up guest with 29 virtio-serial-pci

Actual results:
after step 1, it took 1m to boot up.
after step 2, it took almost 2m to boot up.
after step 3, it took almost 5m to boot up.
after step 4, it took almost 10m to boot up.
after step 5, it took more than 1h to boot up.

and qmp output:
QEMU 2.8.0 monitor - type 'help' for more information
(qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).


Expected results:
guest can boot up quickly with multi virtio-serail-pci.

Additional info:
1. host info:
processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 47
model name	: Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz
stepping	: 2
microcode	: 0x37
cpu MHz		: 2129.000
cache size	: 24576 KB
physical id	: 3
siblings	: 16
core id		: 25
cpu cores	: 8
apicid		: 243
initial apicid	: 243
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt aes lahf_lm ida arat epb dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 4255.89
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management:

2. 
[root@ibm-x3850x5-07 home]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 47
Model name:            Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz
Stepping:              2
CPU MHz:               2129.000
BogoMIPS:              4255.89
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              24576K
NUMA node0 CPU(s):     0-7,16-23
NUMA node1 CPU(s):     8-15,24-31
Comment 2 pagupta 2017-03-29 08:09:57 UTC 
Hi,

"qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)"
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

This error is coming because enough open file descriptors are not available.
Can you please check if increasing the open file descriptors solves the issue 
for you?

Thanks,
Pankaj
Comment 3 Peixiu Hou 2017-06-27 08:46:01 UTC 
Sorry for late to request, I tried to increase the open file descriptors through add "ulimit -n 16384" to /etc/profile, the error ""qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)"
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower)"" disappeared. 

But the guest boot so slowly also can be reproduced on my localhos, Boot guest with 29 virtio-serial-pci, boot command as "-device virtio-serial-pci,id=virtio-serial$i,max_ports=511.

Also keep with 29 virtio-serial-pci, change the boot device command to "-device virtio-serial-pci,id=virtio-serial$i" it can be booted quilckly.

the host cpu info as follows:
1. [root@localhost qemu-14]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Model name:            Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
Stepping:              7
CPU MHz:               3700.023
BogoMIPS:              6784.80
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

2. [root@localhost qemu-14]# cat /proc/cpuinfo
processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
stepping	: 7
microcode	: 0x29
cpu MHz		: 3686.476
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
bogomips	: 6784.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

used version:
kernel-3.10.0-680.el7.x86_64
qemu-img-rhev-2.9.0-14.el7.x86_64
virtio-win-prewhql-137
Comment 4 pagupta 2017-07-26 09:51:33 UTC 
(In reply to Peixiu Hou from comment #3)
> Sorry for late to request, I tried to increase the open file descriptors
> through add "ulimit -n 16384" to /etc/profile, the error ""qemu-kvm:
> virtio_bus_set_host_notifier: unable to init event notifier: Too many open
> files (-24)"
> qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace
> (slower)"" disappeared. 
> 
> But the guest boot so slowly also can be reproduced on my localhos, Boot
> guest with 29 virtio-serial-pci, boot command as "-device
> virtio-serial-pci,id=virtio-serial$i,max_ports=511.
> 
> Also keep with 29 virtio-serial-pci, change the boot device command to
> "-device virtio-serial-pci,id=virtio-serial$i" it can be booted quilckly.

Can you please check if 'max_ports' value is <=31 works fine?

Thanks,
Pankaj

> 
> the host cpu info as follows:
> 1. [root@localhost qemu-14]# lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                8
> On-line CPU(s) list:   0-7
> Thread(s) per core:    2
> Core(s) per socket:    4
> Socket(s):             1
> NUMA node(s):          1
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 42
> Model name:            Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
> Stepping:              7
> CPU MHz:               3700.023
> BogoMIPS:              6784.80
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              8192K
> NUMA node0 CPU(s):     0-7
> 
> 2. [root@localhost qemu-14]# cat /proc/cpuinfo
> processor	: 7
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 42
> model name	: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
> stepping	: 7
> microcode	: 0x29
> cpu MHz		: 3686.476
> cache size	: 8192 KB
> physical id	: 0
> siblings	: 8
> core id		: 3
> cpu cores	: 4
> apicid		: 7
> initial apicid	: 7
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 13
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
> pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
> aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
> ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes
> xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm
> ida arat pln pts
> bogomips	: 6784.80
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 36 bits physical, 48 bits virtual
> power management:
> 
> used version:
> kernel-3.10.0-680.el7.x86_64
> qemu-img-rhev-2.9.0-14.el7.x86_64
> virtio-win-prewhql-137
Comment 5 Peixiu Hou 2017-07-27 02:45:49 UTC 
(In reply to pagupta from comment #4)
> (In reply to Peixiu Hou from comment #3)
> > Sorry for late to request, I tried to increase the open file descriptors
> > through add "ulimit -n 16384" to /etc/profile, the error ""qemu-kvm:
> > virtio_bus_set_host_notifier: unable to init event notifier: Too many open
> > files (-24)"
> > qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace
> > (slower)"" disappeared. 
> > 
> > But the guest boot so slowly also can be reproduced on my localhos, Boot
> > guest with 29 virtio-serial-pci, boot command as "-device
> > virtio-serial-pci,id=virtio-serial$i,max_ports=511.
> > 
> > Also keep with 29 virtio-serial-pci, change the boot device command to
> > "-device virtio-serial-pci,id=virtio-serial$i" it can be booted quilckly.
> 
> Can you please check if 'max_ports' value is <=31 works fine?
> 
> Thanks,
> Pankaj
> 

Tried with 'max_ports' value is <=31, booted in guest quickly.
Also tried with 'max_ports=511', reproduced this issue, booted in guest so slowly.

Best Regards~
Peixiu Hou
Comment 10 pagupta 2018-12-13 07:57:00 UTC 
For virtio-serial we support 'max_ports' <=31, see comment 5. So, we should use use maximum limit for ports. libvirt should take care of this.
I am closing this BZ.

Thanks,
Pankaj
Comment 11 juzhang 2018-12-13 08:22:26 UTC 
(In reply to pagupta from comment #10)
> For virtio-serial we support 'max_ports' <=31, see comment 5. So, we should
> use use maximum limit for ports. libvirt should take care of this.
> I am closing this BZ.
> 
> Thanks,
> Pankaj

Well noted, then kvm qe will disable max_ports>31 test cases accordingly.

Best regards,

Junyi
Note You need to log in before you can comment on or make changes to this bug.