Bug 1429551 - Default FD limit prevents running more than ~470 VMs
Summary: Default FD limit prevents running more than ~470 VMs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: chhu
URL:
Whiteboard:
Depends On:
Blocks: 1442043
TreeView+ depends on / blocked
 
Reported: 2017-03-06 15:21 UTC by Eldad Marciano
Modified: 2019-04-28 13:39 UTC (History)
12 users (show)

Fixed In Version: libvirt-3.2.0-1.el7
Doc Type: Bug Fix
Doc Text:
On very large installations, libvirt could fail to start some guests, giving a "Too many open files error". The default open file handle limits have been substantially increased for the libvirtd, virtlockd, and virtlogd processes, allowing for several thousand guests on a host (if these generous limits are exceeded, the limits can be further increased in the systemd service files for the processes).
Clone Of:
: 1442043 (view as bug list)
Environment:
Last Closed: 2017-08-01 17:24:15 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Eldad Marciano 2017-03-06 15:21:17 UTC
Description of problem:
we faced some issues related to too many open files.

it looks like the issue comes form libvirt since the number of FDs is very close to libvirt limits which is 1024.

we raise something like ~470 vms with 2 disks, which means 940 disks on top of 2 storage domains (each disk on each SD).

the host spec is:
- 64 cores
- 377GB RAM


attaching the output of FDs while the issue occurs.
libvirtd:
[root]# readlink  /proc/$(pgrep -f libvirtd)/fd/* |wc -l
1015

vdsmd:
[root]# readlink  /proc/$(pgrep -f 'python2 /usr/share/vdsm/vdsm')/fd/* |wc -l
551

attaching the vdsm exception as well.
2017-03-05 11:21:20,972+0200 ERROR (vm/adb38c46) [virt.vm] (vmId='adb38c46-e810-4700-8d25-c6b636896a52') The vm start process failed (vm:617)
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 553, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/virt/vm.py", line 1998, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3777, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: Hook script execution failed: Failed to create pipe: Too many open files


Version-Release number of selected component (if applicable):
rhv 4.1.1-2
libvirt-client-2.0.0-10.el7_3.5.x86_64

How reproducible:
100%

Steps to Reproduce:
1. running 470 vms with 2 disks on top of 2 SD (iscsi)


Actual results:
some vms failed to start due to libvirtError: Hook script execution failed: Failed to create pipe: Too many open files

Expected results:
a tuned limits for large scale host.

Additional info:

Comment 2 Eldad Marciano 2017-03-06 15:23:38 UTC
adding kernel version:
3.10.0-514.10.2.el7.x86_64

RHEL version
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Comment 3 Peter Krempa 2017-03-06 15:44:12 UTC
Our configuration file states the following on increasing the file limit:

from /etc/sysconfig/libvirtd:
# Override the maximum number of opened files.
# This only works with traditional init scripts.
# In the systemd world, the limit can only be changed by overriding
# LimitNOFILE for libvirtd.service. To do that, just create a *.conf
# file in /etc/systemd/system/libvirtd.service.d/ (for example
# /etc/systemd/system/libvirtd.service.d/openfiles.conf) and write
# the following two lines in it:
#   [Service]
#   LimitNOFILE=2048


Is this what you are looking for?

Comment 4 Daniel Berrangé 2017-03-06 18:07:39 UTC
There's probably an argument to be made that the default libvirtd.service script should raise the number of FDs to a much larger number. eg I could see us raising it to something like 65536 by default - even if libvirtd misbehaves and mistakenly creates that many FDs when it shouldn't, the resource usage associated with it is negligible compared to resource usage of even 1 single VM we launch.

Comment 5 Eldad Marciano 2017-03-06 18:34:45 UTC
(In reply to Peter Krempa from comment #3)
> Our configuration file states the following on increasing the file limit:
> 
> from /etc/sysconfig/libvirtd:
> # Override the maximum number of opened files.
> # This only works with traditional init scripts.
> # In the systemd world, the limit can only be changed by overriding
> # LimitNOFILE for libvirtd.service. To do that, just create a *.conf
> # file in /etc/systemd/system/libvirtd.service.d/ (for example
> # /etc/systemd/system/libvirtd.service.d/openfiles.conf) and write
> # the following two lines in it:
> #   [Service]
> #   LimitNOFILE=2048
> 
> 
> Is this what you are looking for?

Probably yes, 
and per to comment #4 raise that number by default could be a good and quick solution, I just wonder if we should invest more time and effort for better understanding whether it is a FD's leak or not.
please advise

Comment 6 Nir Soffer 2017-03-06 18:45:15 UTC
ovirt configures libvirt, so we can add our own systemd libvirt dropin to raise
the number of open files, but I think the right place to fix this is in libvirt
defaults, so all users enjoy a better default.

Comment 7 Daniel Berrangé 2017-03-07 10:06:26 UTC
(In reply to Eldad Marciano from comment #5)
> Probably yes, 
> and per to comment #4 raise that number by default could be a good and quick
> solution, I just wonder if we should invest more time and effort for better
> understanding whether it is a FD's leak or not.

I don't think there's any leak there - libvirtd requires at least two file descriptors per VM & you have 470 VMs, so that'll be getting very close to the default 1024 limit.

Comment 8 Eldad Marciano 2017-03-07 11:53:53 UTC
(In reply to Daniel Berrange from comment #7)
> (In reply to Eldad Marciano from comment #5)
> > Probably yes, 
> > and per to comment #4 raise that number by default could be a good and quick
> > solution, I just wonder if we should invest more time and effort for better
> > understanding whether it is a FD's leak or not.
> 
> I don't think there's any leak there - libvirtd requires at least two file
> descriptors per VM & you have 470 VMs, so that'll be getting very close to
> the default 1024 limit.

Hmm make sense, Nir do we know that each vm take at least 2 file descriptors already ?

either way, as Nir suggested lets increase that default number in libvirt.

Comment 10 Daniel Berrangé 2017-03-15 16:44:03 UTC
Can you provide 'lsof -p $(pgrep libvirtd)' for a machine with many RHV guests, so we can just confirm the FDs libvirt has open does match expectations (eg 1 for qemu monitor and 1 for qemu agent).

Comment 11 Daniel Berrangé 2017-03-15 16:57:48 UTC
This patch proposes setting limits to allow for approx 4096 guests to run per guest, with a "typical" file handle usage scenario

https://www.redhat.com/archives/libvir-list/2017-March/msg00678.html

Comment 12 Peter Krempa 2017-03-16 14:44:28 UTC
The commit was pushed upstream:

commit 27cd76350021d36b9bd8b187ce5c8919659e3806
Author: Daniel P. Berrange <berrange@redhat.com>
Date:   Wed Mar 15 16:51:51 2017 +0000

    Increase default file handle limits for daemons
    
    Linux still defaults to a 1024 open file handle limit. This causes
    scalability problems for libvirtd / virtlockd / virtlogd on large
    hosts which might want > 1024 guest to be running. In fact if each
    guest needs > 1 FD, we can't even get to 500 guests. This is not
    good enough when we see machines with 100's of physical cores and
    TBs of RAM.
    
    In comparison to other memory requirements of libvirtd & related
    daemons, the resource usage associated with open file handles
    is essentially line noise. It is thus reasonable to increase the
    limits unconditionally for all installs.

Comment 20 chhu 2017-05-12 03:04:52 UTC
Verified on packages:
qemu-kvm-rhev-2.9.0-3.el7.x86_64
libvirt-3.2.0-4.el7.x86_64
kernel-3.10.0-663.el7.x86_64

1. Check the configuration files: PASS
#cat /usr/lib/systemd/system/libvirtd.service| grep LimitNOFILE
LimitNOFILE=8192
#cat /usr/lib/systemd/system/virtlogd.service| grep LimitNOFILE
LimitNOFILE=8192
#cat /usr/lib/systemd/system/virtlockd.service| grep LimitNOFILE
LimitNOFILE=16384

2. Check the service limit settings: PASS
#cat /proc/`pidof libvirtd`/limits| grep open
Max open files 8192 8192 files
#cat /proc/`pidof virtlogd`/limits| grep open
Max open files 8192 8192 files
#cat /proc/`pidof virtlockd`/limits| grep open
Max open files 16384 16384 files

3. Check the virtlockd limit:16384: PASS
Try to start 4100 guests with 4 disks in each guests, enable virtlockd service. Met error:"Too many open files", when near the virtlockd limit: 16384
Settings:
1) Edit /etc/libvirt/qemu.conf
lock_manager = "lockd"
max_processes = 65535
max_files = 65535

2) Edit /etc/libvirt/qemu-lockd.conf
auto_disk_leases = 1
file_lockspace_dir = "/var/lib/libvirt/lockd/files"

3) Edit max_clients in /etc/libvirt/virtlockd.conf
max_clients = 16385

4) #ulimit -n 65535
5) Restart service
#systemctl start virtlockd
#systemctl restart libvirtd

Steps:
1) Try to define and start 4100 guests. Met error:"Too many open files", when near the virtlockd limit: 16384

#virsh start r7_test3275
error: Failed to start domain r7_test3275
error: Unable to open/create resource /var/lib/libvirt/lockd/files/403e582c39cc88f45b039d796a9cc4ae9175dce9b5323667562d14543436cbac: Too many open files

#ls /proc/`pidof libvirtd`/fd/ | wc -l
3296
#ls /proc/`pidof virtlogd`/fd/ | wc -l
6558
#ls /proc/`pidof virtlockd`/fd/ | wc -l
16381

Comment 21 chhu 2017-05-12 03:05:23 UTC
4. Check the virtlogd limit: 8192: PASS
Try to start 2058 guests with xml below with 1 disk in each guest, hit error in libvirtd.log, when near the virtlogd limit 8192.
"error : virNetClientProgramDispatchError:177 : Unable to duplicate FD 8190: Too many open files"

1) xml with serial log and guest agent.
<serial type='file'>
<source path='/var/log/libvirt/serial_##.log' append='off'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>

#virsh start r7_test2045
error: Failed to start domain r7_test2045
error: Unable to duplicate FD 8190: Too many open files

2) check the limit, near virtlogd limit:8192
#ls /proc/`pidof virtlogd`/fd/ | wc -l
8186
#ls /proc/`pidof libvirtd`/fd/ | wc -l
2067
#ls /proc/`pidof virtlockd`/fd/ | wc -l
4099

Comment 22 chhu 2017-05-12 03:06:28 UTC
5. Check libvirtd limit: 8192: PASS
Try to start 8192 guests, with 1 disk in each guest, hit error in libvirtd.log, when near libvirtd limit: 8192.
"error : getDevNull:401 : cannot open /dev/null: Too many open files"

Settings:
1) Edit the virtlogd limit to 65535
#cat /usr/lib/systemd/system/virtlogd.service| grep LimitNOFILE
#LimitNOFILE=8192
LimitNOFILE=65535

2) check the limits:
#cat /proc/`pidof virtlockd`/limits| grep open
Max open files 16384 16384 files
#cat /proc/`pidof virtlogd`/limits| grep open
Max open files 65535 65535 files
#cat /proc/`pidof libvirtd`/limits| grep open
Max open files 8192 8192 files

Steps:
1) Try to start the 8192 guest with script. Failed to start the VM, and hit error "Too many open files" in libvirtd.log when near libvirtd limit: 8192
--------------------------------------------------------------------
error : getDevNull:401 : cannot open /dev/null: Too many open files
debug : virCommandRunAsync:2451 : Command result -1, with PID -1
debug : virFileClose:110 : Closed fd 8189
debug : virFileClose:110 : Closed fd 8190
debug : qemuProcessLaunch:5700 : QEMU vm=0x7fd689753a70 name=r7_test8161 failed to spawn
debug : qemuProcessLaunch:5703 : Writing early domain status to disk
debug : virFileMakePathHelper:2912 : path=/var/run/libvirt/qemu mode=0777
debug : virFileClose:110 : Closed fd 8189
debug : qemuProcessLaunch:5707 : Waiting for handshake from child
error : virCommandHandshakeWait:2672 : internal error: invalid use of command API
---------------------------------------------------------------

2) check the libvirtd open files near the limit 8192.
#ls /proc/`pidof libvirtd`/fd/ |wc -l
8183
#ls /proc/`pidof virtlockd`/fd/ |wc -l
16331
#ls /proc/`pidof virtlogd`/fd/ |wc -l
16330

3) virsh list --all check 8160 guests are running.
No error in: /var/log/libvirt/qemu/r7_test8161.log

Comment 23 chhu 2017-05-12 03:08:15 UTC
According to the comment20,21,22, set the bug status to VERIFIED.

Comment 24 errata-xmlrpc 2017-08-01 17:24:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 25 errata-xmlrpc 2017-08-02 00:03:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846


Note You need to log in before you can comment on or make changes to this bug.