Bug 2036195

Summary: Provide mechanism to tune the sysctl param fs.aio-max-nr on the host
Product: Red Hat OpenStack Reporter: Alex Stupnikov <astupnik>
Component: openstack-tripleo-heat-templatesAssignee: OSP Team <rhos-maint>
Status: CLOSED ERRATA QA Contact: Joe H. Rahme <jhakimra>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: alifshit, bdobreli, mburns, svigan
Target Milestone: z9Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20220601143346.29a02c1.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2061697 (view as bug list) Environment:
Last Closed: 2022-12-07 20:29:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Stupnikov 2021-12-30 11:13:08 UTC 
Description of problem:


Previously (for example, in RHOSP 13) libvirt-daemon RPM was installed on compute nodes and it provided /usr/lib/sysctl.d/60-libvirtd.conf configuration file [1] to increase fs.aio-max-nr from 65536 (default) to 1048576. libvirt-daemon RPM is no longer installed on RHOSP 16.1 nodes and default value is not changed there.

Documentation tells us that this number should be calculated as number of virtual disks on the host * 128. As a result default limit leaves us with 512 virtual disks, but KCS providing deployment limitations https://access.redhat.com/articles/1436373 tells us about 16384 KVM guests per hypervisor.

I am not sure how default value would behave in containerized environment, but this looks like a possible problem that requires second look.

[1]
[root@compute-1 ~]# cat /usr/lib/sysctl.d/60-libvirtd.conf 
# The kernel allocates aio memory on demand, and this number limits the
# number of parallel aio requests; the only drawback of a larger limit is
# that a malicious guest could issue parallel requests to cause the kernel
# to set aside memory.  Set this number at least as large as
#   128 * (number of virtual disks on the host)
# Libvirt uses a default of 1M requests to allow 8k disks, with at most
# 64M of kernel memory if all disks hit an aio request at the same time.
fs.aio-max-nr = 1048576
Comment 21 errata-xmlrpc 2022-12-07 20:29:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack 16.1.9 (openstack-tripleo-heat-templates) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8796