Bug 2036195 - Provide mechanism to tune the sysctl param fs.aio-max-nr on the host
Summary: Provide mechanism to tune the sysctl param fs.aio-max-nr on the host
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.1 (Train)
Hardware: All
OS: All
medium
medium
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: OSP Team
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-30 11:13 UTC by Alex Stupnikov
Modified: 2022-12-07 20:29 UTC (History)
4 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20220601143346.29a02c1.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2061697 (view as bug list)
Environment:
Last Closed: 2022-12-07 20:29:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 825121 0 None MERGED Defining fs.aio-max-nr for hosts with libvirt service 2022-01-31 16:20:46 UTC
OpenStack gerrit 827018 0 None MERGED Defining fs.aio-max-nr for hosts with libvirt service 2022-10-25 09:39:33 UTC
Red Hat Issue Tracker OSP-11972 0 None None None 2021-12-30 11:17:13 UTC
Red Hat Knowledge Base (Solution) 6676871 0 None None None 2022-01-28 08:55:26 UTC
Red Hat Product Errata RHSA-2022:8796 0 None None None 2022-12-07 20:29:37 UTC

Internal Links: 2122119

Description Alex Stupnikov 2021-12-30 11:13:08 UTC
Description of problem:


Previously (for example, in RHOSP 13) libvirt-daemon RPM was installed on compute nodes and it provided /usr/lib/sysctl.d/60-libvirtd.conf configuration file [1] to increase fs.aio-max-nr from 65536 (default) to 1048576. libvirt-daemon RPM is no longer installed on RHOSP 16.1 nodes and default value is not changed there.

Documentation tells us that this number should be calculated as number of virtual disks on the host * 128. As a result default limit leaves us with 512 virtual disks, but KCS providing deployment limitations https://access.redhat.com/articles/1436373 tells us about 16384 KVM guests per hypervisor.

I am not sure how default value would behave in containerized environment, but this looks like a possible problem that requires second look.

[1]
[root@compute-1 ~]# cat /usr/lib/sysctl.d/60-libvirtd.conf 
# The kernel allocates aio memory on demand, and this number limits the
# number of parallel aio requests; the only drawback of a larger limit is
# that a malicious guest could issue parallel requests to cause the kernel
# to set aside memory.  Set this number at least as large as
#   128 * (number of virtual disks on the host)
# Libvirt uses a default of 1M requests to allow 8k disks, with at most
# 64M of kernel memory if all disks hit an aio request at the same time.
fs.aio-max-nr = 1048576

Comment 21 errata-xmlrpc 2022-12-07 20:29:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack 16.1.9 (openstack-tripleo-heat-templates) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8796


Note You need to log in before you can comment on or make changes to this bug.