Description of problem: The OpenShift nodes by default are having fs.aio-max-nr = 65536. However, this is low and a host with many VMs running can hit this limit and the VM startup fails with error "Resource temporarily unavailable" since io_setup fails with EAGAIN. libvirtd raises this limit to 1048576 with a custom sysctl.conf [1] in other virtualization platforms like RHV and Stack which is hard to hit. Since libvirtd is containerized in OpenShift Virt, this conf is not applied. In my tests, for every VM that has disks with aio=native, the fs.aio-nr is getting incremented by 1024 (Looks like 128 was increased to 1024 [2]) and I can see io_setup called with nr_events=1024 in the strace of the qemu-kvm. So it's easy to hit the limit on a host with many VMs with the default 65536. [1] https://github.com/libvirt/libvirt/commit/5298551e07a9839c046e0987b325e03f8ba801e5 [2] https://github.com/qemu/qemu/commit/2558cb8dd4150512bc8ae6d505cdcd10d0cc46bb Version-Release number of selected component (if applicable): OpenShift Virtualization 4.10.4 How reproducible: 100% Steps to Reproduce: 1. Start a VM with disk aio=native. Use preallocated file or block disks which use aio=native by default. 2. fs.aio-nr is incremented by 1024 for every new VMs. 3. With many VMs running in the host, it will hit the aio-max-nr Actual results: Virtual machine fails to start with error "Unable to use native AIO: failed to create linux AIO context: Resource temporarily unavailable" Expected results: Lift the default limit fs.aio-max-nr so that host can have more VMs. Additional info:
Hi, Unlike RHV, in OCP case the CNV doesn't manage the nodes directly, CNV is an operator installed as an add-on on the cluster. Because of that, the tuning of the RHCOS nodes needs to be done on the OCP level using MachineConfig or the Node Tune Operator. This cannot be done automatically as part of CNV deployment, since then it will cause issue with HyperShift support. Perhaps what we can do is produce some kind of event that the CNV worker nodes are not tuned.
Also, aio-nr per VM also depends on VMs IO threads policy. It is 1024 * disks if the disks are having dedicated IO threads. The High-Performance VM templates have got "ioThreadsPolicy: shared" and will enable one io threads and for these VMs it will be 2048 per VM.
We've found a way to make this tune be part of default OCP installation, details are in this ticket: https://issues.redhat.com/browse/PSAP-900
Tested the verified the following scenario on lab with OCP 4.12.5 and CNV is 4.12.0. Checked aio on the node before starting VMS : sudo sysctl -a | grep aio fs.aio-max-nr = 1048576 fs.aio-nr = 2048 1. Start a VM with disk aio=native. Use'd block local storage ODF. 2. fs.aio-nr is incremented by 1024 for every new VMs. 3. Started 81 VMS on a single node Checked aio on the node with the VMS : sudo sysctl -a | grep aio fs.aio-max-nr = 1048576 fs.aio-nr = 84992
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.13.0 Images security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:3205