Bug 2049796 - [QE][doc] THT parameters to allow for multiple hugepage sizes per compute
Summary: [QE][doc] THT parameters to allow for multiple hugepage sizes per compute
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: RHOS Documentation Team
QA Contact: RHOS Documentation Team
URL:
Whiteboard:
Depends On: 2076498
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-02 17:35 UTC by David Vallee Delisle
Modified: 2022-04-22 15:37 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2043588
Environment:
Last Closed: 2022-04-22 15:37:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-12444 0 None Waiting on Red Hat Provide e.g. /etc/haproxy/conf.d directory and use it in haproxy.service 2022-06-27 11:01:39 UTC

Description David Vallee Delisle 2022-02-02 17:35:47 UTC
+++ This bug was initially created as a clone of Bug #2043588 +++
The parent bug is used to discuss the implementation of a feature that would automate the best we can the handling of hugepages in OSP18

This bug will be used to document a procedure to mount the hugepages folders on the compute nodes during firstboot.

Nova actually supports the handling of multiple pagesizes. 

This is how it can be done [1] using firstboot.

It's important to note that pre-deployed nodes are probably not running the firstboot templates so it might fall on the operator's shoulder to run something similar during their node provisioning process.

It's also important to note that these hugepage mounts need to be passed to the nova_compute and nova_libvirt containers. In this case, we create them under /dev which is automatically bindmounted to these containers, but if operators are choosing another destination, they need to make sure that it's passed to the containers as well.

Here's a quick summary of what happens with this template:
- We mask the default dev-hugepages.mount systemd unit file as we want to create new mounts using the pagesize.
- We make sure that the folders are created beforehand
- We create systemd mount units for each pagesizes
- After the first loop, we need to systemd daemon-reload to include these newly created units
- Finally, we start each one of these mounts

[1]
~~~
(undercloud) [stack@undercloud-0 ~]$ cat 16.2_deployment_files/hugepages.yaml
heat_template_version: queens

description: >
  Hugepages configuration

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: hugepages_config}

  hugepages_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        systemctl mask dev-hugepages.mount || true
        for pagesize in 2M 1G;do
          if ! [ -d "/dev/hugepages${pagesize}" ]; then
            mkdir -p "/dev/hugepages${pagesize}"
            cat << EOF > /etc/systemd/system/dev-hugepages${pagesize}.mount
        [Unit]
        Description=${pagesize} Huge Pages File System
        Documentation=https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
        Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
        DefaultDependencies=no
        Before=sysinit.target
        ConditionPathExists=/sys/kernel/mm/hugepages
        ConditionCapability=CAP_SYS_ADMIN
        ConditionVirtualization=!private-users
        
        [Mount]
        What=hugetlbfs
        Where=/dev/hugepages${pagesize}
        Type=hugetlbfs
        Options=pagesize=${pagesize}
        EOF
          fi
        done
        systemctl daemon-reload
        for pagesize in 2M 1G;do
          systemctl start dev-hugepages${pagesize}.mount
        done

outputs:
  OS::stack_id:
    value: {get_resource: userdata}

(undercloud) [stack@undercloud-0 ~]$ cat 16.2_deployment_files/firstboot.yaml
resource_registry:
  OS::TripleO::ComputeHugepage::NodeUserData: ./hugepages.yaml
~~~

Comment 8 Bogdan Dobrelya 2022-04-19 14:22:44 UTC
I'm not sure if the described procedure in the issue description is self-sufficient, or kernelargs should be tweaked also? If so, we either need to clarify that in docs, or consider the RFE backport process (bz 2076498)

Comment 9 Bogdan Dobrelya 2022-04-20 11:47:11 UTC
I propose to re-test with the following adjustment,
assuming the computes hostname detection pattern should be adjusted by a user to match the reality:

  hugepages_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        hostname | grep -qiE 'co?mp' || exit 0
        systemctl mask dev-hugepages.mount || true
        for pagesize in 2M 1G;do
          if ! [ -d "/dev/hugepages${pagesize}" ]; then
            mkdir -p "/dev/hugepages${pagesize}"
            cat << EOF > /etc/systemd/system/dev-hugepages${pagesize}.mount
        [Unit]
        Description=${pagesize} Huge Pages File System
        Documentation=https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
        Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
        DefaultDependencies=no
        Before=sysinit.target
        ConditionPathExists=/sys/kernel/mm/hugepages
        ConditionCapability=CAP_SYS_ADMIN
        ConditionVirtualization=!private-users
        
        [Mount]
        What=hugetlbfs
        Where=/dev/hugepages${pagesize}
        Type=hugetlbfs
        Options=pagesize=${pagesize}

        [Install]
        WantedBy = sysinit.target
        EOF
          fi
        done
        systemctl daemon-reload
        for pagesize in 2M 1G;do
          systemctl start dev-hugepages${pagesize}.mount
        done

Comment 10 Bogdan Dobrelya 2022-04-20 11:49:08 UTC
Sorry, I forgot to amend the last lines:

        for pagesize in 2M 1G;do
          systemctl enable --now dev-hugepages${pagesize}.mount
        done


Note You need to log in before you can comment on or make changes to this bug.