Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2013341

Summary: [13->16.1] Conflicts between 99-sysctl.conf and 99-tripleo.conf cause rabbitmq service start failure
Product: Red Hat OpenStack Reporter: Jesse Pretorius <jpretori>
Component: openstack-tripleo-heat-templatesAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED DEFERRED QA Contact: Joe H. Rahme <jhakimra>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: coldford, jbadiapa, jeckersb, ltamagno, mburns
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-12-06 14:47:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jesse Pretorius 2021-10-12 16:17:05 UTC
Description of problem:

During a 13->16.1 upgrade, there is disable_ipv6 in /etc/sysctl.d/99-sysctl.conf and /etc/systctl.d/99-tripleo.conf where one have the options:

/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.all.disable_ipv6=1
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.default.disable_ipv6=1
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.lo.disable_ipv6=0
/etc/sysctl.d/99-sysctl.conf:net.ipv6.conf.all.disable_ipv6=0
/etc/sysctl.d/99-sysctl.conf:net.ipv6.conf.default.disable_ipv6=0
/etc/sysctl.d/99-sysctl.conf:net.ipv6.conf.lo.disable_ipv6=0

For some reason this makes systemd interpret net.ipv6.conf.lo.disable_ipv6 as 1 and it disables ipv6 there. This causes epmd to fail to start.

Version-Release number of selected component (if applicable):

OSP 16.1.4

Comment 3 John Eckersberg 2022-10-13 13:41:12 UTC
I fixed this upstream in erlang (epmd) here - https://github.com/erlang/otp/pull/5762

And we pulled it into OSP17 here - https://bugzilla.redhat.com/show_bug.cgi?id=2050624

It's not currently available in OSP16 but it's something that we could backport or rebase to pick it up.

Comment 5 coldford@redhat.com 2023-06-20 12:47:36 UTC
As we sure this isn't tuned processing /etc/sysctl.d/99-sysctl.conf twice?

def _apply_system_sysctl():
        files = {}
        for d in SYSCTL_CONFIG_DIRS:                                              
                try:
                        flist = os.listdir(d)
                except OSError:
                        continue
                for fname in flist:
                        if not fname.endswith(".conf"):
                                continue
                        if fname not in files:
                                files[fname] = d

        for fname in sorted(files.keys()):            
                d = files[fname]
                path = "%s/%s" % (d, fname)
                _apply_sysctl_config_file(path)                                  <----- The enumerated files in SYSCTL_CONFIG_DIRS
        _apply_sysctl_config_file("/etc/sysctl.conf")                            <----- And last /etc/sysctl.conf which was already applied via symlink /etc/sysctl.d/99-sysctl.conf

See: https://bugzilla.redhat.com/show_bug.cgi?id=2213614

Comment 6 coldford@redhat.com 2023-06-20 13:07:32 UTC
Note: 
It also is still present in the upstream master:
- See: https://github.com/redhat-performance/tuned/blob/master/tuned/plugins/plugin_sysctl.py#L123

The highlighted line should be removed.

Comment 7 coldford@redhat.com 2023-06-21 18:46:00 UTC
I should also note some other details from https://bugzilla.redhat.com/show_bug.cgi?id=2213614

// systemd handing:
A review of the systemd-sysctl.service handling and changelogs reveals what should likely be the correct approach:

From: systemd-239/NEWS

        * The systemd-sysctl tool no longer natively reads the file
          /etc/sysctl.conf. If desired, the file should be symlinked
          from /etc/sysctl.d/99-sysctl.conf. Apart from providing
          legacy support by a symlink rather than built-in code, it
          also makes the otherwise hidden order of application of the
          different files visible. (Note that this partly reverts to a
          pre-198 application order of sysctl knobs!)


// Tuned handling
def _apply_system_sysctl():
        files = {}
        for d in SYSCTL_CONFIG_DIRS:                                              
                try:
                        flist = os.listdir(d)
                except OSError:
                        continue
                for fname in flist:
                        if not fname.endswith(".conf"):
                                continue
                        if fname not in files:
                                files[fname] = d

        for fname in sorted(files.keys()):            
                d = files[fname]
                path = "%s/%s" % (d, fname)
                _apply_sysctl_config_file(path)                                  <----- The enumerated files in SYSCTL_CONFIG_DIRS
        _apply_sysctl_config_file("/etc/sysctl.conf")                            <----- And last /etc/sysctl.conf which was already applied via symlink /etc/sysctl.d/99-sysctl.conf

// procps-ng sysctl binary handling:

$ sudo sysctl --system
Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
Applying /usr/lib/sysctl.d/50-coredump.conf ...
Applying /usr/lib/sysctl.d/50-default.conf ...
Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ...
Applying /usr/lib/sysctl.d/50-pid-max.conf ...
Applying /usr/lib/sysctl.d/60-libvirtd.conf ...
Applying /usr/lib/sysctl.d/60-qemu-postcopy-migration.conf ...
Applying /etc/sysctl.d/99-sysctl.conf ...                                <--- SETTINGS FIRST APPLIED
Applying /etc/sysctl.d/99-test.conf ...                                  <--- CUSTOM SETTINGS REQUIRED
Applying /etc/sysctl.conf ...                                            <--- SETTINGS APPLIED AGAIN

So this leads me to some questions/thoughts:
- What was the intent behind tripleo choosing /etc/sysctl.d/99-tripleo.conf ?
- Why the differing behaviours between systemd-udev and ( sysctl --system, and tuned )?
- What is the actual expected behaviour?

Comment 8 Lukas Bezdicka 2024-12-06 14:47:10 UTC
Manually tested now on OSP16.2:
[root@undercloud-0 ~]# sysctl -a | grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.br-ctlplane.disable_ipv6 = 1
net.ipv6.conf.br-int.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
net.ipv6.conf.eth1.disable_ipv6 = 1
net.ipv6.conf.eth2.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.ovs-system.disable_ipv6 = 1



/etc/sysctl.d/99-sysctl.conf:net.ipv6.conf.all.disable_ipv6=0
/etc/sysctl.d/99-sysctl.conf:net.ipv6.conf.default.disable_ipv6=0
/etc/sysctl.d/99-sysctl.conf:net.ipv6.conf.lo.disable_ipv6=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.all.accept_ra=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.default.accept_ra=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.all.autoconf=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.default.autoconf=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.default.accept_redirects=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.all.accept_redirects=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.all.ndisc_notify=1
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.all.disable_ipv6=1
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.default.disable_ipv6=1
/etc/sysctl.d/99-tripleo.conf:net.ipv6.conf.lo.disable_ipv6=0
/etc/sysctl.d/99-tripleo.conf:net.ipv6.ip_nonlocal_bind=1