Bug 1964394

*** Bug 1987288 has been marked as a duplicate of this bug. ***

(In reply to Devendra Singh from comment #0)
> 1. Install Satellite with 6.10 snap1. 

To properly analyze this, the debug installer logs are needed. I must know what the property NeedDaemonReload is right before the service restart. It should execute this command to retrieve it.

systemctl show --property=NeedDaemonReload -- postgresql.service

If you get the warning, it should clearly return yes but the fact it doesn't reload suggests it doesn't.

It may also be helpful to see what systemctl cat postgresql.service says.

Never mind the NEEDINFO, I found a minimal reproducer (on CentOS 7):

yum install -y -q centos-release-scl-rh
yum install -y -q rh-postgresql12-postgresql-server-syspaths
postgresql-setup --initdb
systemctl start postgresql.service
systemctl show --property=NeedDaemonReload -- postgresql.service
mkdir /etc/systemd/system/postgresql.service.d
cat <<EOF > /etc/systemd/system/postgresql.service.d/limits.conf
[Service]
LimitNOFILE=10000
EOF
systemctl show --property=NeedDaemonReload -- postgresql.service
systemctl daemon-reload
systemctl show --property=NeedDaemonReload -- postgresql.service

When you run it:

# yum install -y -q centos-release-scl-rh
warning: /var/cache/yum/x86_64/7/extras/packages/centos-release-scl-rh-2-3.el7.centos.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for centos-release-scl-rh-2-3.el7.centos.noarch.rpm is not installed
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-8.2003.0.el7.centos.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
# yum install -y -q rh-postgresql12-postgresql-server-syspaths
warning: /var/cache/yum/x86_64/7/centos-sclo-rh/packages/rh-postgresql12-postgresql-12.7-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID f2ee9d55: NOKEY
Public key for rh-postgresql12-postgresql-12.7-1.el7.x86_64.rpm is not installed
Importing GPG key 0xF2EE9D55:
 Userid     : "CentOS SoftwareCollections SIG (https://wiki.centos.org/SpecialInterestGroup/SCLo) <security>"
 Fingerprint: c4db d535 b1fb ba14 f8ba 64a8 4eb8 4e71 f2ee 9d55
 Package    : centos-release-scl-rh-2-3.el7.centos.noarch (@extras)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-SCLo
# postgresql-setup --initdb
 * Initializing database in '/var/opt/rh/rh-postgresql12/lib/pgsql/data'
 * Initialized, logs are in /var/lib/pgsql/initdb_rh-postgresql12-postgresql.log
# systemctl start postgresql.service
# systemctl show --property=NeedDaemonReload -- postgresql.service
NeedDaemonReload=no
# mkdir /etc/systemd/system/postgresql.service.d
# cat <<EOF > /etc/systemd/system/postgresql.service.d/limits.conf
> [Service]
> LimitNOFILE=10000
> EOF
# systemctl show --property=NeedDaemonReload -- postgresql.service
NeedDaemonReload=yes
# systemctl daemon-reload
# systemctl show --property=NeedDaemonReload -- postgresql.service
NeedDaemonReload=yes

The problem is that we rely on a service symlink:

# ls -l /usr/lib/systemd/system/postgresql.service 
lrwxrwxrwx. 1 root root 58 Aug  6 17:11 /usr/lib/systemd/system/postgresql.service -> /usr/lib/systemd/system/rh-postgresql12-postgresql.service

The real service name is rh-postgresql12-postgresql. This means systemd doesn't look at /etc/systemd/system/postgresql.service.d. If we do:

# mv /etc/systemd/system/postgresql.service.d /etc/systemd/system/rh-postgresql12-postgresql.service.d

Then we see it does work:

# systemctl show --property=NeedDaemonReload -- postgresql.service
NeedDaemonReload=yes
# systemctl daemon-reload
# systemctl show --property=NeedDaemonReload -- postgresql.service
NeedDaemonReload=no


Running systemctl cat postgresql.service also informs us it loads:

# /usr/lib/systemd/system/rh-postgresql12-postgresql.service
<snip>
# /etc/systemd/system/rh-postgresql12-postgresql.service.d/limits.conf
<snip>

And a really ugly workaround that appears to work:

ln -s postgresql.service.d /etc/systemd/system/rh-postgresql12-postgresql.service.d

*** Bug 1995895 has been marked as a duplicate of this bug. ***

I've proposed a patch upstream to fix the PostgreSQL part of the warnings:
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.


However this will not affect the socket ones:
Warning: Stopping foreman.service, but it can still be activated by:
  foreman.socket

Warning: Stopping pulpcore-api.service, but it can still be activated by:
  pulpcore-api.socket

Warning: Stopping pulpcore-content.service, but it can still be activated by:
  pulpcore-content.socket

Those can be fixed if we stop (and start) the sockets in foreman-maintain too. [The stop needs to happen *before* the service stop].

However, I'd argue that this BZ should be split up in two, one fore PostgreSQL, one for sockets.

Upstream bug assigned to ekohlvan

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/32323 has been resolved.

Tested on Satellite 6.11 and snap 18.5, Not able to see the warning messages for the PostgreSQL, Marking as verified and creating BZ for sockets https://bugzilla.redhat.com/show_bug.cgi?id=2081714


# foreman-maintain service restart

Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services:

Stopping the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- stopping httpd
Warning: Stopping foreman.service, but it can still be activated by:
  foreman.socket
- stopping pulpcore-content
Warning: Stopping pulpcore-api.service, but it can still be activated by:
  pulpcore-api.socket

Warning: Stopping pulpcore-content.service, but it can still be activated by:
  pulpcore-content.socket
\ All services stopped

Starting the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
\ All services started                                                [OK]
--------------------------------------------------------------------------------

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498