Bug 2070991

Summary: Warning: postgresql.service changed on disk, when calling foreman-maintain service restart
Product: Red Hat Satellite Reporter: Brad Buckingham <bbuckingham>
Component: InstallationAssignee: Ewoud Kohl van Wijngaarden <ekohlvan>
Status: CLOSED ERRATA QA Contact: Vladimír Sedmík <vsedmik>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.10.0CC: ahumbe, alsouza, egolov, ehelms, ekohlvan, gpayelka, jerry_d_williams, jkrajice, ktordeur, mkalyat, mmccune, pcfe, saydas, swadeley, vogt, vsedmik, zhunting
Target Milestone: 6.10.5Keywords: Regression, Triaged, UpgradeBlocker
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: foreman-installer-2.5.2.11-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1964394 Environment:
Last Closed: 2022-05-04 12:59:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 6 Vladimír Sedmík 2022-04-21 09:46:50 UTC 
Upgrade output:
...
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 6:
 2022-04-21 04:06:00 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-04-21 04:06:07 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-04-21 04:06:07 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-04-21 04:06:15 [WARN  ] [pre] Skipping system checks.
2022-04-21 04:06:15 [WARN  ] [pre] Skipping system checks.
2022-04-21 04:07:21 [NOTICE] [configure] Starting system configuration.
2022-04-21 04:07:36 [NOTICE] [configure] 250 configuration steps out of 2176 steps complete.
2022-04-21 04:07:57 [NOTICE] [configure] 500 configuration steps out of 2176 steps complete.
2022-04-21 04:08:20 [NOTICE] [configure] 750 configuration steps out of 2178 steps complete.
2022-04-21 04:08:42 [NOTICE] [configure] 1000 configuration steps out of 2182 steps complete.
2022-04-21 04:08:44 [NOTICE] [configure] 1250 configuration steps out of 2207 steps complete.
2022-04-21 04:08:44 [ERROR ] [configure] Systemd start for rh-postgresql12-postgresql failed!
2022-04-21 04:08:44 [ERROR ] [configure] journalctl log for rh-postgresql12-postgresql:
2022-04-21 04:08:44 [ERROR ] [configure] -- Logs begin at Thu 2022-04-21 02:49:27 EDT, end at Thu 2022-04-21 04:08:44 EDT. --
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: Starting PostgreSQL database server...
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: rh-postgresql12-postgresql.service: main process exited, code=exited, status=1/FAILURE
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: Failed to start PostgreSQL database server.
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: Unit rh-postgresql12-postgresql.service entered failed state.
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: rh-postgresql12-postgresql.service failed.
2022-04-21 04:08:44 [ERROR ] [configure] /Stage[main]/Postgresql::Server::Service/Service[postgresqld]/ensure: change from 'stopped' to 'running' failed: Systemd start for rh-postgresql12-postgresql failed!
2022-04-21 04:08:44 [ERROR ] [configure] journalctl log for rh-postgresql12-postgresql:
2022-04-21 04:08:44 [ERROR ] [configure] -- Logs begin at Thu 2022-04-21 02:49:27 EDT, end at Thu 2022-04-21 04:08:44 EDT. --
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: Starting PostgreSQL database server...
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: rh-postgresql12-postgresql.service: main process exited, code=exited, status=1/FAILURE
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: Failed to start PostgreSQL database server.
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: Unit rh-postgresql12-postgresql.service entered failed state.
2022-04-21 04:08:44 [ERROR ] [configure] Apr 21 04:08:44 satellite.redhat.com systemd[1]: rh-postgresql12-postgresql.service failed.
2022-04-21 04:08:49 [NOTICE] [configure] 1500 configuration steps out of 2213 steps complete.
2022-04-21 04:09:16 [NOTICE] [configure] 1750 configuration steps out of 2213 steps complete.
2022-04-21 04:09:20 [NOTICE] [configure] 2000 configuration steps out of 3017 steps complete.
2022-04-21 04:09:21 [NOTICE] [configure] 2250 configuration steps out of 3017 steps complete.
2022-04-21 04:09:22 [NOTICE] [configure] 2500 configuration steps out of 3017 steps complete.
2022-04-21 04:09:43 [NOTICE] [configure] 2750 configuration steps out of 3017 steps complete.
2022-04-21 04:09:44 [NOTICE] [configure] 3000 configuration steps out of 3017 steps complete.
2022-04-21 04:09:52 [NOTICE] [configure] System configuration has finished.

From journal:
Apr 21 04:38:06 satellite systemd: Starting PostgreSQL database server...
Apr 21 04:38:06 satellite sh: 2022-04-21 04:38:06 EDT FATAL:  lock file "postmaster.pid" already exists
Apr 21 04:38:06 satellite sh: 2022-04-21 04:38:06 EDT HINT:  Is another postmaster (PID 7714) running in data directory "/var/opt/rh/rh-postgresql12/lib/pgsql/data"?
Apr 21 04:38:06 satellite systemd: rh-postgresql12-postgresql.service: main process exited, code=exited, status=1/FAILURE
Apr 21 04:38:06 satellite systemd: Failed to start PostgreSQL database server.
Apr 21 04:38:06 satellite systemd: Unit rh-postgresql12-postgresql.service entered failed state.
Apr 21 04:38:06 satellite systemd: rh-postgresql12-postgresql.service failed.
Comment 7 Vladimír Sedmík 2022-04-25 15:38:27 UTC 
Verified in 6.10.5 snap 2:

1) The postgresql related warning disappeared (the socket related warnings persist, tracked by BZ#2067120)

Warning: Stopping foreman.service, but it can still be activated by:
  foreman.socket
\ stopping pulpcore-content                                                     
Warning: Stopping pulpcore-api.service, but it can still be activated by:
  pulpcore-api.socket
Warning: Stopping pulpcore-content.service, but it can still be activated by:
  pulpcore-content.socket
| All services stopped 

2) 6.9.9 snap 2 → 6.10.5 snap 2 upgrade succeeded

3) 6.10.5 snap 2 → 6.11.0 snap 17 upgrade succeeded
Comment 12 errata-xmlrpc 2022-05-04 12:59:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.10.5 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1708