Bug 1867311

Summary: Upgrade fails when checkpoint_segments postgres parameter configured
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: InstallationAssignee: Ewoud Kohl van Wijngaarden <ekohlvan>
Status: CLOSED ERRATA QA Contact: Devendra Singh <desingh>
Severity: high Docs Contact:
Priority: high    
Version: 6.8.0CC: chrobert, ehelms, ekohlvan, gpadholi, janarula, mawerner, pcreech, smajumda
Target Milestone: 6.8.0Keywords: Triaged, Upgrades
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: foreman-installer-2.1.2.6-1 Doc Type: Known Issue
Doc Text:
Upgrade fails when checkpoint_segments postgres parameter configured
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 13:05:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2020-08-08 14:41:19 UTC
Description of problem:
Upgrading from 6.7 with /etc/foreman-installer/custom-hiera.yaml :

postgresql::server::config_entries:
  checkpoint_segments: 32

to 6.8 fails on starting postgres service:


Aug  8 16:18:34 pmoravec-sat-test sh: 2020-08-08 14:18:34.450 GMT [17329] LOG:  unrecognized configuration parameter "checkpoint_segments" in file "/var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf" line 760
Aug  8 16:18:34 pmoravec-sat-test sh: 2020-08-08 14:18:34.450 GMT [17329] FATAL:  configuration file "/var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf" contains errors
Aug  8 16:18:34 pmoravec-sat-test systemd: postgresql.service: main process exited, code=exited, status=1/FAILURE


Further attempt to remove the tuning from custom hiera fails: commenting out the parameter from the yaml file and running satellite-installer does _not_ update the /var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf file, hence postgresql fails to start again.


Version-Release number of selected component (if applicable):
satellite-6.8.0-0.7.beta.el7sat.noarch
satellite-installer-6.8.0.8-1.el7sat.noarch


How reproducible:
100% I guess?


Steps to Reproduce:
1. Have Sat 6.7 with /etc/foreman-installer/custom-hiera.yaml :
postgresql::server::config_entries:
  checkpoint_segments: 32

2. Run upgrade to 6.8


Actual results:
2. is supposed to fail with above error


Expected results:
2. upgrade to proceed well


Additional info:

Comment 2 Ewoud Kohl van Wijngaarden 2020-08-10 10:10:35 UTC
This is indeed expected. PostgreSQL has dropped the checkpoint_segments option. See https://www.postgresql.org/docs/9.5/release-9-5.html which states:

> Replace configuration parameter checkpoint_segments with min_wal_size and max_wal_size (Heikki Linnakangas)

The option is not removed due to a technical limitation. The code can only set the value:

https://github.com/puppetlabs/puppetlabs-postgresql/blob/ef64c30b96ee22804de6e0bcf3e83b988c23b22a/manifests/server.pp#L185-L189

And this particular implementation only manages individual ini settings which means unmanaged entries are ignored.

I have submitted https://github.com/puppetlabs/puppetlabs-postgresql/pull/1187 and with that, you should be able to set checkpoint_segments to ~ in custom-hiera.yaml to remove an entry.

By its very nature, we don't touch custom-hiera.yaml at all - this is really supposed to be a user way of setting things. It'd be my recommendation that we add a release note that users who previously followed the tuning guide (which recommended this) update to the new tuning parameter.

Comment 3 Eric Helms 2020-08-10 14:46:26 UTC
Adding that I think this is should be a docs bug with release notes to inform users who had this configured that they need to:

 1) Remove it from custom-hiera.yaml
 2) Edit /var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf to remove the checkpoint_segments entry manually

Comment 4 Ewoud Kohl van Wijngaarden 2020-08-10 14:53:27 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1851773 is a larger overarching issue. I think it makes sense to keep this open for now since this can actually prevent upgrades while the other might result in a working but unexpected situation.

Comment 5 Chris Roberts 2020-08-10 16:41:47 UTC
Added a bz to create a foreman-maintain check for this.

Comment 6 Bryan Kearney 2020-08-20 20:01:37 UTC
Upstream bug assigned to ekohlvan

Comment 7 Bryan Kearney 2020-08-20 20:01:39 UTC
Upstream bug assigned to ekohlvan

Comment 8 Bryan Kearney 2020-08-24 16:01:40 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/28928 has been resolved.

Comment 12 Devendra Singh 2020-09-28 17:19:12 UTC
Verified on 6.8.0 Snap17

Verification points:

1- foreman-maintain upgrade "Check and Run" both fail with a proper error message if checkpoint_segments configured.

# foreman-maintain upgrade check --target-version 6.8 -y
Checking for new version of satellite-maintain...
Nothing to update, can't find new version of satellite-maintain.
Running preparation steps required to run the next scenarios
================================================================================
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: 
| Checking repositories enabled on the system                         [OK]      
--------------------------------------------------------------------------------


Running Checks before upgrading to Satellite 6.8
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check for verifying syntax for ISP DHCP configurations:               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check to make sure root(/) partition has enough space:                [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:                                 [OK]
--------------------------------------------------------------------------------
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for old tasks in paused/stopped state:                          [OK]
--------------------------------------------------------------------------------
Check for pending tasks which are safe to delete:                     [OK]
--------------------------------------------------------------------------------
Check for tasks in planning state:                                    [OK]
--------------------------------------------------------------------------------
Check to verify if any hotfix installed on system: 
- Checking for presence of hotfix(es). It may take some time to verify.         
                                                                      [OK]
--------------------------------------------------------------------------------
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: 
| Checking repositories enabled on the system                         [OK]      
--------------------------------------------------------------------------------
Check if TMOUT environment variable is set:                           [OK]
--------------------------------------------------------------------------------
Check if any upstream repositories are enabled on system: 
| Checking for presence of upstream repositories                      [OK]      
--------------------------------------------------------------------------------
Check if yum exclude list is configured:                              [OK]
--------------------------------------------------------------------------------
Check for roles that have filters with multiple resources attached:   [OK]
--------------------------------------------------------------------------------
Check for duplicate permissions from database:                        [OK]
--------------------------------------------------------------------------------
Check whether reports have correct associations:                      [OK]
--------------------------------------------------------------------------------
Check if checkpoint_segments configuration exists on the system:      [FAIL]
ERROR: Tuning option 'checkpoint_segments' found.
This option is no longer valid for PostgreSQL 9.5 or newer.
Please remove it from following file and re-run the command.
- /etc/foreman-installer/custom-hiera.yaml
The presence of checkpoint_segments in /etc/foreman-installer/custom-hiera.yaml indicates manual tuning.
Manual tuning can override values provided by the --tuning parameter.
Review /etc/foreman-installer/custom-hiera.yaml for values that are already provided by the built in tuning profiles.
Built in tuning profiles also provide a supported upgrade path.

--------------------------------------------------------------------------------
Validate availability of repositories: 
| Validating availability of repositories for 6.8                               
\ Validating availability of repositories for 6.8                     [OK]      
--------------------------------------------------------------------------------
Scenario [Checks before upgrading to Satellite 6.8] failed.

The following steps ended up in failing state:

  [check-postgresql-checkpoint-segments]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="check-postgresql-checkpoint-segments"

2. foreman-installer version

# rpm -qa|grep  foreman-installer
foreman-installer-2.1.2.8-1.el7sat.noarch

Comment 15 errata-xmlrpc 2020-10-27 13:05:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366

Comment 16 errata-xmlrpc 2020-10-27 13:08:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366