Bug 1935301
| Summary: | postgresql:12/postgresql: Presence of data_directory in /var/lib/pgsql/data/postgresql.conf breaks postgresql-upgrade | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Ewoud Kohl van Wijngaarden <ekohlvan> | ||||
| Component: | postgresql | Assignee: | mkulik | ||||
| Status: | CLOSED ERRATA | QA Contact: | Vaclav Danek <vdanek> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 8.6 | CC: | bstinson, carl, databases-maint, fjanus, hhorak, jwboyer, mkulik, pkubat, praiskup | ||||
| Target Milestone: | rc | Keywords: | EasyFix, TestCaseProvided, Triaged | ||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 2024567 2024568 (view as bug list) | Environment: | |||||
| Last Closed: | 2022-05-10 13:47:44 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 2024567, 2024568 | ||||||
| Attachments: |
|
||||||
I forgot to mention this, but I initially found this on regular CentOS Linux. I didn't have a RHEL installation at hand, but I'd guess it also affects RHEL. Thanks for reporting. Confirmed on RHEL 8.5 release as well: [root@ci-vm-10-0-137-204 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.5 Beta (Ootpa) [root@ci-vm-10-0-137-204 ~]# postgresql-setup --upgrade * Upgrading database. ERROR: pg_upgrade tool failed ERROR: Upgrade failed. * See /var/lib/pgsql/upgrade_postgresql.log for details. [root@ci-vm-10-0-137-204 ~]# cat /var/lib/pgsql/upgrade_postgresql.log Performing Consistency Checks ----------------------------- Checking cluster versions ok *failure* Consult the last few lines of "pg_upgrade_server.log" for the probable cause of the failure. connection to database failed: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/lib/pgsql/.s.PGSQL.5432"? could not connect to source postmaster started with the command: "/usr/lib64/pgsql/postgresql-10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data-old" -o "-p 5432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql'" start Failure, exiting One more log that shows what is hapenning: [root@ci-vm-10-0-137-204 ~]# cat /var/lib/pgsql/pg_upgrade_server.log ----------------------------------------------------------------- pg_upgrade run on Wed Apr 14 11:30:38 2021 ----------------------------------------------------------------- command: "/usr/lib64/pgsql/postgresql-10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data-old" -o "-p 5432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql'" start >> "pg_upgrade_server.log" 2>&1 waiting for server to start....2021-04-14 11:30:38.080 EDT [8083] FATAL: database files are incompatible with server 2021-04-14 11:30:38.080 EDT [8083] DETAIL: The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 10.15. stopped waiting pg_ctl: could not start server Examine the log output. It all looks like the combination of -D on the command-line and data_directory option in the config file is not doing what the upgrade process expects. It's not clear whether the option parsing is not clever enough here (didn't find a clear definition of what is the expected behavior in this case), or whether the upgrade should make sure the option is not present. This seems to be worth discussing with upstream. (In reply to Honza Horak from comment #4) > This seems to be worth discussing with upstream. Doing that: https://www.postgresql.org/message-id/0b24aa0a-c0c6-e870-af11-e18400ada98b%40redhat.com The upstream's response actually suggests to not use (or filter out) the data_directory option when we manipulate with the data directory: https://www.postgresql.org/message-id/3566642.1618422939%40sss.pgh.pa.us Or we can alternatively stop the upgrade early in case data_directory is defined in the postgresql.conf. The upgrade process should be likely interrupted in case the data_directory has a different value than the default. This may be also a great content for Insights or/and LEAPP. I blocked execution for:
- postgresql-setup --upgrade
- postgresql-setup --initdb
when `data_directory` entry in config file is detected. According to postgresql docs:
> Notice that data_directory overrides -D and PGDATA for the location of the data directory, but not for the location of the configuration files.
That means, when user define pgdata folder in config file it will take a priority over ENV variable in .service file and -D parameter in command line.
Currently postgresql-setup does not take into consideration `data_directory` entry in configuration file. Support for it can be added but it would require additional changes that might be invasive.
That would require removing data_directory entry from config file but also treating it as real PGDATA. That would allow us to use pgsql command with -D arg for upgrade process.
After finalizing, configuration entry should be restored.
@hhorak
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (postgresql:12 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1831 |
Created attachment 1760714 [details] pg-10-to-12-upgrade.log Description of problem: When data_directory is present in /var/lib/pgsql/data/postgresql.conf the upgrade fails because it can't start up postgresql with /var/lib/pgsql/data-old as a data directory. This is common for user who use the Puppet module puppetlabs/postgresql which always writes this setting. Version-Release number of selected component (if applicable): # dnf list installed postgresql* Installed Packages postgresql.x86_64 12.5-1.module_el8.4.0+597+7b8b5722 @appstream postgresql-server.x86_64 12.5-1.module_el8.4.0+597+7b8b5722 @appstream postgresql-upgrade.x86_64 12.5-1.module_el8.4.0+597+7b8b5722 @appstream How reproducible: Always Steps to Reproduce: 1. Install PostgreSQL 10 with some data on a clean CentOS 8 Stream dnf install postgresql-server -y postgresql-setup --initdb echo "data_directory = '/var/lib/pgsql/data'" >> /var/lib/pgsql/data/postgresql.conf systemctl start postgresql.service sudo -u postgres createdb mydb echo 'CREATE TABLE mytable (id INT); INSERT INTO mytable VALUES (1)' | sudo -u postgres psql mydb 2. Upgrade to 12 systemctl stop postgresql.service dnf module reset postgresql -y dnf module install postgresql:12 -y dnf install postgresql-upgrade -y postgresql-setup --upgrade Actual results: Upgrade breaks Expected results: Upgrade succeeds Additional info: See attachment for a full log.