I forgot to mention this, but I initially found this on regular CentOS Linux. I didn't have a RHEL installation at hand, but I'd guess it also affects RHEL.

Thanks for reporting. Confirmed on RHEL 8.5 release as well:

[root@ci-vm-10-0-137-204 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[root@ci-vm-10-0-137-204 ~]# postgresql-setup --upgrade
 * Upgrading database.
ERROR: pg_upgrade tool failed
ERROR: Upgrade failed.
 * See /var/lib/pgsql/upgrade_postgresql.log for details.

[root@ci-vm-10-0-137-204 ~]# cat /var/lib/pgsql/upgrade_postgresql.log
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/lib/pgsql/.s.PGSQL.5432"?

could not connect to source postmaster started with the command:
"/usr/lib64/pgsql/postgresql-10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data-old" -o "-p 5432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql'" start
Failure, exiting

One more log that shows what is hapenning:

[root@ci-vm-10-0-137-204 ~]# cat /var/lib/pgsql/pg_upgrade_server.log

-----------------------------------------------------------------
  pg_upgrade run on Wed Apr 14 11:30:38 2021
-----------------------------------------------------------------

command: "/usr/lib64/pgsql/postgresql-10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data-old" -o "-p 5432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2021-04-14 11:30:38.080 EDT [8083] FATAL:  database files are incompatible with server
2021-04-14 11:30:38.080 EDT [8083] DETAIL:  The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 10.15.
 stopped waiting
pg_ctl: could not start server
Examine the log output.

It all looks like the combination of -D on the command-line and data_directory option in the config file is not doing what the upgrade process expects. It's not clear whether the option parsing is not clever enough here (didn't find a clear definition of what is the expected behavior in this case), or whether the upgrade should make sure the option is not present. This seems to be worth discussing with upstream.

(In reply to Honza Horak from comment #4)
> This seems to be worth discussing with upstream.

Doing that:
https://www.postgresql.org/message-id/0b24aa0a-c0c6-e870-af11-e18400ada98b%40redhat.com

The upstream's response actually suggests to not use (or filter out) the data_directory option when we manipulate with the data directory:
https://www.postgresql.org/message-id/3566642.1618422939%40sss.pgh.pa.us

Or we can alternatively stop the upgrade early in case data_directory is defined in the postgresql.conf.

The upgrade process should be likely interrupted in case the data_directory has a different value than the default.

This may be also a great content for Insights or/and LEAPP.

I blocked execution for:

- postgresql-setup --upgrade
- postgresql-setup --initdb

when `data_directory` entry in config file is detected. According to postgresql docs:

> Notice that data_directory overrides -D and PGDATA for the location of the data directory, but not for the location of the configuration files.

That means, when user define pgdata folder in config file it will take a priority over ENV variable in .service file and -D parameter in command line.

Currently postgresql-setup does not take into consideration `data_directory` entry in configuration file. Support for it can be added but it would require additional changes that might be invasive.

That would require removing data_directory entry from config file but also treating it as real PGDATA. That would allow us to use pgsql command with -D arg for upgrade process.

After finalizing, configuration entry should be restored.

@hhorak

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (postgresql:12 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1831