Bug 2090820 - satellite upgrade to 6.11 fails in installer with "Could not open SSL root certificate file /root/.postgresql/root.crt" error for external DB setup
Summary: satellite upgrade to 6.11 fails in installer with "Could not open SSL root ce...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installation
Version: 6.11.0
Hardware: All
OS: Unspecified
unspecified
high
Target Milestone: 6.11.0
Assignee: Amit Upadhye
QA Contact: Gaurav Talreja
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-26 15:47 UTC by Gaurav Talreja
Modified: 2022-07-19 17:05 UTC (History)
5 users (show)

Fixed In Version: rubygem-foreman_maintain-1.0.12
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-07-05 14:35:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 35029 0 Normal Closed satellite upgrade to 6.11 fails in installer with "Could not open SSL root certificate file /root/.postgresql/root.crt" ... 2022-06-08 13:44:44 UTC
Red Hat Product Errata RHSA-2022:5498 0 None None None 2022-07-05 14:36:07 UTC

Description Gaurav Talreja 2022-05-26 15:47:44 UTC
Description of problem:


Version-Release number of selected component (if applicable):
Satellite 6.10.6 

How reproducible:
Always

Steps to Reproduce:
1. Prepare Satellite 6.10.6 with external PostgreSQL (with SSL)
2. Setup all required repositories and upgrade foreman-maintain packages
3. # foreman-maintain upgrade run --target-version 6.11 --whitelist='repositories-validate,repositories-setup'
...
Running Migration scripts to Satellite 6.11
================================================================================
Setup repositories:                                                   [SKIPPED]
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade: 2022-05-26 11:15:55 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-05-26 11:16:02 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-05-26 11:16:02 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-05-26 11:16:08 [WARN  ] [pre] Skipping system checks.
2022-05-26 11:16:08 [WARN  ] [pre] Skipping system checks.
2022-05-26 11:16:33 [NOTICE] [configure] Starting system configuration.
2022-05-26 11:16:48 [NOTICE] [configure] 250 configuration steps out of 1849 steps complete.
2022-05-26 11:17:04 [NOTICE] [configure] 500 configuration steps out of 1851 steps complete.
2022-05-26 11:17:11 [ERROR ] [configure] Execution of '/usr/share/candlepin/cpdb --update --dbhost=<satellite> --dbport=5432 --database=candlepin1db?ssl=true --user=candlepin1! --<passwd>=RedHat1!' returned 1: ########## ERROR ############
2022-05-26 11:17:11 [ERROR ] [configure] Error running command: /usr/share/candlepin/liquibase.sh --driver=org.postgresql.Driver --classpath=/var/lib/tomcat/webapps/candlepin/WEB-INF/lib/postgresql-42.3.3.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-update.xml --url="jdbc:postgresql://<satellite>:5432/candlepin1db?ssl=true" --username=$DBUSERNAME --<passwd>=$DB<passwd> --logLevel=severe migrate -Dcommunity=False
2022-05-26 11:17:11 [ERROR ] [configure] Status code: 65280
2022-05-26 11:17:11 [ERROR ] [configure] Command output: Liquibase update Failed: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.
2022-05-26 11:17:11 [ERROR ] [configure] SEVERE 5/26/22, 11:17 AM:liquibase: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.
2022-05-26 11:17:11 [ERROR ] [configure] liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:61)
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.integration.commandline.Main.doMigration(Main.java:788)
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.integration.commandline.Main.main(Main.java:133)
2022-05-26 11:17:11 [ERROR ] [configure] Caused by: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:231)
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:141)
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:52)
2022-05-26 11:17:11 [ERROR ] [configure] ... 2 more
2022-05-26 11:17:11 [ERROR ] [configure] Caused by: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.ssl.LibPQFactory.<init>(LibPQFactory.java:150)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.core.SocketFactoryFactory.getSslSocketFactory(SocketFactoryFactory.java:61)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:34)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:571)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:168)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.Driver.makeConnection(Driver.java:400)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.Driver.connect(Driver.java:259)
2022-05-26 11:17:11 [ERROR ] [configure] at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:223)
2022-05-26 11:17:11 [ERROR ] [configure] ... 4 more
2022-05-26 11:17:11 [ERROR ] [configure] Caused by: java.io.FileNotFoundException: /root/.postgresql/root.crt (No such file or directory)
2022-05-26 11:17:11 [ERROR ] [configure] at java.base/java.io.FileInputStream.open0(Native Method)
2022-05-26 11:17:11 [ERROR ] [configure] at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
2022-05-26 11:17:11 [ERROR ] [configure] at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
2022-05-26 11:17:11 [ERROR ] [configure] at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
2022-05-26 11:17:11 [ERROR ] [configure] at org.postgresql.ssl.LibPQFactory.<init>(LibPQFactory.java:147)
2022-05-26 11:17:11 [ERROR ] [configure] ... 14 more
...

Actual results:
Upgrade fails

Expected results:
Successful upgrade to 6.11

Additional info:
Seems related to https://bugzilla.redhat.com/show_bug.cgi?id=2062189

Comment 8 Amit Upadhye 2022-06-07 10:27:42 UTC
Created redmine issue https://projects.theforeman.org/issues/35029 from this bug

Comment 9 Bryan Kearney 2022-06-08 12:04:38 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35029 has been resolved.

Comment 10 Gaurav Talreja 2022-06-13 19:51:28 UTC
Hi,

Tested upgrade for Satellite with external DB SSL setup from 6.10.7 to 6.11, and it works perfectly by adding new required flag to the installer step in the 6.11 upgrade scenario.

--------------------------------------------------------------------------------
Run installer with Candlepin SSL CA when using external database with SSL:
- Running installer with --katello-candlepin-db-ssl-ca /usr/share/foreman/root.crt argument!
                                                                      [OK]
--------------------------------------------------------------------------------


As well, I check upgrades for regular Satellite and Satellite with external DB on non-SSL setup, which also return the below message for the installer step, so I was wondering if anyone knew if this flag is supposed to be set for these setups? if yes, so it is still set to UNDEF after an upgrade, or is it just a message from the description of the procedure?

--------------------------------------------------------------------------------
Run installer with Candlepin SSL CA when using external database with SSL:
| Executing installer                                                 [OK]
--------------------------------------------------------------------------------


As you can see, this description/message is misleading for regular and non-SSL Satellite setups, so I believe it should only be modified if `extdb_and_ssl?`, so how do you recommend handling this BZ?


Thanks,
Gaurav

Comment 11 Amit Upadhye 2022-06-14 10:25:50 UTC
Hello Gaurav,

Thanks for testing the change.

I feel the current messaging is correct as we also show if installer is getting executed with extra options in both cases. We can change the description of the procedure however I also feel that's the use of the procedure?

This should not be the blocker for the GA, if required another bugzilla can be opened to change the description if needed.

Regards,
Amit Upadhye.

Comment 12 Gaurav Talreja 2022-06-14 12:36:18 UTC
Hello Amit,

Thanks for looking into this.

>> I feel the current messaging is correct as we also show if an installer is getting executed with extra options in both cases. We can change the description of the procedure however I also feel that's the use of the procedure?

Looking at the code below I don't think an installer is executed with extra options in both cases, but shouldn't the extra options be specific to extdb_and_ssl? 
```
    def run
      if extdb_and_ssl?
        run_installer_with_extra_option
      else
        run_installer
      end
    end
```
and if the installer runs with extra options for both cases then I check it is not being set after a successful upgrade to 6.11 for regular and non-SSL external DB Satellite setups

# satellite-installer --full-help | grep katello-candlepin-db-ssl-ca
    --katello-candlepin-db-ssl-ca  The CA certificate to verify the SSL connection to the database with (current: UNDEF)
    --reset-katello-candlepin-db-ssl-ca Reset candlepin_db_ssl_ca to the default value (UNDEF)

>> This should not be the blocker for the GA, if required another bugzilla can be opened to change the description if needed.
Yes, I totally agree with you. I've opened a BZ to track this description issue separately BZ 2096849.

Hence verifying this BZ for Satellite 6.11.0 Snap 24.0 with version rubygem-foreman_maintain-1.0.12-1.el7sat.noarch

Comment 15 errata-xmlrpc 2022-07-05 14:35:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498


Note You need to log in before you can comment on or make changes to this bug.