Bug 1993791
| Summary: | upgrade run fails on connecting to external candlepin | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Kenny Tordeurs <ktordeur> |
| Component: | Satellite Maintain | Assignee: | Amit Upadhye <aupadhye> |
| Status: | CLOSED ERRATA | QA Contact: | Gaurav Talreja <gtalreja> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.9.0 | CC: | ahumbe, apatel, aupadhye, kgaikwad, wclark, zhunting |
| Target Milestone: | 6.10.0 | Keywords: | Triaged, Upgrades |
| Target Release: | Unused | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | rubygem-foreman_maintain-0.8.18 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-16 13:48:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hello Kenny, I doubt you are hiting it because candlepin config(/etc/candlepin/candlepin.conf) is not having valid ext db details? And there is no change in candlepin config and there is no change in terms of how foreman-maintain queries the candlepin db, On my 6.9 system I dont see this happening before, at the time of upgrade and on 6.10 sat too candlepin db can be queried without auth errors, # bin/foreman-maintain service start Running Start Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-mongodb34-mongod, rh-redis5-redis, postgresql (candlepin), postgresql (pulpcore), postgresql (foreman), pulpcore-api, pulpcore-content, pulpcore-resource-manager, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy - starting rh-redis5-redis rh-mongodb34-mongod is remote and is UP. \ starting squid postgresql (candlepin) is remote and is UP. postgresql (pulpcore) is remote and is UP. postgresql (foreman) is remote and is UP. Check for paused tasks: [OK] -------------------------------------------------------------------------------- Check to verify no empty CA cert requests exist: [OK] -------------------------------------------------------------------------------- Check whether system is self-registered or not: [OK] -------------------------------------------------------------------------------- Check to make sure root(/) partition has enough space: [OK] -------------------------------------------------------------------------------- Check to make sure /var/lib/candlepin has enough space: [OK] -------------------------------------------------------------------------------- Check to validate candlepin database: [OK] _ Amit Upadhye Created redmine issue https://projects.theforeman.org/issues/33583 from this bug Verified. Tested on Satellite 6.9.7 Snap 1.0 Version: rubygem-foreman_maintain-0.8.19-1.el7sat.noarch (From Satellite 6.10.0 Snap 22) Setup: 1. Configure satellite with external Postgres database for foreman and candlepin. (Have special characters in username/password for foreman and candlepin, like ! or # or @) Steps: 1. # foreman-maintain upgrade run --target-version=6.10 Observation: Upgrade from 6.9.7 to 6.10 completed successfully, when having special characters in username/password for foreman and candlepin in external Postgres. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Satellite 6.10 Satellite Maintenance Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4697 |
Description of problem: #satellite-maintain upgrade run --target-version=6.10 fails to connect to external Candlepin -------------------------------------------------------------------------------- Check to validate candlepin database: [FAIL] ########## ERROR ############ Error running command: /usr/share/candlepin/liquibase.sh --driver=org.postgresql.Driver --classpath=/var/lib/tomcat/webapps/candlepin/WEB-INF/lib/postgresql-42.2.2.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db /changelog/changelog-validate.xml --url="jdbc:postgresql://ktordeur-external-postgres.sysmgmt.lan:5432/candlepin" --username=$DBUSERNAME --password=$DBPASSWORD --logLevel=debug migrate -Dcommunity=False Status code: 65280 Command output: Aug 15, 2021 4:07:48 PM org.postgresql.core.v3.ConnectionFactoryImpl log WARNING: SQLException occurred while connecting to ktordeur-external-postgres.sysmgmt.lan:5432 org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:473) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:452) at org.postgresql.Driver.connect(Driver.java:254) at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:223) at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:141) at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:52) at liquibase.integration.commandline.Main.doMigration(Main.java:788) at liquibase.integration.commandline.Main.main(Main.java:133) The connection works fine when testing: # PGPASSWORD='password' psql -h ktordeur-external-postgres.sysmgmt.lan -p 5432 -U candlepin -d candlepin -c "SELECT 1 as ping" ~~~ ping ------ 1 (1 row) ~~~ Version-Release number of selected component (if applicable): 6.9.4 rubygem-foreman_maintain-0.8.10-1.el7sat.noarch How reproducible: 100% Steps to Reproduce: 1. External postgres for foreman and candlepin 2. Attempt upgrade 3. Actual results: ~~~ Check to validate candlepin database: [FAIL] ########## ERROR ############ Error running command: /usr/share/candlepin/liquibase.sh --driver=org.postgresql.Driver --classpath=/var/lib/tomcat/webapps/candlepin/WEB-INF/lib/postgresql-42.2.2.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db /changelog/changelog-validate.xml --url="jdbc:postgresql://ktordeur-external-postgres.sysmgmt.lan:5432/candlepin" --username=$DBUSERNAME --password=$DBPASSWORD --logLevel=debug migrate -Dcommunity=False Status code: 65280 Command output: Aug 15, 2021 4:07:48 PM org.postgresql.core.v3.ConnectionFactoryImpl log WARNING: SQLException occurred while connecting to ktordeur-external-postgres.sysmgmt.lan:5432 org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:473) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:452) at org.postgresql.Driver.connect(Driver.java:254) at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:223) at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:141) at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:52) at liquibase.integration.commandline.Main.doMigration(Main.java:788) at liquibase.integration.commandline.Main.main(Main.java:133) Aug 15, 2021 4:07:48 PM org.postgresql.Driver connect SEVERE: Connection error: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:473) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:452) at org.postgresql.Driver.connect(Driver.java:254) at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:223) at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:141) at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:52) at liquibase.integration.commandline.Main.doMigration(Main.java:788) at liquibase.integration.commandline.Main.main(Main.java:133) Liquibase update Failed: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" SEVERE 8/15/21 4:07 PM:liquibase: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:61) at liquibase.integration.commandline.Main.doMigration(Main.java:788) at liquibase.integration.commandline.Main.main(Main.java:133) Caused by: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:231) at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:141) at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:52) ... 2 more Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin" at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:473) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:452) at org.postgresql.Driver.connect(Driver.java:254) at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:223) ... 4 more For more information, use the --logLevel flag Configuring PostgreSQL with JDBC URL: jdbc:postgresql://ktordeur-external-postgres.sysmgmt.lan:5432/candlepin Validating Candlepin database --driver=org.postgresql.Driver --classpath=/var/lib/tomcat/webapps/candlepin/WEB-INF/lib/postgresql-42.2.2.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url="jdbc:postgresql: //ktordeur-external-postgres.sysmgmt.lan:5432/candlepin" --username=$DBUSERNAME --password=$DBPASSWORD --logLevel=debug Traceback (most recent call last): File "/usr/share/candlepin/cpdb", line 286, in <module> dbsetup.validate() File "/usr/share/candlepin/cpdb", line 75, in validate self._run_liquibase("db/changelog/changelog-validate.xml") File "/usr/share/candlepin/cpdb", line 113, in _run_liquibase output = run_command("/usr/share/candlepin/liquibase.sh %s migrate -Dcommunity=%s" % (liquibase_options, self.community)) File "/usr/share/candlepin/cpdb", line 43, in run_command error_out(command, status, output) File "/usr/share/candlepin/cpdb", line 51, in error_out raise Exception("Error running command") Exception: Error running command ~~~ Expected results: No errors Additional info: ~~~ # hammer ping database: Status: ok Server Response: Duration: 0ms candlepin: Status: ok Server Response: Duration: 47ms candlepin_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms candlepin_auth: Status: ok Server Response: Duration: 28ms katello_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms pulp: Status: ok Server Response: Duration: 119ms pulp_auth: Status: ok Server Response: Duration: 45ms foreman_tasks: Status: ok Server Response: Duration: 8ms ~~~ ~~~ # satellite-maintain service status --only postgresql Running Status Services ================================================================================ Get status of applicable services: Displaying the following service(s): postgresql (candlepin), postgresql (foreman), postgresql | displaying postgresql (candlepin) postgresql (candlepin) is remote and is UP. | displaying postgresql (foreman) postgresql (foreman) is remote and is UP. | displaying postgresql ● postgresql.service - PostgreSQL database server Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2021-08-15 10:49:16 CEST; 22h ago Main PID: 2349 (postmaster) Tasks: 8 CGroup: /system.slice/postgresql.service ├─2349 postmaster -D /var/opt/rh/rh-postgresql12/lib/pgsql/data ├─2652 postgres: logger ├─2711 postgres: checkpointer ├─2712 postgres: background writer ├─2713 postgres: walwriter ├─2715 postgres: autovacuum launcher ├─2716 postgres: stats collector └─2720 postgres: logical replication launcher Aug 15 10:49:15 ktordeur-satellite-latest.sysmgmt.lan systemd[1]: Starting PostgreSQL database server... Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT LOG: starting PostgreSQL 12.7 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT LOG: listening on IPv6 address "::1", port 5432 Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT LOG: listening on IPv4 address "127.0.0.1", port 5432 Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT LOG: redirecting log output to logging collector process Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan sh[2349]: 2021-08-15 04:49:16 EDT HINT: Future log output will appear in directory "log". Aug 15 10:49:16 ktordeur-satellite-latest.sysmgmt.lan systemd[1]: Started PostgreSQL database server. | All services are running [OK] -------------------------------------------------------------------------------- ~~~