Description of problem: So far, the failure on the validation was mostly caused by an issue described in the article: https://access.redhat.com/solutions/3362821 The ask of this BZ is: 1. implement the fix procedure 2. in the error message of the failure, provide a link to the KCS article for further details (using a similar approach, as we have with foreman-tasks ui investigate https://github.com/theforeman/foreman_maintain/blob/master/definitions/procedures/foreman_tasks/ui_investigate.rb might be suitable here, with having a procedure called KnowledgeBase article.
Upstream bug assigned to kgaikwad
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22759 has been resolved.
VERIFIED Version tested: # rpm -qa | grep maintain rubygem-foreman_maintain-0.1.5-1.el7sat.noarch Steps: 1. Manage to fail candlepin check # su - postgres -bash-4.2$ psql candlepin psql (9.2.23) Type "help" for help. candlepin=# update cp_env_content set contentid='doesnt_exist' where id='8a889de06261a563016261cb7dc917fd'; UPDATE 1 2. # foreman-maintain upgrade check --target-version 6.3 Running Checks before upgrading to Satellite 6.3 ================================================================================ Check for verifying syntax for ISP DHCP configurations: [OK] -------------------------------------------------------------------------------- Check for paused tasks: [OK] -------------------------------------------------------------------------------- Check whether all services are running using hammer ping: [OK] -------------------------------------------------------------------------------- Check to validate candlepin database: [FAIL] ########## ERROR ############ Error running command: liquibase --driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/W EB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url=jdbc:postgresql:candlepin --username=candlepin --password=K4oxn3ukwHeC8tb Q5uMPXJsrYsJ3qnGd --logLevel=debug migrate -Dcommunity=False Status code: 65280 Command output: DEBUG 3/26/18 8:33 AM:liquibase: Connected to candlepin@jdbc:postgresql:candlepin DEBUG 3/26/18 8:33 AM:liquibase: Setting auto commit to false from true DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: select count(*) from public.databasechangeloglock DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1 DEBUG 3/26/18 8:33 AM:liquibase: Lock Database DEBUG 3/26/18 8:33 AM:liquibase: Executing UPDATE database command: UPDATE public.databasechangeloglock SET LOCKED = TRUE, LOCKEDBY = 'qe-sat6-clie nt-arch.usersys.redhat.com (192.168.100.1)', LOCKGRANTED = '2018-03-26 08:33:31.877' WHERE ID = 1 AND LOCKED = FALSE INFO 3/26/18 8:33 AM:liquibase: Successfully acquired change log lock DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT MD5SUM FROM public.databasechangelog WHERE MD5SUM IS NOT NULL INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: select count(*) from public.databasechangeloglock INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Computed checksum for customChange:[ class="org.candlepin.liquibase.PerOrgProductsMigrationValidationLiquibaseWrapper" param=[] ] as 9e8d4c92133890d6f26d4ca7c849b8dc DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Computed checksum for 7:9e8d4c92133890d6f26d4ca7c849b8dc: as 381a868ecfd0af4909cb5b6674596f8d INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC INFO 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::201711081 40157-1::crog: Reading from public.databasechangelog DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIPTION,COMMENTS FRO M public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Computed checksum for customChange:[ class="org.candlepin.liquibase.PerOrgProductsMigrationValidationLiquibaseWrapper" param=[] ] as 9e8d4c92133890d6f26d4ca7c849b8dc DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Computed checksum for 7:9e8d4c92133890d6f26d4ca7c849b8dc: as 381a868ecfd0af4909cb5b6674596f8d DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Running Changeset:db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Changeset db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Performs per-org-product specific migration validation DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108 140157-1::crog: Reading ChangeSet: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog INFO 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::201711081 40157-1::crog: Validating data for org Default_Organization (8a889de06261980e01626199862c0001) (1 of 1) SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110 8140157-1::crog: Environment "1" references unresolvable content: doesnt_exist SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110 8140157-1::crog: Org Default_Organization (8a889de06261980e01626199862c0001) failed data validation SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110 8140157-1::crog: Change Set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog failed. Error: liquibase.e xception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more org s failed data validation at liquibase.change.custom.CustomChangeWrapper.generateStatements(Unknown Source) at liquibase.database.AbstractJdbcDatabase.executeStatements(Unknown Source) at liquibase.changelog.ChangeSet.execute(Unknown Source) at liquibase.changelog.visitor.UpdateVisitor.visit(Unknown Source) at liquibase.changelog.ChangeLogIterator.run(Unknown Source) at liquibase.Liquibase.update(Unknown Source) at liquibase.Liquibase.update(Unknown Source) at liquibase.integration.commandline.Main.doMigration(Unknown Source) at liquibase.integration.commandline.Main.main(Unknown Source) Caused by: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:74) ... 9 more Caused by: liquibase.exception.DatabaseException: One or more orgs failed data validation at org.candlepin.liquibase.PerOrgProductsMigrationValidationTask.execute(PerOrgProductsMigrationValidationTask.java:189) at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:71) ... 9 more DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Release Database Lock DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Executing UPDATE data base command: UPDATE public.databasechangeloglock SET LOCKED = FALSE, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1 INFO 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Successfully released change log lock Liquibase update Failed: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::cro g: Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On e or more orgs failed data validation SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On e or more orgs failed data validation liquibase.exception.MigrationFailedException: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml:: 20171108140157-1::crog: Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On e or more orgs failed data validation at liquibase.changelog.ChangeSet.execute(Unknown Source) at liquibase.changelog.visitor.UpdateVisitor.visit(Unknown Source) at liquibase.changelog.ChangeLogIterator.run(Unknown Source) at liquibase.Liquibase.update(Unknown Source) at liquibase.Liquibase.update(Unknown Source) at liquibase.integration.commandline.Main.doMigration(Unknown Source) at liquibase.integration.commandline.Main.main(Unknown Source) Caused by: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation at liquibase.change.custom.CustomChangeWrapper.generateStatements(Unknown Source) at liquibase.database.AbstractJdbcDatabase.executeStatements(Unknown Source) ... 7 more Caused by: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:74) ... 9 more Caused by: liquibase.exception.DatabaseException: One or more orgs failed data validation at org.candlepin.liquibase.PerOrgProductsMigrationValidationTask.execute(PerOrgProductsMigrationValidationTask.java:189) at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:71) ... 9 more For more information, use the --logLevel flag Validating Candlepin database --driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url=jdbc:postgresql:candlepin --username=candlepin --password=K4oxn3ukwHeC8tbQ5uMPXJsrYsJ3qnGd --logLevel=debug Traceback (most recent call last): File "/usr/share/candlepin/cpdb", line 297, in <module> dbsetup.validate() File "/usr/share/candlepin/cpdb", line 70, in validate self._run_liquibase("db/changelog/changelog-validate.xml") File "/usr/share/candlepin/cpdb", line 106, in _run_liquibase output = run_command("liquibase %s migrate -Dcommunity=%s" % (liquibase_options, self.community)) File "/usr/share/candlepin/cpdb", line 38, in run_command error_out(command, status, output) File "/usr/share/candlepin/cpdb", line 46, in error_out raise Exception("Error running command") Exception: Error running command -------------------------------------------------------------------------------- There are multiple steps to proceed: 1) Delete orphaned record(s) from cp_env_content with unresolvable content 2) Show knowledge base article for troubleshooting Select step to continue, [n(next), q(quit)] 2 Show knowledge base article for troubleshooting: Go to https://access.redhat.com/solutions/3362821 please follow steps from above article to resolve this issue press ENTER once done. If selected 1, There are multiple steps to proceed: 1) Delete orphaned record(s) from cp_env_content with unresolvable content 2) Show knowledge base article for troubleshooting Select step to continue, [n(next), q(quit)] 1 Delete orphaned record(s) from cp_env_content with unresolvable content: | Taking a backup of the candlepin database Note: Database backup file path - /var/lib/foreman-maintain/db-backups/candlepin_2018-03-26_08-34-55.dump.bz2 In case of any exception, use above dump file to restore DB. Total 1 records with unresolvable content / Deleting record(s) from cp_env_content table [OK] -------------------------------------------------------------------------------- Rerunning the check after fix procedure Check to validate candlepin database: [OK] --------------------------------------------------------------------------------
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0633