Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1550924

Summary:	[RFE] Enhance the cpdb validate with additional fix procedures + provide a link to the KCS
Product:	Red Hat Satellite	Reporter:	Ivan Necas <inecas>
Component:	Satellite Maintain	Assignee:	Kavita <kgaikwad>
Status:	CLOSED ERRATA	QA Contact:	Nikhil Kathole <nkathole>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.3.0	CC:	apatel, cmarinea, egolov, inecas, kgaikwad, nkathole, rjerrido
Target Milestone:	Unspecified	Keywords:	FutureFeature
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-04 14:02:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ivan Necas 2018-03-02 09:53:23 UTC

Description of problem:

So far, the failure on the validation was mostly caused by an issue described in the article:

https://access.redhat.com/solutions/3362821

The ask of this BZ is:

1. implement the fix procedure
2. in the error message of the failure, provide a link to the KCS article for further details (using a similar approach, as we have with foreman-tasks ui investigate https://github.com/theforeman/foreman_maintain/blob/master/definitions/procedures/foreman_tasks/ui_investigate.rb might be suitable here, with having a procedure called KnowledgeBase article.

Comment 3 Satellite Program 2018-03-05 11:18:31 UTC

Upstream bug assigned to kgaikwad

Comment 4 Satellite Program 2018-03-20 10:18:44 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22759 has been resolved.

Comment 5 Nikhil Kathole 2018-03-26 12:58:32 UTC

VERIFIED

Version tested:
# rpm -qa | grep maintain
rubygem-foreman_maintain-0.1.5-1.el7sat.noarch

Steps:
1. Manage to fail candlepin check

# su - postgres
-bash-4.2$ psql candlepin
psql (9.2.23)
Type "help" for help.

candlepin=# update cp_env_content set contentid='doesnt_exist' where id='8a889de06261a563016261cb7dc917fd';
UPDATE 1

2. # foreman-maintain upgrade check --target-version 6.3
Running Checks before upgrading to Satellite 6.3
================================================================================
Check for verifying syntax for ISP DHCP configurations:               [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using hammer ping:             [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:                                 [FAIL]
########## ERROR ############
Error running command: liquibase --driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/W
EB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url=jdbc:postgresql:candlepin --username=candlepin --password=K4oxn3ukwHeC8tb
Q5uMPXJsrYsJ3qnGd --logLevel=debug migrate -Dcommunity=False
Status code: 65280
Command output: DEBUG 3/26/18 8:33 AM:liquibase: Connected to candlepin@jdbc:postgresql:candlepin
DEBUG 3/26/18 8:33 AM:liquibase: Setting auto commit to false from true
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: select count(*) from public.databasechangeloglock
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1
DEBUG 3/26/18 8:33 AM:liquibase: Lock Database
DEBUG 3/26/18 8:33 AM:liquibase: Executing UPDATE database command: UPDATE public.databasechangeloglock SET LOCKED = TRUE, LOCKEDBY = 'qe-sat6-clie
nt-arch.usersys.redhat.com (192.168.100.1)', LOCKGRANTED = '2018-03-26 08:33:31.877' WHERE ID = 1 AND LOCKED = FALSE
INFO 3/26/18 8:33 AM:liquibase: Successfully acquired change log lock
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT MD5SUM FROM public.databasechangelog WHERE MD5SUM IS NOT NULL
INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP
TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: select count(*) from public.databasechangeloglock
INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP
TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for customChange:[
    class="org.candlepin.liquibase.PerOrgProductsMigrationValidationLiquibaseWrapper"
    param=[]
] as 9e8d4c92133890d6f26d4ca7c849b8dc
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for 7:9e8d4c92133890d6f26d4ca7c849b8dc: as 381a868ecfd0af4909cb5b6674596f8d
INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP
TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
INFO 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::201711081
40157-1::crog: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIPTION,COMMENTS FRO
M public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for customChange:[
    class="org.candlepin.liquibase.PerOrgProductsMigrationValidationLiquibaseWrapper"
    param=[]
] as 9e8d4c92133890d6f26d4ca7c849b8dc
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for 7:9e8d4c92133890d6f26d4ca7c849b8dc: as 381a868ecfd0af4909cb5b6674596f8d
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Running Changeset:db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Changeset db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Performs per-org-product specific migration validation
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Reading ChangeSet: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog
INFO 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::201711081
40157-1::crog: Validating data for org Default_Organization (8a889de06261980e01626199862c0001) (1 of 1)
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110
8140157-1::crog:   Environment "1" references unresolvable content: doesnt_exist
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110
8140157-1::crog: Org Default_Organization (8a889de06261980e01626199862c0001) failed data validation
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110
8140157-1::crog: Change Set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog failed.  Error: liquibase.e
xception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more org
s failed data validation
        at liquibase.change.custom.CustomChangeWrapper.generateStatements(Unknown Source)
        at liquibase.database.AbstractJdbcDatabase.executeStatements(Unknown Source)
        at liquibase.changelog.ChangeSet.execute(Unknown Source)
        at liquibase.changelog.visitor.UpdateVisitor.visit(Unknown Source)
        at liquibase.changelog.ChangeLogIterator.run(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.integration.commandline.Main.doMigration(Unknown Source)
        at liquibase.integration.commandline.Main.main(Unknown Source)
Caused by: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:74)
        ... 9 more
Caused by: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.PerOrgProductsMigrationValidationTask.execute(PerOrgProductsMigrationValidationTask.java:189)
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:71)
        ... 9 more
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Release Database Lock
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Executing UPDATE data
base command: UPDATE public.databasechangeloglock SET LOCKED = FALSE, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1
INFO 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Successfully released
change log lock
Liquibase update Failed: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::cro
g:
     Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On
e or more orgs failed data validation
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Migration failed for
 change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog:
     Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On
e or more orgs failed data validation
liquibase.exception.MigrationFailedException: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::
20171108140157-1::crog:
     Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On
e or more orgs failed data validation
        at liquibase.changelog.ChangeSet.execute(Unknown Source)
        at liquibase.changelog.visitor.UpdateVisitor.visit(Unknown Source)
        at liquibase.changelog.ChangeLogIterator.run(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.integration.commandline.Main.doMigration(Unknown Source)
        at liquibase.integration.commandline.Main.main(Unknown Source)
Caused by: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One
or more orgs failed data validation
        at liquibase.change.custom.CustomChangeWrapper.generateStatements(Unknown Source)
        at liquibase.database.AbstractJdbcDatabase.executeStatements(Unknown Source)
        ... 7 more
Caused by: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:74)
        ... 9 more
Caused by: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.PerOrgProductsMigrationValidationTask.execute(PerOrgProductsMigrationValidationTask.java:189)
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:71)
        ... 9 more


For more information, use the --logLevel flag
Validating Candlepin database
--driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url=jdbc:postgresql:candlepin --username=candlepin --password=K4oxn3ukwHeC8tbQ5uMPXJsrYsJ3qnGd --logLevel=debug
Traceback (most recent call last):
  File "/usr/share/candlepin/cpdb", line 297, in <module>
    dbsetup.validate()
  File "/usr/share/candlepin/cpdb", line 70, in validate
    self._run_liquibase("db/changelog/changelog-validate.xml")
  File "/usr/share/candlepin/cpdb", line 106, in _run_liquibase
    output = run_command("liquibase %s migrate -Dcommunity=%s" % (liquibase_options, self.community))
  File "/usr/share/candlepin/cpdb", line 38, in run_command
    error_out(command, status, output)
  File "/usr/share/candlepin/cpdb", line 46, in error_out
    raise Exception("Error running command")
Exception: Error running command
--------------------------------------------------------------------------------
There are multiple steps to proceed:
1) Delete orphaned record(s) from cp_env_content with unresolvable content
2) Show knowledge base article for troubleshooting
Select step to continue, [n(next), q(quit)] 2
Show knowledge base article for troubleshooting:
Go to https://access.redhat.com/solutions/3362821
please follow steps from above article to resolve this issue
press ENTER once done.

If selected 1,

There are multiple steps to proceed:
1) Delete orphaned record(s) from cp_env_content with unresolvable content
2) Show knowledge base article for troubleshooting
Select step to continue, [n(next), q(quit)] 1
Delete orphaned record(s) from cp_env_content with unresolvable content:
| Taking a backup of the candlepin database
 Note: Database backup file path - /var/lib/foreman-maintain/db-backups/candlepin_2018-03-26_08-34-55.dump.bz2

 In case of any exception, use above dump file to restore DB.
Total 1 records with unresolvable content
/ Deleting record(s) from cp_env_content table                        [OK]
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check to validate candlepin database:                                 [OK]
--------------------------------------------------------------------------------

Comment 8 errata-xmlrpc 2018-04-04 14:02:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0633