Bug 1550924

Summary:	[RFE] Enhance the cpdb validate with additional fix procedures + provide a link to the KCS
Product:	Red Hat Satellite	Reporter:	Ivan Necas <inecas>
Component:	Satellite Maintain	Assignee:	Kavita <kgaikwad>
Status:	CLOSED ERRATA	QA Contact:	Nikhil Kathole <nkathole>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.3.0	CC:	apatel, cmarinea, egolov, inecas, kgaikwad, nkathole, rjerrido
Target Milestone:	Unspecified	Keywords:	FutureFeature
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-04 14:02:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ivan Necas 2018-03-02 09:53:23 UTC

Description of problem:

So far, the failure on the validation was mostly caused by an issue described in the article:

https://access.redhat.com/solutions/3362821

The ask of this BZ is:

1. implement the fix procedure
2. in the error message of the failure, provide a link to the KCS article for further details (using a similar approach, as we have with foreman-tasks ui investigate https://github.com/theforeman/foreman_maintain/blob/master/definitions/procedures/foreman_tasks/ui_investigate.rb might be suitable here, with having a procedure called KnowledgeBase article.

Comment 3 Satellite Program 2018-03-05 11:18:31 UTC

Upstream bug assigned to kgaikwad

Comment 4 Satellite Program 2018-03-20 10:18:44 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22759 has been resolved.

Comment 5 Nikhil Kathole 2018-03-26 12:58:32 UTC

VERIFIED

Version tested:
# rpm -qa | grep maintain
rubygem-foreman_maintain-0.1.5-1.el7sat.noarch

Steps:
1. Manage to fail candlepin check

# su - postgres
-bash-4.2$ psql candlepin
psql (9.2.23)
Type "help" for help.

candlepin=# update cp_env_content set contentid='doesnt_exist' where id='8a889de06261a563016261cb7dc917fd';
UPDATE 1

2. # foreman-maintain upgrade check --target-version 6.3
Running Checks before upgrading to Satellite 6.3
================================================================================
Check for verifying syntax for ISP DHCP configurations:               [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using hammer ping:             [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:                                 [FAIL]
########## ERROR ############
Error running command: liquibase --driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/W
EB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url=jdbc:postgresql:candlepin --username=candlepin --password=K4oxn3ukwHeC8tb
Q5uMPXJsrYsJ3qnGd --logLevel=debug migrate -Dcommunity=False
Status code: 65280
Command output: DEBUG 3/26/18 8:33 AM:liquibase: Connected to candlepin@jdbc:postgresql:candlepin
DEBUG 3/26/18 8:33 AM:liquibase: Setting auto commit to false from true
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: select count(*) from public.databasechangeloglock
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1
DEBUG 3/26/18 8:33 AM:liquibase: Lock Database
DEBUG 3/26/18 8:33 AM:liquibase: Executing UPDATE database command: UPDATE public.databasechangeloglock SET LOCKED = TRUE, LOCKEDBY = 'qe-sat6-clie
nt-arch.usersys.redhat.com (192.168.100.1)', LOCKGRANTED = '2018-03-26 08:33:31.877' WHERE ID = 1 AND LOCKED = FALSE
INFO 3/26/18 8:33 AM:liquibase: Successfully acquired change log lock
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT MD5SUM FROM public.databasechangelog WHERE MD5SUM IS NOT NULL
INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP
TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: select count(*) from public.databasechangeloglock
INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP
TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for customChange:[
    class="org.candlepin.liquibase.PerOrgProductsMigrationValidationLiquibaseWrapper"
    param=[]
] as 9e8d4c92133890d6f26d4ca7c849b8dc
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for 7:9e8d4c92133890d6f26d4ca7c849b8dc: as 381a868ecfd0af4909cb5b6674596f8d
INFO 3/26/18 8:33 AM:liquibase: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIP
TION,COMMENTS FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
INFO 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::201711081
40157-1::crog: Reading from public.databasechangelog
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Executing QUERY database command: SELECT FILENAME,AUTHOR,ID,MD5SUM,DATEEXECUTED,ORDEREXECUTED,TAG,EXECTYPE,DESCRIPTION,COMMENTS FRO
M public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for customChange:[
    class="org.candlepin.liquibase.PerOrgProductsMigrationValidationLiquibaseWrapper"
    param=[]
] as 9e8d4c92133890d6f26d4ca7c849b8dc
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Computed checksum for 7:9e8d4c92133890d6f26d4ca7c849b8dc: as 381a868ecfd0af4909cb5b6674596f8d
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Running Changeset:db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Changeset db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Performs per-org-product specific migration validation
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108
140157-1::crog: Reading ChangeSet: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog
INFO 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::201711081
40157-1::crog: Validating data for org Default_Organization (8a889de06261980e01626199862c0001) (1 of 1)
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110
8140157-1::crog:   Environment "1" references unresolvable content: doesnt_exist
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110
8140157-1::crog: Org Default_Organization (8a889de06261980e01626199862c0001) failed data validation
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/changelog-validate.xml: db/changelog/20171108140157-perorgproducts-migration-validation.xml::2017110
8140157-1::crog: Change Set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog failed.  Error: liquibase.e
xception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more org
s failed data validation
        at liquibase.change.custom.CustomChangeWrapper.generateStatements(Unknown Source)
        at liquibase.database.AbstractJdbcDatabase.executeStatements(Unknown Source)
        at liquibase.changelog.ChangeSet.execute(Unknown Source)
        at liquibase.changelog.visitor.UpdateVisitor.visit(Unknown Source)
        at liquibase.changelog.ChangeLogIterator.run(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.integration.commandline.Main.doMigration(Unknown Source)
        at liquibase.integration.commandline.Main.main(Unknown Source)
Caused by: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:74)
        ... 9 more
Caused by: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.PerOrgProductsMigrationValidationTask.execute(PerOrgProductsMigrationValidationTask.java:189)
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:71)
        ... 9 more
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Release Database Lock
DEBUG 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Executing UPDATE data
base command: UPDATE public.databasechangeloglock SET LOCKED = FALSE, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1
INFO 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Successfully released
change log lock
Liquibase update Failed: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::cro
g:
     Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On
e or more orgs failed data validation
SEVERE 3/26/18 8:33 AM:liquibase: db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog: Migration failed for
 change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::20171108140157-1::crog:
     Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On
e or more orgs failed data validation
liquibase.exception.MigrationFailedException: Migration failed for change set db/changelog/20171108140157-perorgproducts-migration-validation.xml::
20171108140157-1::crog:
     Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: On
e or more orgs failed data validation
        at liquibase.changelog.ChangeSet.execute(Unknown Source)
        at liquibase.changelog.visitor.UpdateVisitor.visit(Unknown Source)
        at liquibase.changelog.ChangeLogIterator.run(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.Liquibase.update(Unknown Source)
        at liquibase.integration.commandline.Main.doMigration(Unknown Source)
        at liquibase.integration.commandline.Main.main(Unknown Source)
Caused by: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One
or more orgs failed data validation
        at liquibase.change.custom.CustomChangeWrapper.generateStatements(Unknown Source)
        at liquibase.database.AbstractJdbcDatabase.executeStatements(Unknown Source)
        ... 7 more
Caused by: liquibase.exception.CustomChangeException: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:74)
        ... 9 more
Caused by: liquibase.exception.DatabaseException: One or more orgs failed data validation
        at org.candlepin.liquibase.PerOrgProductsMigrationValidationTask.execute(PerOrgProductsMigrationValidationTask.java:189)
        at org.candlepin.liquibase.LiquibaseCustomTaskWrapper.execute(LiquibaseCustomTaskWrapper.java:71)
        ... 9 more


For more information, use the --logLevel flag
Validating Candlepin database
--driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-validate.xml --url=jdbc:postgresql:candlepin --username=candlepin --password=K4oxn3ukwHeC8tbQ5uMPXJsrYsJ3qnGd --logLevel=debug
Traceback (most recent call last):
  File "/usr/share/candlepin/cpdb", line 297, in <module>
    dbsetup.validate()
  File "/usr/share/candlepin/cpdb", line 70, in validate
    self._run_liquibase("db/changelog/changelog-validate.xml")
  File "/usr/share/candlepin/cpdb", line 106, in _run_liquibase
    output = run_command("liquibase %s migrate -Dcommunity=%s" % (liquibase_options, self.community))
  File "/usr/share/candlepin/cpdb", line 38, in run_command
    error_out(command, status, output)
  File "/usr/share/candlepin/cpdb", line 46, in error_out
    raise Exception("Error running command")
Exception: Error running command
--------------------------------------------------------------------------------
There are multiple steps to proceed:
1) Delete orphaned record(s) from cp_env_content with unresolvable content
2) Show knowledge base article for troubleshooting
Select step to continue, [n(next), q(quit)] 2
Show knowledge base article for troubleshooting:
Go to https://access.redhat.com/solutions/3362821
please follow steps from above article to resolve this issue
press ENTER once done.

If selected 1,

There are multiple steps to proceed:
1) Delete orphaned record(s) from cp_env_content with unresolvable content
2) Show knowledge base article for troubleshooting
Select step to continue, [n(next), q(quit)] 1
Delete orphaned record(s) from cp_env_content with unresolvable content:
| Taking a backup of the candlepin database
 Note: Database backup file path - /var/lib/foreman-maintain/db-backups/candlepin_2018-03-26_08-34-55.dump.bz2

 In case of any exception, use above dump file to restore DB.
Total 1 records with unresolvable content
/ Deleting record(s) from cp_env_content table                        [OK]
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check to validate candlepin database:                                 [OK]
--------------------------------------------------------------------------------

Comment 8 errata-xmlrpc 2018-04-04 14:02:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0633