Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2076294

Summary:	After restore from 6.10.2 (and older) backup to 6.10.3 candlepin is broken
Product:	Red Hat Satellite	Reporter:	Brad Buckingham <bbuckingham>
Component:	Satellite Maintain	Assignee:	Evgeni Golov <egolov>
Status:	CLOSED ERRATA	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.10.3	CC:	apatel, aupadhye, egolov, ehelms, gtalreja, kgaikwad, osousa, vijsingh, zhunting
Target Milestone:	6.10.6	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-foreman_maintain-0.8.30	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2069135	Environment:
Last Closed:	2022-05-31 16:37:24 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 2 Lukas Pramuk 2022-04-25 15:07:24 UTC

FailedQA.

@Satellite 6.10.5 Snap2
rubygem-foreman_maintain-0.8.27-1.el7sat.noarch
foreman-installer-2.5.2.14-1.el7sat.noarch

by the reproducer in comment#0:

1) Install 6.10.2 (using internal repo in order to be able to pin at 6.10.2)

2) Create/Have a 6.10.2 backup (or older)
# satellite-maintain backup offline /var/backup

3) Upgrade to 6.10.5 (used internal repo) and check candlepin status after upgrade
# hammer ping
...
candlepin:
    Status:          ok
    Server Response: Duration: 50ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 46ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms

4) Restore from the 6.10.2 backup
# satellite-maintain restore /var/backup/satellite-backup-*
...
| All services started                                                [OK]
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------

5) Check candlepin status after restore from the backup
# hammer ping
...
candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 2ms

>>> failed the same way

Additional information:

# ls -alh /var/lib/candlepin/.puppet*
-rw-r--r--. 1 root root 25 Apr 25 07:47 /var/lib/candlepin/.puppet-candlepin-cpdb-create-done
-rw-r--r--. 1 root root  6 Apr 25 07:47 /var/lib/candlepin/.puppet-candlepin-rpm-version

>>> puppet candlepin locks were not removed thus no candlepin db migration was run

Comment 7 Evgeni Golov 2022-04-26 13:31:49 UTC

https://github.com/theforeman/foreman_maintain/pull/609 / https://projects.theforeman.org/issues/34821 is done and needs backporting to the right f-m branch now.

Comment 10 Lukas Pramuk 2022-05-18 09:41:55 UTC

VERIFIED.

@Satellite 6.10.6 Snap2
rubygem-foreman_maintain-0.8.30-1.el7sat.noarch
foreman-installer-2.5.2.14-1.el7sat.noarch

by the reproducer in comment#0:

1) Install 6.10.2 (using internal repo in order to be able to pin at 6.10.2)

2) Create/Have a 6.10.2 backup (or older)
# satellite-maintain backup offline /var/backup

3) Upgrade to 6.10.6 (used internal repo) and check candlepin status after upgrade
# hammer ping
...
candlepin:
    Status:          ok
    Server Response: Duration: 50ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 46ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms

4) Restore from the 6.10.2 backup
# satellite-maintain restore /var/backup/satellite-backup-*
...
| All services started                                                [OK]
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------

5) Check candlepin status after restore from the backup
# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 39ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 39ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:            
    Status:          ok
    Server Response: Duration: 167ms
pulp3_content:    
    Status:          FAIL
    Server Response: Message: No pulpcore content apps are running at https://sat.example.com/pulp/api/v3.
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 10ms

>>> candlepin status is green now as cpdb migration is being run upon restore

>>> there is now unrelated issue with pulp3_content (see comment#8 and comment#9)

Comment 13 errata-xmlrpc 2022-05-31 16:37:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.10.6 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4851