Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2076294 - After restore from 6.10.2 (and older) backup to 6.10.3 candlepin is broken
Summary: After restore from 6.10.2 (and older) backup to 6.10.3 candlepin is broken
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Satellite Maintain
Version: 6.10.3
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.10.6
Assignee: Evgeni Golov
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-18 15:16 UTC by Brad Buckingham
Modified: 2022-07-19 11:08 UTC (History)
9 users (show)

Fixed In Version: rubygem-foreman_maintain-0.8.30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2069135
Environment:
Last Closed: 2022-05-31 16:37:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 33281 0 Normal Closed Restore should run "upgrade" steps after the data restore finished 2022-04-18 15:16:35 UTC
Foreman Issue Tracker 34686 0 Normal Closed --reset-data does not remove /var/lib/candlepin/.puppet-candlepin-rpm-version 2022-04-18 15:16:35 UTC
Foreman Issue Tracker 34821 0 Normal New restoring an older candlepin DB results in broken system 2022-04-26 11:06:03 UTC
Red Hat Product Errata RHBA-2022:4851 0 None None None 2022-05-31 16:37:29 UTC

Comment 2 Lukas Pramuk 2022-04-25 15:07:24 UTC 
FailedQA.

@Satellite 6.10.5 Snap2
rubygem-foreman_maintain-0.8.27-1.el7sat.noarch
foreman-installer-2.5.2.14-1.el7sat.noarch

by the reproducer in comment#0:

1) Install 6.10.2 (using internal repo in order to be able to pin at 6.10.2)

2) Create/Have a 6.10.2 backup (or older)
# satellite-maintain backup offline /var/backup

3) Upgrade to 6.10.5 (used internal repo) and check candlepin status after upgrade
# hammer ping
...
candlepin:
    Status:          ok
    Server Response: Duration: 50ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 46ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms

4) Restore from the 6.10.2 backup
# satellite-maintain restore /var/backup/satellite-backup-*
...
| All services started                                                [OK]
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------

5) Check candlepin status after restore from the backup
# hammer ping
...
candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 2ms

>>> failed the same way

Additional information:

# ls -alh /var/lib/candlepin/.puppet*
-rw-r--r--. 1 root root 25 Apr 25 07:47 /var/lib/candlepin/.puppet-candlepin-cpdb-create-done
-rw-r--r--. 1 root root  6 Apr 25 07:47 /var/lib/candlepin/.puppet-candlepin-rpm-version

>>> puppet candlepin locks were not removed thus no candlepin db migration was run
Comment 7 Evgeni Golov 2022-04-26 13:31:49 UTC 
https://github.com/theforeman/foreman_maintain/pull/609 / https://projects.theforeman.org/issues/34821 is done and needs backporting to the right f-m branch now.
Comment 10 Lukas Pramuk 2022-05-18 09:41:55 UTC 
VERIFIED.

@Satellite 6.10.6 Snap2
rubygem-foreman_maintain-0.8.30-1.el7sat.noarch
foreman-installer-2.5.2.14-1.el7sat.noarch

by the reproducer in comment#0:

1) Install 6.10.2 (using internal repo in order to be able to pin at 6.10.2)

2) Create/Have a 6.10.2 backup (or older)
# satellite-maintain backup offline /var/backup

3) Upgrade to 6.10.6 (used internal repo) and check candlepin status after upgrade
# hammer ping
...
candlepin:
    Status:          ok
    Server Response: Duration: 50ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 46ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms

4) Restore from the 6.10.2 backup
# satellite-maintain restore /var/backup/satellite-backup-*
...
| All services started                                                [OK]
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------

5) Check candlepin status after restore from the backup
# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 39ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 39ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:            
    Status:          ok
    Server Response: Duration: 167ms
pulp3_content:    
    Status:          FAIL
    Server Response: Message: No pulpcore content apps are running at https://sat.example.com/pulp/api/v3.
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 10ms

>>> candlepin status is green now as cpdb migration is being run upon restore

>>> there is now unrelated issue with pulp3_content (see comment#8 and comment#9)
Comment 13 errata-xmlrpc 2022-05-31 16:37:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.10.6 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4851
Note You need to log in before you can comment on or make changes to this bug.