Bug 2076294

Summary:	After restore from 6.10.2 (and older) backup to 6.10.3 candlepin is broken
Product:	Red Hat Satellite	Reporter:	Brad Buckingham <bbuckingham>
Component:	Satellite Maintain	Assignee:	Evgeni Golov <egolov>
Status:	CLOSED ERRATA	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.10.3	CC:	apatel, aupadhye, egolov, ehelms, gtalreja, kgaikwad, osousa, vijsingh, zhunting
Target Milestone:	6.10.6	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-foreman_maintain-0.8.30	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2069135	Environment:
Last Closed:	2022-05-31 16:37:24 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 2 Lukas Pramuk 2022-04-25 15:07:24 UTC

FailedQA.

@Satellite 6.10.5 Snap2
rubygem-foreman_maintain-0.8.27-1.el7sat.noarch
foreman-installer-2.5.2.14-1.el7sat.noarch

by the reproducer in comment#0:

1) Install 6.10.2 (using internal repo in order to be able to pin at 6.10.2)

2) Create/Have a 6.10.2 backup (or older)
# satellite-maintain backup offline /var/backup

3) Upgrade to 6.10.5 (used internal repo) and check candlepin status after upgrade
# hammer ping
...
candlepin:
    Status:          ok
    Server Response: Duration: 50ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 46ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms

4) Restore from the 6.10.2 backup
# satellite-maintain restore /var/backup/satellite-backup-*
...
| All services started                                                [OK]
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------

5) Check candlepin status after restore from the backup
# hammer ping
...
candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 2ms

>>> failed the same way

Additional information:

# ls -alh /var/lib/candlepin/.puppet*
-rw-r--r--. 1 root root 25 Apr 25 07:47 /var/lib/candlepin/.puppet-candlepin-cpdb-create-done
-rw-r--r--. 1 root root  6 Apr 25 07:47 /var/lib/candlepin/.puppet-candlepin-rpm-version

>>> puppet candlepin locks were not removed thus no candlepin db migration was run

Comment 7 Evgeni Golov 2022-04-26 13:31:49 UTC

https://github.com/theforeman/foreman_maintain/pull/609 / https://projects.theforeman.org/issues/34821 is done and needs backporting to the right f-m branch now.

Comment 10 Lukas Pramuk 2022-05-18 09:41:55 UTC

VERIFIED.

@Satellite 6.10.6 Snap2
rubygem-foreman_maintain-0.8.30-1.el7sat.noarch
foreman-installer-2.5.2.14-1.el7sat.noarch

by the reproducer in comment#0:

1) Install 6.10.2 (using internal repo in order to be able to pin at 6.10.2)

2) Create/Have a 6.10.2 backup (or older)
# satellite-maintain backup offline /var/backup

3) Upgrade to 6.10.6 (used internal repo) and check candlepin status after upgrade
# hammer ping
...
candlepin:
    Status:          ok
    Server Response: Duration: 50ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 46ms
candlepin_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms

4) Restore from the 6.10.2 backup
# satellite-maintain restore /var/backup/satellite-backup-*
...
| All services started                                                [OK]
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------

5) Check candlepin status after restore from the backup
# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 39ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 39ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:            
    Status:          ok
    Server Response: Duration: 167ms
pulp3_content:    
    Status:          FAIL
    Server Response: Message: No pulpcore content apps are running at https://sat.example.com/pulp/api/v3.
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 10ms

>>> candlepin status is green now as cpdb migration is being run upon restore

>>> there is now unrelated issue with pulp3_content (see comment#8 and comment#9)

Comment 13 errata-xmlrpc 2022-05-31 16:37:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.10.6 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4851