Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1996476 - Failed executing foreman-rake katello:pulp3_migration, exit status 1: rake aborted!
Summary: Failed executing foreman-rake katello:pulp3_migration, exit status 1: rake a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Satellite Maintain
Version: 6.10.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: 6.10.0
Assignee: Amit Upadhye
QA Contact: Gaurav Talreja
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-22 20:02 UTC by Taft Sanders
Modified: 2021-12-01 09:07 UTC (History)
7 users (show)

Fixed In Version: rubygem-foreman_maintain-0.8.14
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 13:48:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 33346 0 None None None 2021-08-25 14:32:37 UTC
Red Hat Product Errata RHBA-2021:4697 0 None None None 2021-11-16 13:48:29 UTC

Description Taft Sanders 2021-08-22 20:02:38 UTC
Description of problem:
During the upgrade the beta repos were set and the environment variable "FOREMAN_MAINTAIN_USE_BETA='1'" was not used. To work around this the whitelist option of 'repositories-validate' was used.

After the pulp3 migration completed the installer runs again to upgrade with pulp3 settings. Once the installer reaches the point of:
Re-running the installer to switch specified content over to pulp3    [OK]
--------------------------------------------------------------------------------
Setup repositories: 

the installer fails because the name of this whitelist step is 'repositories-setup'. After adding this step to the whitelist options and running again, the installer fails to run continuously because the content is already migrated but the migration steps from the installer are still being called every time.

Version-Release number of selected component (if applicable):
6.10.0 Snap14

How reproducible:
Every time

Steps to Reproduce:
1. Do not use standard repositories or environment variable "FOREMAN_MAINTAIN_USE_BETA='1'" or figure out another way to cause the installer to fail AFTER the pulp migration
2. Upgrade using installer option '--whitelist="repositories-validate"'
3.

Actual results:
Installer fails continuously because pulp content is already migrated

Expected results:
After the pulp content is upgraded and the installer re-runs to upgrade the Satellite with pulp3 as seen here in this step:
===============================
Starting the following service(s):
pulpcore-api, pulpcore-content, pulpcore-resource-manager, pulpcore-worker@1, pulpcore-worker@2, pulpcore-worker@3, pulpcore-worker@4
/ All services started                                                [OK]      
--------------------------------------------------------------------------------
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content
Checking for valid Katello configuraton.
Starting task.
2021-08-22 14:27:36 -0400: Importing migrated content type erratum: 14000/12657            
Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated

Switching specified content over to pulp 3
Checking for valid Katello configuraton.
Content Switchover completed successfully
Re-running the installer to switch specified content over to pulp3    [OK]
===============================
The installer should be able to detect that the pulp migration has completed and is not needed again.

Additional info:
After hitting this, setting the environment variable "FOREMAN_MAINTAIN_USE_BETA='1'" and/or including the whitelist value 'repositories-setup' has no effect on the failing outcome of the upgrade. This suggests that currently, at this point or any other point in the re-run of the upgrade after the migration will indefinitely fail the upgrade.
Whitelisting this step appears to be the only way to continue forward with the upgrade which results in the failure of the pulp3 migration and hammer ping being the next failure step in the upgrade:
# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 38ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 35ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp:             
    Status:          ok
    Server Response: Duration: 100ms
pulp_auth:        
    Status:          ok
    Server Response: Duration: 43ms
pulp3:            
    Status:          FAIL
    Server Response: Message: 503 Service Unavailable
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 3ms

Comment 11 Gaurav Talreja 2021-09-09 18:22:20 UTC
Verified.

Tested on Satellite 6.10.0 Snap 17.0
Version: rubygem-foreman_maintain-0.8.14-1.el7sat.noarch

Steps:
1. Perform pulp2to3 migration
2. Run twice # foreman-maintain upgrade run --target-version 6.10 --whitelist="repositories-validate" (Fails post-switchover for repositories-setup)

Observation:
For the first time, the switchover completes, the second time switchover gets skipped.
If the upgrade fails post-switchover/post-package upgrade, suppose in installer like BZ[1] with packages updated including satellite and katello, so content-switchover step will be skipped from that scenario if a version of katello > 4.0

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2001229

Comment 14 errata-xmlrpc 2021-11-16 13:48:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.10 Satellite Maintenance Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4697


Note You need to log in before you can comment on or make changes to this bug.