Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2181744

Summary: [RFE] Add a new prerequisite to take backup of product certificates prior to attempting the leapp upgrade
Product: Red Hat Enterprise Linux 7 Reporter: Sayan Das <saydas>
Component: leapp-repositoryAssignee: Leapp Notifications Bot <leapp-notifications-bot>
Status: CLOSED MIGRATED QA Contact: upgrades-and-conversions
Severity: medium Docs Contact:
Priority: medium    
Version: 7.9CC: agadhave, ehelms, prjagtap, pstodulk
Target Milestone: rcKeywords: Documentation, MigratedToJIRA, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-12 13:24:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sayan Das 2023-03-25 15:13:02 UTC
Document URL: 

Section Number and Name: 

Describe the issue: 

Suggestions for improvement: 

Additional information: 
Document URL: 

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html-single/upgrading_and_updating_red_hat_satellite/index#upgrading-satellite-or-proxy-in-place-using-leapp_upgrade-guide

Section Number and Name: 

Chapter 4. Upgrading Satellite or Capsule to Red Hat Enterprise Linux 8 In-Place Using Leapp

Prerequisites


Describe the issue: 

If for some reason, leapp fails to convert a 6.11 satellite\capsule's OS to RHEL 8, during reboot then users can boot back with the RHEL 7 kernel but The product certifcates will be missing from the RHEL 7 environment and would further cause the satellite\capsule to not able to see any repos to enable at OS level.  

When this happens, Users goes into panic mode and RH support would need to help them out by providing them with the details of required product certs to be restored inside the /etc/pki/product and /etc/pki/product-default directories. 

If users can keep a backup of those directories even before starting the leapp upgrade, then this would of a great help.


Suggestions for improvement: 

Add a point in the prerequisites section i.e. 

* Keep a backup of the /etc/pki/product and /etc/pki/product-default directories in a safe location. If OS conversion fails during the reboot after leapp upgrade, Then you can boot back with the RHEL 7 kernel and would require to restore those directory contents to retry the OS conversion.  


Or anything suitable as per doc standards that supports my statement. 


Additional information: 

NA

Comment 2 Sayan Das 2023-03-25 15:20:00 UTC
NOTE: This BZ is only applicable for the 6.11 doc but not for any other version of satellite

Comment 7 Petr Stodulka 2023-04-03 15:52:14 UTC
Hi Sayan, thank you for the report. However I am confused as IIRC they will have to fix much more stuff then only content of /etc/pki/product* directories regarding your description. Additionally it seems as duplicate information as we have already written that customers are supposed to do the backup of the whole system prior the upgrade. So /etc/pki should be covered by that. Quote:
~~~
Always back up your system prior to upgrading. For example, you can use the Relax-and-Recover (ReaR) utility, LVM snapshots, RAID splitting, or a virtual machine snapshot.
~~~

Also, /etc/pki is not the only thing they have to fix after getting back. The system has already applied additional multiple changes and /etc/pki would be my least concern to be honest. In time the certificates are switched on the host systems, booting back with RHEL 7 kernel does not mean they booted to RHEL 7 system at all. IIRC, the only possible recover from the phase when the switch of certificates in that phase is the complete recover from the full system back up (including reload of the original bootloader). But I had not time to go from the attached cases yet so I cannot say, so possibly I misunderstand the problem from the description.

We will look at attached cases later to investigate what exactly happened, but from my current level of understanding this does not seems to fix the problem and it will provide more text for the reading.

Additional info:
* the product certificate under /etc/pki/product-default is owned by rpm, so if changed it means that target RHEL packages are already installed and system needs to be recovered from the back up
* certificates under /etc/pki/product are managed by RHSM, that is supposed to recover certs for the used product when not present (does not have to work when the system is in inconsistent state, mixing the content for target and original system)

Comment 9 Sayan Das 2023-04-03 16:03:35 UTC
The three support cases associated with this support case are all related to either satellite or capsules being leapp'ed.

We have often found ourselves in a scenario where, For some reason CU had to boot back with the RHEL 7 kernel after a failed RHEl conversion ( post a successful leapp + reboot ), but then users could not retry leapp as there are no repos present. 

When I got them the product certs related to RHEL and Satellite\rhscl\ansible created inside /etc/pki/product, Then

* They were able to get the repos visible for sub-man\yum\leapp
* They fixed the underlying issues regarding the last failure
* Then they retried the leapp preupgrade + upgrade or else just continued on that RHEL 7 instance. 


Yes, Of course, depending on how far the conversion had taken place during the reboot, We may need to perform additional steps to get back a stable working RHEL 7 instance but the first blocker is to get repos working and without product certs, they won't. 

You may check either of the cases attached to the BZ and you will get the same gist as I am explaining here. Long story short, If users are happy and can get their work done just by restoring the product certs instead of needing to restore the whole system ( which they don't want to do most of the time ), I would definitely want to see the step for backing up /etc/pki/product , additionally.

And my whole intention was to put that step in the Satellite 6.11 upgrade guide as a special pre-requisite for satellite\capsule customers ( as you can see in Description \ Comment 0 )

Comment 11 RHEL Program Management 2023-09-12 13:19:25 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 12 RHEL Program Management 2023-09-12 13:24:53 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.