Bug 2181744

Summary:	[RFE] Add a new prerequisite to take backup of product certificates prior to attempting the leapp upgrade
Product:	Red Hat Enterprise Linux 7	Reporter:	Sayan Das <saydas>
Component:	leapp-repository	Assignee:	Leapp Notifications Bot <leapp-notifications-bot>
Status:	NEW ---	QA Contact:	upgrades-and-conversions
Severity:	medium	Docs Contact:
Priority:	medium
Version:	7.9	CC:	agadhave, ehelms, prjagtap, pstodulk
Target Milestone:	rc	Keywords:	Documentation, Triaged
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sayan Das 2023-03-25 15:13:02 UTC

Document URL: 

Section Number and Name: 

Describe the issue: 

Suggestions for improvement: 

Additional information: 
Document URL: 

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html-single/upgrading_and_updating_red_hat_satellite/index#upgrading-satellite-or-proxy-in-place-using-leapp_upgrade-guide

Section Number and Name: 

Chapter 4. Upgrading Satellite or Capsule to Red Hat Enterprise Linux 8 In-Place Using Leapp

Prerequisites


Describe the issue: 

If for some reason, leapp fails to convert a 6.11 satellite\capsule's OS to RHEL 8, during reboot then users can boot back with the RHEL 7 kernel but The product certifcates will be missing from the RHEL 7 environment and would further cause the satellite\capsule to not able to see any repos to enable at OS level.  

When this happens, Users goes into panic mode and RH support would need to help them out by providing them with the details of required product certs to be restored inside the /etc/pki/product and /etc/pki/product-default directories. 

If users can keep a backup of those directories even before starting the leapp upgrade, then this would of a great help.


Suggestions for improvement: 

Add a point in the prerequisites section i.e. 

* Keep a backup of the /etc/pki/product and /etc/pki/product-default directories in a safe location. If OS conversion fails during the reboot after leapp upgrade, Then you can boot back with the RHEL 7 kernel and would require to restore those directory contents to retry the OS conversion.  


Or anything suitable as per doc standards that supports my statement. 


Additional information: 

NA

Comment 2 Sayan Das 2023-03-25 15:20:00 UTC

NOTE: This BZ is only applicable for the 6.11 doc but not for any other version of satellite

Comment 7 Petr Stodulka 2023-04-03 15:52:14 UTC

Hi Sayan, thank you for the report. However I am confused as IIRC they will have to fix much more stuff then only content of /etc/pki/product* directories regarding your description. Additionally it seems as duplicate information as we have already written that customers are supposed to do the backup of the whole system prior the upgrade. So /etc/pki should be covered by that. Quote:
~~~
Always back up your system prior to upgrading. For example, you can use the Relax-and-Recover (ReaR) utility, LVM snapshots, RAID splitting, or a virtual machine snapshot.
~~~

Also, /etc/pki is not the only thing they have to fix after getting back. The system has already applied additional multiple changes and /etc/pki would be my least concern to be honest. In time the certificates are switched on the host systems, booting back with RHEL 7 kernel does not mean they booted to RHEL 7 system at all. IIRC, the only possible recover from the phase when the switch of certificates in that phase is the complete recover from the full system back up (including reload of the original bootloader). But I had not time to go from the attached cases yet so I cannot say, so possibly I misunderstand the problem from the description.

We will look at attached cases later to investigate what exactly happened, but from my current level of understanding this does not seems to fix the problem and it will provide more text for the reading.

Additional info:
* the product certificate under /etc/pki/product-default is owned by rpm, so if changed it means that target RHEL packages are already installed and system needs to be recovered from the back up
* certificates under /etc/pki/product are managed by RHSM, that is supposed to recover certs for the used product when not present (does not have to work when the system is in inconsistent state, mixing the content for target and original system)

Comment 9 Sayan Das 2023-04-03 16:03:35 UTC

The three support cases associated with this support case are all related to either satellite or capsules being leapp'ed.

We have often found ourselves in a scenario where, For some reason CU had to boot back with the RHEL 7 kernel after a failed RHEl conversion ( post a successful leapp + reboot ), but then users could not retry leapp as there are no repos present. 

When I got them the product certs related to RHEL and Satellite\rhscl\ansible created inside /etc/pki/product, Then

* They were able to get the repos visible for sub-man\yum\leapp
* They fixed the underlying issues regarding the last failure
* Then they retried the leapp preupgrade + upgrade or else just continued on that RHEL 7 instance. 


Yes, Of course, depending on how far the conversion had taken place during the reboot, We may need to perform additional steps to get back a stable working RHEL 7 instance but the first blocker is to get repos working and without product certs, they won't. 

You may check either of the cases attached to the BZ and you will get the same gist as I am explaining here. Long story short, If users are happy and can get their work done just by restoring the product certs instead of needing to restore the whole system ( which they don't want to do most of the time ), I would definitely want to see the step for backing up /etc/pki/product , additionally.

And my whole intention was to put that step in the Satellite 6.11 upgrade guide as a special pre-requisite for satellite\capsule customers ( as you can see in Description \ Comment 0 )