Bug 2176361

I believe the problem is that, when https://access.redhat.com/errata/RHSA-2023:1151 was released, Sat 6.11.5 got an updated candlepin and candlepin-selinux i.e. version 4.1.19-1.el8sat

This specific version of candlepin-selinux i.e. 4.1.19-1.el8sat, requires selinux-policy package of version 3.14.3-108.el8 which is not part of RHEL 8.6. It was released for RHEL 8.7. It's not even part of 8.6 EUS as well.

But, leapp cannot go to 8.7. It can only target 8.6 as the latest version to upgrade to. 

So that effectively makes this BZ a blocker for RHEL 7.9 -> 8.6 upgrade via leapp when the Satellite version is 6.11.5 ( on el7 ).

We don't officially support it but leapp has a way to get the system upgraded to RHEL 8.7 directly which perhaps would eliminate the cause:

How to upgrade using leapp from RHEL 7 to any RHEL 8 minor release. - Red Hat Customer Portal
https://access.redhat.com/solutions/6322411

LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_TARGET_RELEASE=8.7 leapp <preupgrade/upgrade>


Otherwise, the only option is to use the Backup\Restore method of migration to RHEL 8 as users who will upgrade from 6.10 to 6.11, they will directly land on 6.11.5 and then they are stuck there. 

For the sake of having a workaround, @Dhaval, can you test that UNSUPPORTED approach once, just to see if that works or not?

Hello,

I was able to upgrade the Satellite server from RHEL-7 to RHEL-8 using the steps suggested in https://access.redhat.com/solutions/6322411.

I executed the below commands: 
# LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_TARGET_RELEASE=8.7 leapp preupgrade
# LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_TARGET_RELEASE=8.7 leapp upgrade

The Satellite server is upgraded successfully.

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 (Ootpa)

# rpm -qa  | grep -i satellite
satellite-maintain-0.0.1-1.el8sat.noarch
ansible-collection-redhat-satellite-3.6.0-3.el8sat.noarch
satellite-common-6.11.5-1.el8sat.noarch
satellite-cli-6.11.5-1.el8sat.noarch
satellite-installer-6.11.0.7-1.el8sat.noarch
rubygem-foreman_theme_satellite-9.0.0.12-1.el8sat.noarch
satellite-6.11.5-1.el8sat.noarch
ansible-collection-redhat-satellite_operations-1.2.3-1.el8sat.noarch

NOTE: This is not an officially supported scenario. Please take a complete backup or snapshot of the Satellite server before trying the above steps.

Thanks and Regards,
Dhaval Joshi

Two clarifications are needed here perhaps,

A) Based on some discussion that happened yesterday, It seems that It would not matter what minor release of Satellite 6.11 that is, When Leapp tries to covert the system to RHEL 8, It will fetch the latest packages for 6.11 on RHEL 8, and hence the bug would still be triggered. 

   Assuming that is true, We would need to set the right expectations for the end user and update the Release Notes of Satellite 6.11 &\or the KB attached to this BZ.


B) In another discussion, It was pointed out that we can make use of a CV to restrict the content that leapp can see i.e. perhaps having an exclude filter added on this ERRATA RHSA-2023:1151. But If I am correct, that fix is only going to work for Capsules. The satellite would still be consuming content from Red Hat CDN directly and hence no scope for filtering ( unless someone has a way to hack yum\dnf\leapp in some way to exclude that ERRATA ).

   For Disconnected systems, there may still be an exception present but not sure about connected ones.

We are working on putting together a proper solution. To re-emphasize, as previously noted, the method using LEAP_UNSUPPORTED is unsupported.

Anyone attempting a Leapp based upgrade should be aware of these two issues as well:

 * https://bugzilla.redhat.com/show_bug.cgi?id=2142270
 * https://bugzilla.redhat.com/show_bug.cgi?id=2167984

Releases for these will be out soon. Given those, I would encourage folks to just hold on till those come out and we get this fix out rather than going an unsupported route.

While a proper fix is being developed, I tried a few approaches and this one seems to work great whether its a connected or disconnected satellite ( as the approach is disconnected only i.e. using Binary DVDs )

** For Satellite, if we can set up the content in this way, The upgrade will work flawlessly and we will end up having a Sat 6.11.4 on RHEL 8 
---------------------------------------------------------------------------------------------------------------------------------------------

** Download the binary ISOs from here:

--> https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.6/x86_64/product-software

--> https://access.redhat.com/downloads/content/250/ver=6.11/rhel---8/6.11.4.1/x86_64/product-software

** Copy them into the satellite 

** Start staging the offline content in satellite for leapp :

# mkdir /var/yumserver
# mount -o loop Satellite-6.11.4.1-rhel-8-x86_64.dvd.iso /mnt
# cp -ar /mnt/{Maintenance,Satellite} /var/www/html/pub/yumserver/
# umount /mnt


# mount -o loop rhel-8.6-x86_64-dvd.iso /mnt
# cp -ar /mnt/{AppStream,BaseOS} /var/www/html/pub/yumserver/
# umount /mnt

# du -sh /var/www/html/pub/yumserver/*
8.6G	/var/www/html/pub/yumserver/AppStream
1.3G	/var/www/html/pub/yumserver/BaseOS
416K	/var/www/html/pub/yumserver/Maintenance
560M	/var/www/html/pub/yumserver/Satellite


## Create this file ##

# cat /etc/yum.repos.d/rhel8.repo
[BaseOS]
name=rhel-8-for-x86_64-baseos-rpms
baseurl=http://localhost/pub/yumserver/BaseOS/
enabled=1

[AppStream]
name=rhel-8-for-x86_64-appstream-rpms
baseurl=http://localhost/pub/yumserver/AppStream/
enabled=1

# cat /etc/yum.repos.d/satellite.repo
[satellite-6.11-for-rhel-8-x86_64-rpms]
name=satellite-6.11-for-rhel-8-x86_64-rpms
baseurl=http://localhost/pub/yumserver/Satellite/
enabled=1

[satellite-maintenance-6.11-for-rhel-8-x86_64-rpms]
name=satellite-maintenance-6.11-for-rhel-8-x86_64-rpms
baseurl=http://localhost/pub/yumserver/Maintenance/
enabled=1


## Now run leapp:

# rmmod pata_acpi
# rmmod floppy
# leapp answer --section remove_pam_pkcs11_module_check.confirm=True
# leapp preupgrade --no-rhsm --enablerepo BaseOS --enablerepo AppStream --enablerepo satellite-6.11-for-rhel-8-x86_64-rpms --enablerepo satellite-maintenance-6.11-for-rhel-8-x86_64-rpms


# leapp preupgrade --no-rhsm --enablerepo BaseOS --enablerepo AppStream --enablerepo satellite-6.11-for-rhel-8-x86_64-rpms --enablerepo satellite-maintenance-6.11-for-rhel-8-x86_64-rpms


## If no issues proceed with:

# leapp upgrade --no-rhsm --enablerepo BaseOS --enablerepo AppStream --enablerepo satellite-6.11-for-rhel-8-x86_64-rpms --enablerepo satellite-maintenance-6.11-for-rhel-8-x86_64-rpms


## If no issues proceed with reboot:

# satellite-maintain service stop && reboot

Once the system comes back up, proceed from Step 9 of https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html-single/upgrading_and_updating_red_hat_satellite/index#upgrading-satellite-or-proxy-in-place-using-leapp_upgrade-guide 

## Once the system comes up, It will run the satellite-installer at the backend but will fail. For me, that was because, The system had /etc/pki/product/69.pem still present despite a successful conversion to RHEL 8.6 and hence it was able to see both rhel 7 and rhel 8 repos. 

## To fix that

# rm -f /etc/pki/product/69.pem

# subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms \
--enable=rhel-8-for-x86_64-appstream-rpms \
--enable=satellite-6.11-for-rhel-8-x86_64-rpms \
--enable=satellite-maintenance-6.11-for-rhel-8-x86_64-rpms

# subscription-manager repos --list-enabled

# mv /etc/yum.repos.d/satellite.repo /etc/yum.repos.d/satellite.repo_bkup

# mv /etc/yum.repos.d/rhel8.repo /etc/yum.repos.d/rhel8.repo_bkup

# satellite-installer 


And then continue with the rest of the steps from the upgrade guide ( step 10 and later ) once the installer is successfully executed. 



** For Capsule,

--> No need to follow this process.

--> Just have a CV prepared for all required repos for the leapp upgrade and then first create a filter to exclude the ERRATA RHSA-2023:1151 and then publish the CV version. 

--> Use that version of CV for the leapp of the capsule and it should technically only see Capsule 6.11.4 stuff now. 


I have tested the satellite-related steps but have not tested the capsule part ( although it should work just fine if our assumption is correct ).  I guess this approach is at least something that we can support officially.

# mkdir /var/yumserver 

should be

# mkdir /var/www/html/pub/yumserver/

Continuing from Comment 10 , I was able to test out the Capsule end approach as well i.e. 

* Add all required RHEL 7 and RHEL 8 repos for leapp`ing a capsule 6.11

* Add a Exclude ERRATA filter like this i.e. 

# hammer content-view filter info --id 1 --content-view-id 2
Filter ID:    1
Name:         Exclude RHSA-2023:1151
Type:         erratum
Inclusion:    false
Description:  
Repositories: 

Rules:        
 1) Id:        1
    Errata ID: RHSA-2023:1151
    Created:   2023/03/09 16:43:36
    Updated:   2023/03/09 16:43:36

* Publish the CV

* Associate that CV and LCE with the capsule

* Perform the leapp preupgrade and upgrade normally.


The only odd thing that happened was that, After the upgrade and reboot, Leapp brought up the capsule fine on RHEL 8 but the installer execution as a part of the same failed at the backend. I did nothing but just re-executed "satellite-installer --disable-system-checks" manually and it completed just fine.


NOTE: Whether it's the approach mentioned for satellite ( comment 10 ) or the approach mentioned for Capsule, Both will be upgraded to 6.11.4 only. So one might need to complete the post-upgrade steps of leapp, setup the content-views for capsule and run a 6.11.z upgrade for both satellite and capsules, to get to 6.11.5.

VERIFIED.

@Satellite 6.11.5.1 Snap1
candlepin-selinux-4.1.19-2.el8sat.noarch
candlepin-4.1.19-2.el8sat.noarch

by running automation

>>> leapp upgrade passes successfully

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.11.5.1 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1351