Bug 2024269

Summary: Attempt of upgrading Satellite server to 6.7 or 6.8 stops with message "Please run 'foreman-maintain prep-6.10-upgrade' prior to upgrading." when using latest rubygem-foreman_maintain package
Product: Red Hat Satellite Reporter: Shekhar Raut <sraut>
Component: Satellite MaintainAssignee: Amit Upadhye <aupadhye>
Status: CLOSED ERRATA QA Contact: sganar
Severity: high Docs Contact:
Priority: high    
Version: 6.8.0CC: ahumbe, apatel, aupadhye, gtalreja, jsherril, kgaikwad, mkalyat, mmccune, pcreech, pmendezh, saydas, sganar
Target Milestone: 6.11.0Keywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2027353 (view as bug list) Environment:
Last Closed: 2022-07-05 14:30:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shekhar Raut 2021-11-17 17:58:41 UTC
Description of problem:

After installing latest package of rubygem-foreman_maintain.noarch and while performing upgrade on Satellite 6.6, 6.7, 6.8, 6.9; satellite-maintain tool is printing below message:

~~~
Please run 'foreman-maintain prep-6.10-upgrade' prior to upgrading.
~~~

Version-Release number of selected component (if applicable):

Red Hat Satellite 6.6, 6.7, 6.8, 6.9


How reproducible:

Steps to Reproduce:
1. Install latest version of rubygem-foreman_maintain.noarch
   
   rubygem-foreman_maintain.noarch                                          1:0.8.20-1.el7sat                                           rhel-7-server-satellite-maintenance-6-rpms

2. Run command to perform upgrade using "satellite-maintain" tool:

   For example:

   # satellite-maintain upgrade check --target-version 6.8

3. Command will print below message:

   Nothing to update, can't find new version of satellite-maintain.
   Please run 'foreman-maintain prep-6.10-upgrade' prior to upgrading.

4. Now downgrade rubygem-foreman_maintain.noarch using below commands:

   # yum history

   # yum history undo <Transaction ID>

   # rpm -qa |grep rubygem-foreman_maintain.noarch

5. Again run command to perform upgrade using "satellite-maintain" tool:

   # satellite-maintain upgrade check --target-version 6.8

6. Command will print below message:

   Checking for new version of satellite-maintain...
   
   rubygem-foreman_maintain.noarch                                                                         1:0.8.20-1.el7sat                                                                          rhel-7-server-satellite-maintenance-6-rpms

   Updating satellite-maintain package.

   The satellite-maintain package successfully updated.
   Re-run satellite-maintain with required options!

7. And receive the same output again.



Expected results:

Satellite-maintain command should proceed for the upgrade.


Additional info:

As a workaround we can perform below steps:

   # yum history

   # yum history undo <Transaction ID>

   # satellite-maintain upgrade check --target-version 6.8 --disable-self-upgrade

   And then command will proceed to perform satellite upgrade.

Comment 1 Sayan Das 2021-11-24 07:35:35 UTC
NOTE: 

The last working version is 0.7.14-1.el7sat . Any version of foreman-maintain beyond that i.e. 0.8.X is affected by this problem. So downgrading to rubygem-foreman_maintain-0.7.14-1.el7sat.noarch and then using --disable-self-upgrade is the current choice we have.

The solution article has been modified to provide the same information.

Comment 4 Amit Upadhye 2021-11-30 14:19:36 UTC
Created redmine issue https://projects.theforeman.org/issues/34039 from this bug

Comment 6 Bryan Kearney 2021-12-03 12:05:47 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34039 has been resolved.

Comment 7 Mike McCune 2021-12-07 16:59:00 UTC
WORKAROUND:

The check is likely failing on the ownership of /var/lib/pulp which may be apache:apache, you can check via:

# ls -ld /var/lib/pulp
drwxr-xr-x. 9 apache apache 181 Sep 11  2020 /var/lib/pulp

If this is the case, run:

# chown apache:pulp /var/lib/pulp

and re-try the upgrade

Comment 10 sganar 2022-01-17 06:55:15 UTC
@

Comment 11 sganar 2022-01-17 07:09:51 UTC
Will there be a fix for this https://bugzilla.redhat.com/show_bug.cgi?id=2024269#c9  ?

Comment 12 Sayan Das 2022-01-17 08:25:44 UTC
(In reply to sganar from comment #11)
> Will there be a fix for this
> https://bugzilla.redhat.com/show_bug.cgi?id=2024269#c9  ?

Hey Shubham,

The scenario in comment 9 is very unique and will only happen if the external LDAP source has some groups with the same GID as some internal groups of satellite i.e. pulp. But to be honest there is no way to deal with this apart from 

A) Either change the GID of those LDAP groups in External Source

B) Or, We change /etc/nsswitch.conf in satellite to exclude the External Auth-Source from it for User\Group lookup.


I had mentioned about it just to ensure that we are aware of different scenarios that can lead to this problem, but this should or perhaps cannot be included in QE testing\verification. 

So I guess, The only proper way to test it would be to leave the ownership of /var/lib/pulp to apache:apache and then see, if we try to upgrade the Satellite 6 to any version below 6.10, whether it still prompts for that "Please run 'foreman-maintain prep-6.10-upgrade' prior to upgrading" message or not.

It should only be prompted during Sat 6.9 -> 6.10 upgrade and if the "foreman-maintain prep-6.10-upgrade" command was not executed for any reason causing the actual group_ownership conflict.

Comment 13 sganar 2022-01-18 11:01:13 UTC
Verified.

Tested on Satellite 6.9.8 Snap 1.0
rubygem-foreman_maintain-0.8.22

Steps followed: 
1. Setup the repositories required for Satellite 6.10 upgrade
2. Run "satellite-maintain upgrade check --target-version" 6.10 / "satellite-maintain upgrade run --target-version 6.10"

Observation:

satellite-maintain command proceeds for the upgrade as expected.
A check is added to check the pulp content's group ownership for 6.9 to 6.10 upgrade.

Comment 20 errata-xmlrpc 2022-07-05 14:30:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498