Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2038577

Summary: At the end of upgrade to Satellite 6.10, the httpd service fails to start due to missing mod_passenger.so or mod_wsgi.so libraries
Product: Red Hat Satellite Reporter: Sayan Das <saydas>
Component: DocumentationAssignee: Brian Angelica <bangelic>
Status: CLOSED CURRENTRELEASE QA Contact: SAT-DOCS <sat-bz-docs-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.10.1CC: alsouza, arsingh, aupadhye, bangelic, bhoefer, dhjoshi, egolov, mdolezel, pdudley
Target Milestone: UnspecifiedKeywords: Triaged, Upgrades
Target Release: Unused   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-08 08:53:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sayan Das 2022-01-08 15:15:18 UTC
Description of problem:

In certain scenarios, when a satellite is being upgraded from 6,6 -> 6.7 -> 6.8 -> 6.9 -> 6.10, In a few scenario's, The installer executed in prior upgrades were not able to get rid of existing apache configurations referring to mod_passenger.so or mod_wsgi.so and in such scenarios, at the end of the upgrade process to 6.10, httpd fails to get started due to having either of those libraries missing from the satellite.

Version-Release number of selected component (if applicable):

Satellite 6.10.1


How reproducible:

Under a specific scenario where older apache configurations somehow are present in the Satellite server which should not be present since Satellite 6.9

Steps to Reproduce:
1.
2.
3.

Actual results:

The first failure happens at the end of the satellite-installer execution during "satellite-maintain upgrade run --target-version=6.10" and httpd fails to start.

~~
Jan  4 21:34:18 lmpsatellite01 systemd: Stopped The Apache HTTP Server.
Jan  4 21:34:18 lmpsatellite01 systemd: Starting The Apache HTTP Server...
Jan  4 21:34:18 lmpsatellite01 httpd: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/zpassenger.load: Cannot load modules/mod_passenger.so into server: /etc/httpd/modules/mod_passenger.so: cannot open shared object file: No such file or directory
Jan  4 21:34:18 lmpsatellite01 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan  4 21:34:18 lmpsatellite01 systemd: Failed to start The Apache HTTP Server.
Jan  4 21:34:18 lmpsatellite01 systemd: Unit httpd.service entered failed state.
~~

Now, if we edit the file "/etc/httpd/conf.modules.d/zpassenger.load", comment out the entry in that file 
or, if we install back the mod_passenger package from https://access.redhat.com/downloads/content/mod_passenger/4.0.18-24.el7sat/x86_64/fd431d51/package and re-run the upgrade everything will go smooth.


Again, as soon as we will proceed for "satellite-maintain content remove-pulp2" , It will remove mod_wsgi package as a part of it but since the older apache config still asks to load it, at the end of the cleanup, if the services will be restarted, the httpd process will fail to get started again.

~~
Jan  4 22:08:21 lmpsatellite01 httpd: httpd: Syntax error on line 40 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/wsgi.load: Cannot load modules/mod_wsgi.so into server: /etc/httpd/modules/mod_wsgi.so: cannot open shared object file: No such file or directory
Jan  4 22:08:21 lmpsatellite01 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
~~

So either have the mod_wsgi package installed back in satellite from https://access.redhat.com/downloads/content/mod_wsgi/3.4-18.el7/x86_64/fd431d51/package or else just comment out all entries in "/etc/httpd/conf.modules.d/wsgi.load" and that will allow httpd to get started.


Additionally, The final cleanup needs to be done this way to get rid of both mod_passenger and mod_wsgi rpms and those old configuration files from Satellite 6.10

####################
* Stop all services

* Take a tar of /etc/httpd  and /usr/lib64/httpd/modules


*  Remove the rpms:

# rpm -ev --nodeps mod_wsgi mod_passenger


* If any of these files are present get rid of them.

Under '/etc/httpd/conf.modules.d':
10-wsgi.conf
passenger.conf
passenger_extra.conf
wsgi.conf
wsgi.load
zpassenger.load

Under '/etc/httpd/conf.d':
01-status.conf
03-crane.conf
05-foreman_proxy_content.conf
10-pulp.conf
pulp_iso.conf


* Start back httpd and see if it is working.

* If yes, start back all services.
####################



Expected results:

Satellite 6.10 upgrade process should be able to take care of these manual cleanup process 



Additional info:

NA

Comment 1 Evgeni Golov 2022-01-11 14:43:46 UTC
Can you make sure your custom-hiera.yaml doesn't contain "apache::purge_configs: false", as that's unsupported and what's triggering this issue.

Background: mod_passenger and mod_wsgi isn't used anymore starting with 6.10 and thus removed from the system. The installer would have also removed the configuration files if "apache::purge_configs: false" wouldn't be set.

Comment 2 Sayan Das 2022-01-11 15:03:20 UTC
I am confirming that all three affected users had "apache::purge_configs: false" set in the /etc/foreman-installer/custom-hiera.yaml file and your explanation makes perfect sense now.

So two questions from here:

A) Should foreman-maintain detect this hiera entry and WARN user about it during pre-upgrade check only ? If Yes, would a new BZ be needed ?


B) Considering that all three affected users had already in the middle of the upgrade when this issue happened and older passenger and wsgi related configs were still present, Would doing this clear those older configurations and complete the upgrade i.e.

  * Edit /etc/foreman-installer/custom-hiera.yaml and comment out "apache::purge_configs: false"

  * Re-run the upgrade command.


Or, I would still require to perform the manual cleanup as I have documented in https://access.redhat.com/solutions/6629031 .



Best Regards,

Sayan

Comment 3 Evgeni Golov 2022-01-11 15:55:12 UTC
A) sounds like a fair RFE, yes
B) running installer with purge_configs:true should clean things up, but might purge things the customer configured explicitly.

Comment 15 Red Hat Bugzilla 2023-09-18 04:29:57 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days