Bug 1272478 - 403 error upon upgrade to 6.1.3
Summary: 403 error upon upgrade to 6.1.3
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Packaging
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
urgent vote
Target Milestone: Unspecified
Assignee: Mike McCune
QA Contact: sthirugn@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-16 13:44 UTC by Mike McCune
Modified: 2019-08-15 05:41 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-19 16:45:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1993493 0 None None None Never
Red Hat Product Errata RHBA-2015:1914 0 normal SHIPPED_LIVE Satellite 6.1.3 bug fix update 2015-10-19 20:45:13 UTC

Description Mike McCune 2015-10-16 13:44:44 UTC
There is a packaging error in the ruby-rubygem-passenger and ruby193-rubygem-passenger packages that causes the process to be unable to load after a reboot causing 403 and other errors in Satellite 6.1.3.

Short term workaround:

mkdir -m0755 /var/run/rubygem-passenger
systemctl restart httpd

Comment 4 sthirugn@redhat.com 2015-10-16 17:35:53 UTC
Verified.

Scenario 1: Satellite upgrade - TEST COMPLETE
    Step 1 : CDN install [PASS] - 
    Step 2: Basic verification:
    CLI - [PASS] - hammer ping, hammer organization list
    UI - [PASS] - Able to access UI, create Org
    Step 3 : Reboot:
    * Error in /var/log/httpd/error_log:
     [Fri Oct 16 12:04:10.290734 2015] [passenger:error] [pid 3560] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog because it encountered the following error during startup: Cannot create server instance directory '/var/run/rubygem-passenger/passenger.1.0.3560': No such file or directory (errno=2)
    * UI is not accessible - Forbidden - You don't have permission to access / on this server.
    * CLI is accessible
    Step 4 : Apply fix - run katello-installer --upgrade - [PASS]
    Step 5: Basic verification:
    UI - [PASS] - Able to access UI, create Org
    CLI - [PASS] - hammer ping, hammer organization list

* Capsule: - TEST COMPLETE
    Step 1: Installation - PASS
    Step 2: Reboot - PASS
    Step 3: Error reproduced after reboot:
    [Fri Oct 16 12:45:36.963501 2015] [passenger:error] [pid 1303] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog because it encountered the following error during startup: Cannot create server instance directory '/var/run/rubygem-passenger/passenger.1.0.1303': No such file or directory (errno=2)
    Step 4: Apply fix - run capsule-installer --upgrade - [PASS]
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    (1/4): mod_passenger-4.0.18-21.el7sat.x86_64.rpm                     | 195 kB  00:00:00     
    (2/4): rubygem-passenger-4.0.18-21.el7sat.x86_64.rpm                 | 179 kB  00:00:00     
    (3/4): rubygem-passenger-native-4.0.18-21.el7sat.x86_64.rpm          | 708 kB  00:00:00     
    (4/4): rubygem-passenger-native-libs-4.0.18-21.el7sat.x86_64.rpm     | 712 kB  00:00:00
    Step 5: Reboot
    Step 6: PASS - no error in httpd error log.

Comment 5 Reartes Guillermo 2015-10-16 18:48:03 UTC
Hi, 

My experience today:

# katello-service stop
# yum upgrade

# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade Step: errata_import (this may take a while) ...
Upgrade Step: update_gpg_urls (this may take a while) ...
Upgrade Step: update_repository_metadata (this may take a while) ...
Katello upgrade completed!


Everithing seems ok, but after a reboot:

* Cannot see the Satellite 6.1.3 webui (forbidden)
* hammer shell does not work (server does not support such operation) (back translated from spanish)

hammer> organization list
[Foreman] Username: admin
[Foreman] Password for admin: 
El servidor no soporta esa operación.


Executed again, thinking this migth fix the issue:

* an answer file exist and it is specified in katello-installer config file, so i do not use any parameter.

# katello-installer 
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[XXXXXXXXX.XXXXX.XXX.XXX]: Could not evaluate: 404 Resource Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Installing             Done                                               [100%] [..................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/katello-installer/katello-installer.log

Found in httpd logs:
[Fri Oct 16 11:47:09.863124 2015] [passenger:error] [pid 4514] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog because it encountered the following error during startup: Cannot create server instance directory '/var/run/rubygem-passenger/passenger.1.0.4514': No such file or directory (errno=2)

Then found: BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1272478

Workaround:  (for every time the server is rebooted)

# mkdir -m0755 /var/run/rubygem-passenger
# systemctl restart httpd

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1272478

* All seems to be ok now, the workaround was usefull
* I did not tri another katello-installer --upgrade again

Comment 6 thunt 2015-10-16 21:23:27 UTC
Slightly more persistent workaround is to edit
/usr/lib/tmpfiles.d/rubygem-passenger.conf and
/opt/rh/ruby193/root/usr/lib/tmpfiles.d/rubygem-passenger.conf

and change contents to

d /var/run/rubygem-passenger 0755 root root -

See also http://projects.theforeman.org/issues/10001

Comment 8 errata-xmlrpc 2015-10-19 16:45:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1914


Note You need to log in before you can comment on or make changes to this bug.