Bug 1477816 - Upgrade of director node to 7.4 with re-deploy or upgrade to overcloud causes horizon to error with 504 Gateway Timeout
Upgrade of director node to 7.4 with re-deploy or upgrade to overcloud causes...
Status: CLOSED DUPLICATE of bug 1478042
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-apache (Show other bugs)
11.0 (Ocata)
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: RHOS Maint
Depends On:
  Show dependency treegraph
Reported: 2017-08-03 00:02 EDT by Ken Savich
Modified: 2018-01-07 05:31 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-08-07 08:44:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1708655 None None None 2017-08-04 08:46 EDT

  None (edit)
Description Ken Savich 2017-08-03 00:02:03 EDT
Description of problem:

Upgrading your RHOSP 11 director node and/or overcloud to RHEL 7.4 or re-deploying RHOSP 11 after the director is upgraded to 7.4 causes Horizon to timeout with a 504 error.

Version-Release number of selected component (if applicable):
RHOSP 11, RHEL 7.4

How reproducible:
100% thus far

Steps to Reproduce:
1.Upgrade your director or overcloud to 7.4
2.Deploy (if you didn't do an upgrade of an existing overcloud)
3.Load Horizon on the public vip of the overcloud

Actual results:
504 Gateway Timeout error from Horizon (apache)

Expected results:
Horizon management UI should load and ask for login

Additional info:

Not sure of which component this is in as of yet.

Myself and two other team-mates have reproduced this in our labs, and high-value customer (Verizon) has also reproduced this as of this evening.

We'll be looking into root cause on Thursday, first thing.

Logs we have so far from httpd:

Ken Holden [4:22 PM] 
[Wed Aug 02 21:22:20.073090 2017] [:error] [pid 282369] WARNING:root:"dashboards" and "default_dashboard" in (local_)settings is DEPRECATED now and may be unsupported in some future release. The preferred way to specify the order of dashboards and the default dashboard is the pluggable dashboard mechanism (in /usr/share/openstack-dashboard/openstack_dashboard/enabled, /usr/share/openstack-dashboard/openstack_dashboard/local/enabled).

tcp dump from the controller for the request:

root@controller2 heat-admin]# tcpdump -i vlan5 tcp port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan5, link-type EN10MB (Ethernet), capture size 262144 bytes
21:18:38.490167 IP > overcloud.localdomain.http: Flags [.], ack 2577692198, win 229, options [nop,nop,TS val 1694569769 ecr 3827652], length 0
21:18:38.490195 IP overcloud.localdomain.http > Flags [.], ack 1, win 235, options [nop,nop,TS val 3837892 ecr 1694518644], length 0
Comment 3 Rhys Oxenham 2017-08-04 04:08:22 EDT
Similar report here, along with a workaround - this looks like a problem with Apache/WSGI rather than something that has happened during the upgrade.

Comment 4 Ken Savich 2017-08-04 08:08:56 EDT
Confirmed workaround here:

On each controller, edit /etc/httpd/conf.d/10-horizon_vhost.conf - add:

WSGIApplicationGroup %{GLOBAL}

Inside of  <VirtualHost></VirtualHost>

and restart httpd on each.

Agree with Rhys - may not have anything to do with 7.4.
Comment 5 Lukas Bezdicka 2017-08-07 08:44:06 EDT

*** This bug has been marked as a duplicate of bug 1478042 ***

Note You need to log in before you can comment on or make changes to this bug.