Bug 1477816 - Upgrade of director node to 7.4 with re-deploy or upgrade to overcloud causes horizon to error with 504 Gateway Timeout
Summary: Upgrade of director node to 7.4 with re-deploy or upgrade to overcloud causes...
Keywords:
Status: CLOSED DUPLICATE of bug 1478042
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-apache
Version: 11.0 (Ocata)
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-03 04:02 UTC by Ken Savich
Modified: 2020-12-14 09:23 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-07 12:44:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1708655 0 None None None 2017-08-04 12:46:36 UTC

Description Ken Savich 2017-08-03 04:02:03 UTC
Description of problem:

Upgrading your RHOSP 11 director node and/or overcloud to RHEL 7.4 or re-deploying RHOSP 11 after the director is upgraded to 7.4 causes Horizon to timeout with a 504 error.

Version-Release number of selected component (if applicable):
RHOSP 11, RHEL 7.4

How reproducible:
100% thus far

Steps to Reproduce:
1.Upgrade your director or overcloud to 7.4
2.Deploy (if you didn't do an upgrade of an existing overcloud)
3.Load Horizon on the public vip of the overcloud

Actual results:
504 Gateway Timeout error from Horizon (apache)

Expected results:
Horizon management UI should load and ask for login

Additional info:

Not sure of which component this is in as of yet.

Myself and two other team-mates have reproduced this in our labs, and high-value customer (Verizon) has also reproduced this as of this evening.

We'll be looking into root cause on Thursday, first thing.



Logs we have so far from httpd:

Ken Holden [4:22 PM] 
[Wed Aug 02 21:22:20.073090 2017] [:error] [pid 282369] WARNING:root:"dashboards" and "default_dashboard" in (local_)settings is DEPRECATED now and may be unsupported in some future release. The preferred way to specify the order of dashboards and the default dashboard is the pluggable dashboard mechanism (in /usr/share/openstack-dashboard/openstack_dashboard/enabled, /usr/share/openstack-dashboard/openstack_dashboard/local/enabled).

tcp dump from the controller for the request:

root@controller2 heat-admin]# tcpdump -i vlan5 tcp port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan5, link-type EN10MB (Ethernet), capture size 262144 bytes
21:18:38.490167 IP 192.168.1.28.58382 > overcloud.localdomain.http: Flags [.], ack 2577692198, win 229, options [nop,nop,TS val 1694569769 ecr 3827652], length 0
21:18:38.490195 IP overcloud.localdomain.http > 192.168.1.28.58382: Flags [.], ack 1, win 235, options [nop,nop,TS val 3837892 ecr 1694518644], length 0

Comment 3 Rhys Oxenham 2017-08-04 08:08:22 UTC
Similar report here, along with a workaround - this looks like a problem with Apache/WSGI rather than something that has happened during the upgrade.

https://bugzilla.redhat.com/show_bug.cgi?id=1478042

Comment 4 Ken Savich 2017-08-04 12:08:56 UTC
Confirmed workaround here:

On each controller, edit /etc/httpd/conf.d/10-horizon_vhost.conf - add:

WSGIApplicationGroup %{GLOBAL}

Inside of  <VirtualHost></VirtualHost>

and restart httpd on each.

Agree with Rhys - may not have anything to do with 7.4.

Comment 5 Lukas Bezdicka 2017-08-07 12:44:06 UTC

*** This bug has been marked as a duplicate of bug 1478042 ***


Note You need to log in before you can comment on or make changes to this bug.