Bug 1348558

Summary:	Nova novnc console doesn't load 2/3 times: Failed to connect to server (code: 1006)
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo-heat-templates	Assignee:	Angus Thomas <athomas>
Status:	CLOSED ERRATA	QA Contact:	Gabriel Szasz <gszasz>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	10.0 (Newton)	CC:	dbecker, eglynn, jcoufal, jschluet, jslagle, mburns, mcornea, morazi, panbalag, rhel-osp-director-maint, sferdjao, srevivo, svanders
Target Milestone:	rc	Keywords:	Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1344507	Environment:
Last Closed:	2016-12-14 15:41:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1344507
Bug Blocks:

Description Marius Cornea 2016-06-21 12:45:15 UTC

+++ This bug was initially created as a clone of Bug #1344507 +++

Description of problem:
Nova novnc console fails to load 2/3 requests in an HA environment with 3 controllers. The issue seems pretty much the same as the one reported in BZ#1238336

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-2.0.1-0.20160602175532.bcd726f.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud 
2. Log in to Horizon
3. Access instance console

Actual results:
2/3 times you get the following message:
Failed to connect to server (code: 1006)

Expected results:
The noVNC console always gets loaded.

Additional info:
Haproxy config:
listen nova_novncproxy
  bind 10.0.0.11:6080 transparent
  bind 172.16.18.25:6080 transparent
  balance source
  timeout tunnel 1h
  server overcloud-controller-0 10.0.0.15:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 10.0.0.14:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 10.0.0.13:6080 check fall 5 inter 2000 rise 2


nova-novncproxy.log shows the following messages for a failed connection:

2016-06-09 20:03:18.049 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Plain non-SSL (ws://) WebSocket connection
2016-06-09 20:03:18.050 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Version hybi-13, base64: 'False'
2016-06-09 20:03:18.050 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Path: '/websockify'
2016-06-09 20:03:18.054 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.055 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.055 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.146 14778 INFO nova.console.websocketproxy [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] handler exception: The token 'ccadff42-91ef-4716-9882-eba5edad987b' is invalid or has expired

--- Additional comment from Marius Cornea on 2016-06-09 16:27:49 EDT ---

Workaround:

On controller nodes in /etc/nova/nova.conf add this cache section:

[cache]
backend = oslo_cache.memcache_pool
enabled = true
memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211

Comment 2 Sahid Ferdjaoui 2016-10-14 15:48:28 UTC

(In reply to Marius Cornea from comment #0)
> --- Additional comment from Marius Cornea on 2016-06-09 16:27:49 EDT ---
> 
> Workaround:
> 
> On controller nodes in /etc/nova/nova.conf add this cache section:
> 
> [cache]
> backend = oslo_cache.memcache_pool
> enabled = true
> memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211

Why do you consider that as workaround ? It's something which must have to be configured on an environment with more than one controller. Without that requirement the tokens are stored locally so depending on where the user is connecting itself the host might or not might have the token.

Comment 3 Marius Cornea 2016-10-14 15:56:24 UTC

(In reply to Sahid Ferdjaoui from comment #2)
> (In reply to Marius Cornea from comment #0)
> > --- Additional comment from Marius Cornea on 2016-06-09 16:27:49 EDT ---
> > 
> > Workaround:
> > 
> > On controller nodes in /etc/nova/nova.conf add this cache section:
> > 
> > [cache]
> > backend = oslo_cache.memcache_pool
> > enabled = true
> > memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211
> 
> Why do you consider that as workaround ? It's something which must have to
> be configured on an environment with more than one controller. Without that
> requirement the tokens are stored locally so depending on where the user is
> connecting itself the host might or not might have the token.

Because it's a temporary solution that addresses the issue, a proper fix is to have the installer set proper configuration.

Comment 4 Sven Anderson 2016-10-21 11:54:08 UTC

This should be fixed since https://review.openstack.org/#/c/328347/

Please verify with latest builds.

Comment 9 errata-xmlrpc 2016-12-14 15:41:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html