Bug 1348558 - Nova novnc console doesn't load 2/3 times: Failed to connect to server (code: 1006)
Summary: Nova novnc console doesn't load 2/3 times: Failed to connect to server (code:...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 10.0 (Newton)
Assignee: Angus Thomas
QA Contact: Gabriel Szasz
URL:
Whiteboard:
Depends On: 1344507
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-21 12:45 UTC by Marius Cornea
Modified: 2016-12-14 15:41 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1344507
Environment:
Last Closed: 2016-12-14 15:41:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 328347 0 None None None 2016-10-21 11:50:42 UTC
Red Hat Product Errata RHEA-2016:2948 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Marius Cornea 2016-06-21 12:45:15 UTC
+++ This bug was initially created as a clone of Bug #1344507 +++

Description of problem:
Nova novnc console fails to load 2/3 requests in an HA environment with 3 controllers. The issue seems pretty much the same as the one reported in BZ#1238336

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-2.0.1-0.20160602175532.bcd726f.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud 
2. Log in to Horizon
3. Access instance console

Actual results:
2/3 times you get the following message:
Failed to connect to server (code: 1006)

Expected results:
The noVNC console always gets loaded.

Additional info:
Haproxy config:
listen nova_novncproxy
  bind 10.0.0.11:6080 transparent
  bind 172.16.18.25:6080 transparent
  balance source
  timeout tunnel 1h
  server overcloud-controller-0 10.0.0.15:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 10.0.0.14:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 10.0.0.13:6080 check fall 5 inter 2000 rise 2


nova-novncproxy.log shows the following messages for a failed connection:

2016-06-09 20:03:18.049 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Plain non-SSL (ws://) WebSocket connection
2016-06-09 20:03:18.050 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Version hybi-13, base64: 'False'
2016-06-09 20:03:18.050 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Path: '/websockify'
2016-06-09 20:03:18.054 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.055 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.055 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.146 14778 INFO nova.console.websocketproxy [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] handler exception: The token 'ccadff42-91ef-4716-9882-eba5edad987b' is invalid or has expired

--- Additional comment from Marius Cornea on 2016-06-09 16:27:49 EDT ---

Workaround:

On controller nodes in /etc/nova/nova.conf add this cache section:

[cache]
backend = oslo_cache.memcache_pool
enabled = true
memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211

Comment 2 Sahid Ferdjaoui 2016-10-14 15:48:28 UTC
(In reply to Marius Cornea from comment #0)
> --- Additional comment from Marius Cornea on 2016-06-09 16:27:49 EDT ---
> 
> Workaround:
> 
> On controller nodes in /etc/nova/nova.conf add this cache section:
> 
> [cache]
> backend = oslo_cache.memcache_pool
> enabled = true
> memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211

Why do you consider that as workaround ? It's something which must have to be configured on an environment with more than one controller. Without that requirement the tokens are stored locally so depending on where the user is connecting itself the host might or not might have the token.

Comment 3 Marius Cornea 2016-10-14 15:56:24 UTC
(In reply to Sahid Ferdjaoui from comment #2)
> (In reply to Marius Cornea from comment #0)
> > --- Additional comment from Marius Cornea on 2016-06-09 16:27:49 EDT ---
> > 
> > Workaround:
> > 
> > On controller nodes in /etc/nova/nova.conf add this cache section:
> > 
> > [cache]
> > backend = oslo_cache.memcache_pool
> > enabled = true
> > memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211
> 
> Why do you consider that as workaround ? It's something which must have to
> be configured on an environment with more than one controller. Without that
> requirement the tokens are stored locally so depending on where the user is
> connecting itself the host might or not might have the token.

Because it's a temporary solution that addresses the issue, a proper fix is to have the installer set proper configuration.

Comment 4 Sven Anderson 2016-10-21 11:54:08 UTC
This should be fixed since https://review.openstack.org/#/c/328347/

Please verify with latest builds.

Comment 9 errata-xmlrpc 2016-12-14 15:41:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.