Bug 1344507

Summary:	Nova novnc console doesn't load 2/3 times: Failed to connect to server (code: 1006)
Product:	[Community] RDO	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo	Assignee:	James Slagle <jslagle>
Status:	CLOSED ERRATA	QA Contact:	Shai Revivo <srevivo>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	trunk	CC:	chris.brown, mcornea, svanders
Target Milestone:	---
Target Release:	trunk
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.0.1-0.20161021142101.fa5c89c.el7.centos	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1348558 (view as bug list)		Environment:
Last Closed:	2017-06-22 15:24:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1348558

Description Marius Cornea 2016-06-09 20:04:15 UTC

Description of problem:
Nova novnc console fails to load 2/3 requests in an HA environment with 3 controllers. The issue seems pretty much the same as the one reported in BZ#1238336

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-2.0.1-0.20160602175532.bcd726f.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud 
2. Log in to Horizon
3. Access instance console

Actual results:
2/3 times you get the following message:
Failed to connect to server (code: 1006)

Expected results:
The noVNC console always gets loaded.

Additional info:
Haproxy config:
listen nova_novncproxy
  bind 10.0.0.11:6080 transparent
  bind 172.16.18.25:6080 transparent
  balance source
  timeout tunnel 1h
  server overcloud-controller-0 10.0.0.15:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 10.0.0.14:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 10.0.0.13:6080 check fall 5 inter 2000 rise 2


nova-novncproxy.log shows the following messages for a failed connection:

2016-06-09 20:03:18.049 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Plain non-SSL (ws://) WebSocket connection
2016-06-09 20:03:18.050 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Version hybi-13, base64: 'False'
2016-06-09 20:03:18.050 14778 INFO nova.console.websocketproxy [-] 10.0.0.15 - - [09/Jun/2016 20:03:18] 10.0.0.15: Path: '/websockify'
2016-06-09 20:03:18.054 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.055 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.055 14778 WARNING oslo_config.cfg [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2016-06-09 20:03:18.146 14778 INFO nova.console.websocketproxy [req-626d918a-c898-470e-ac45-4388e935882d - - - - -] handler exception: The token 'ccadff42-91ef-4716-9882-eba5edad987b' is invalid or has expired

Comment 1 Marius Cornea 2016-06-09 20:27:49 UTC

Workaround:

On controller nodes in /etc/nova/nova.conf add this cache section:

[cache]
backend = oslo_cache.memcache_pool
enabled = true
memcache_servers = 10.0.0.15:11211,10.0.0.14:11211,10.0.0.13:11211

Comment 2 Sven Anderson 2016-10-21 11:55:32 UTC

This should be fixed since https://review.openstack.org/#/c/328347/

Please verify with latest builds.

Comment 3 Christopher Brown 2017-06-22 15:24:50 UTC

Fixed, closing.