Bug 974846 - NoVNC Console fails to open.
NoVNC Console fails to open.
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: novnc (Show other bugs)
3.0
Unspecified Linux
urgent Severity urgent
: rc
: 3.0
Assigned To: Nikola Dipanov
Nir Magnezi
: Regression
: 975249 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-16 09:25 EDT by Nir Magnezi
Modified: 2016-04-22 01:02 EDT (History)
13 users (show)

See Also:
Fixed In Version: novnc-0.4-6.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, a VNC console would fail to open when one tried to open the console from the URL provided by the command: nova get-vnc-console <INSTANCE_ID> novnc. This has been fixed and the console now opens.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-27 13:17:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
monkey patch in separate process (2.40 KB, patch)
2013-06-20 06:45 EDT, Nikola Dipanov
pbrady: review+
Details | Diff

  None (edit)
Description Nir Magnezi 2013-06-16 09:25:25 EDT
Description of problem:
=======================
NoVNC Console fails to open.
I tried to restart openstack-nova-novncproxy, but it didn't help.

Version-Release number of selected component (if applicable):
=============================================================
pm -qa | grep nova
openstack-nova-api-2013.1.2-2.el6ost.noarch
openstack-nova-network-2013.1.2-2.el6ost.noarch
openstack-nova-conductor-2013.1.2-2.el6ost.noarch
python-novaclient-2.13.0-1.el6ost.noarch
openstack-nova-common-2013.1.2-2.el6ost.noarch
openstack-nova-compute-2013.1.2-2.el6ost.noarch
openstack-nova-cert-2013.1.2-2.el6ost.noarch
openstack-nova-novncproxy-0.4-4.el6ost.noarch
openstack-nova-scheduler-2013.1.2-2.el6ost.noarch
python-nova-2013.1.2-2.el6ost.noarch
openstack-nova-console-2013.1.2-2.el6ost.noarch

How reproducible:
=================
3/3

Steps to Reproduce:
===================
1. Install openstack via packstack[1]
2. start an instance
3. nova get-vnc-console <INSTANCE_ID> novnc
4. Open the vnc URL

[1] Ver: openstack-packstack-2013.1.1-0.17.dev631.el6ost.noarch

Actual results:
===============
The vnc session won't start.
It gets stuck on 'Starting VNC handshake'.

Expected results:
=================
VNC session should Open with no errors.

Additional info:
================
Nova logs (DEBUG & Verbose set to true):
2013-06-16 16:13:33.463 31333 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [], u'_msg_id': u'6e15bc6350bd482f96a7b4cb54d7b8b7', u'_context_quota_class': None, u'_context_user_name': None, u'_context_request_id': u'req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979', u'_context_service_catalog': [], u'_reply_q': u'reply_0fee9c4b4f994c8db37dc1eb5af644b1', u'_context_auth_token': '<SANITIZED>', u'args': {u'token': u'82abe089-3b59-48a6-ae21-c1bca4912242'}, u'_context_tenant': None, u'_context_instance_lock_checked': False, u'_context_project_name': None, u'_context_is_admin': True, u'_context_timestamp': u'2013-06-16T13:13:33.390238', u'_context_project_id': None, u'_context_user': None, u'_unique_id': u'f0d1beeabc194a9c9d5dd70999d6fed2', u'_context_read_deleted': u'no', u'_context_user_id': None, u'method': u'check_token', u'_context_remote_address': None} _safe_log /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py:276
2013-06-16 16:13:33.464 31333 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'read_deleted': u'no', 'project_name': None, 'user_id': None, 'roles': [], 'timestamp': u'2013-06-16T13:13:33.390238', 'auth_token': '<SANITIZED>', 'remote_address': None, 'quota_class': None, 'is_admin': True, 'user': None, 'service_catalog': [], 'request_id': u'req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979', 'instance_lock_checked': False, 'project_id': None, 'user_name': None, 'tenant': None} _safe_log /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py:276
2013-06-16 16:13:33.465 AUDIT nova.consoleauth.manager [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] Checking Token: 82abe089-3b59-48a6-ae21-c1bca4912242, True)
2013-06-16 16:13:33.465 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] Making synchronous call on conductor ... multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:583
2013-06-16 16:13:33.466 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] MSG_ID is b005a263100c42349978b0961bad9c08 multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:586
2013-06-16 16:13:33.466 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 5ed13596fcf24b13a5f4eec81d017f27. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337
2013-06-16 16:13:33.550 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] Making synchronous call on compute.fqdn ... multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:583
2013-06-16 16:13:33.550 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] MSG_ID is 49166d532ade4662ba1a15ed0fe590c0 multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:586
2013-06-16 16:13:33.550 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 8ea538661dbf499dac815f1dc4cbbef5. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337
2013-06-16 16:13:33.592 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 3d98e53ffdff4e6c9e2bb372a62c769c. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337
2013-06-16 16:13:33.599 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 9754f66d880a48b98779fdc99f499eb8. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337
Comment 3 Gary Kotton 2013-06-16 09:34:56 EDT
The issue is not related to selinux - when setting to permissive it still does not work.
Comment 5 Jaroslav Henner 2013-06-16 16:52:20 EDT
Zombies are attacking!:

    1 root       20   0 19228  1540  1264 S  0.0  0.1  0:00.79 /sbin/init
 5332 nova       20   0  292M 29564  5212 S  0.0  1.5  0:00.32 ├─ python /usr/bin/nova-novncproxy --web /usr/share/novnc/
 5345 nova       20   0  390M 30152  2916 S  0.0  1.6  0:00.07 │  ├─ python /usr/bin/nova-novncproxy --web /usr/share/novnc/
 5344 nova       20   0     0     0     0 Z  0.0  0.0  0:00.00 │  ├─ python
 5343 nova       20   0     0     0     0 Z  0.0  0.0  0:00.00 │  ├─ python
 5342 nova       20   0     0     0     0 Z  0.0  0.0  0:00.00 │  ├─ python
 5341 nova       20   0     0     0     0 Z  0.0  0.0  0:00.00 │  ├─ python
 5340 nova       20   0     0     0     0 Z  0.0  0.0  0:00.00 │  ├─ python
 5339 nova       20   0     0     0     0 Z  0.0  0.0  0:00.00 │  └─ python
Comment 8 Alan Pevec 2013-06-17 16:50:34 EDT
What's the conclusion here?
Patch "Fix novnc fail due to unpatched sockets for eventlet" has a change in nova-novncproxy:
+import eventlet; eventlet.monkey_patch()
and nova-dist.conf added in openstack-nova-novncproxy initscript.
Comment 10 Alan Pevec 2013-06-18 06:40:59 EDT
*** Bug 975249 has been marked as a duplicate of this bug. ***
Comment 11 Nikola Dipanov 2013-06-18 09:14:06 EDT
The issue here was twofold:

* nova-novncproxy script that we ship didn't do eventlet.monkey_patch() and this was causing it to block on the rpc call to authenticate the token (which uses eventlet). This only became apparent after we have added the fix for 971565.

The reason this was moves to the novnc component is that openstack-nova-novncproxy is currently a subpackage of novnce.

* In addition, we were not supplying the proper --config-file params in the openstack-nova-novncproxy.init script which would cause it to not pick up the options set in nova-dist.conf

In order to test this - just simply make sure that you can boot an instance and get a vnc console working in your browser (exactly as per the reporter's notes)
Comment 12 Jay Turner 2013-06-19 09:36:02 EDT
Moving back to Assigned.  There is additional work necessary here.
Comment 14 Bruce Reeler 2013-06-20 03:04:42 EDT
Nikola, could you also please check that the Doc Text I created for this bug is correct. I am not sure about "NoVNC Console" (comment 0) and just "VNC console" (comment 11). Are these the same thing? Thx.
Comment 15 Nikola Dipanov 2013-06-20 06:45:46 EDT
Created attachment 763403 [details]
monkey patch in separate process
Comment 16 Nikola Dipanov 2013-06-20 06:47:10 EDT
Description of the problem is in the provided patch.

Bruce - the text looks fine
Comment 17 Mark McLoughlin 2013-06-20 06:57:13 EDT
Patch looks pretty reasonable to me
Comment 18 Nikola Dipanov 2013-06-20 09:44:28 EDT
The reason this failed QA (which was not described in detail) is that after the last fix was applied novnc would work for a single server but would start to misbehave with more parallel connections. Solution seems to be to avoid monkey patching in the process that does proxying and do validation in a separate process where monkey-patching can happen.

To confirm that this is indeed not happening any more - please start at least two instances and try to connect to them using vnc as described in the report. Make sure that you can use the consoles and refresh the web page. For bonus points - make sure that after they are closed - there are no wayward processes left on the system.
Comment 19 Nir Magnezi 2013-06-23 10:07:20 EDT
Verified NVR: novnc-0.4-6.el6ost.noarch

Verification Steps:
===================
1. Repeated the reproduction steps in Comment #0 (installed with a Cloud Controller and 2 Compute nodes).
2. Tested via 'get-vnc-console'
3. Tested via horizon
4. Tested with 2 running instances

Result:
=======
The NoVNC Console is working fine now.
Comment 21 errata-xmlrpc 2013-06-27 13:17:52 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0968.html

Note You need to log in before you can comment on or make changes to this bug.