Description of problem: ======================= NoVNC Console fails to open. I tried to restart openstack-nova-novncproxy, but it didn't help. Version-Release number of selected component (if applicable): ============================================================= pm -qa | grep nova openstack-nova-api-2013.1.2-2.el6ost.noarch openstack-nova-network-2013.1.2-2.el6ost.noarch openstack-nova-conductor-2013.1.2-2.el6ost.noarch python-novaclient-2.13.0-1.el6ost.noarch openstack-nova-common-2013.1.2-2.el6ost.noarch openstack-nova-compute-2013.1.2-2.el6ost.noarch openstack-nova-cert-2013.1.2-2.el6ost.noarch openstack-nova-novncproxy-0.4-4.el6ost.noarch openstack-nova-scheduler-2013.1.2-2.el6ost.noarch python-nova-2013.1.2-2.el6ost.noarch openstack-nova-console-2013.1.2-2.el6ost.noarch How reproducible: ================= 3/3 Steps to Reproduce: =================== 1. Install openstack via packstack[1] 2. start an instance 3. nova get-vnc-console <INSTANCE_ID> novnc 4. Open the vnc URL [1] Ver: openstack-packstack-2013.1.1-0.17.dev631.el6ost.noarch Actual results: =============== The vnc session won't start. It gets stuck on 'Starting VNC handshake'. Expected results: ================= VNC session should Open with no errors. Additional info: ================ Nova logs (DEBUG & Verbose set to true): 2013-06-16 16:13:33.463 31333 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [], u'_msg_id': u'6e15bc6350bd482f96a7b4cb54d7b8b7', u'_context_quota_class': None, u'_context_user_name': None, u'_context_request_id': u'req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979', u'_context_service_catalog': [], u'_reply_q': u'reply_0fee9c4b4f994c8db37dc1eb5af644b1', u'_context_auth_token': '<SANITIZED>', u'args': {u'token': u'82abe089-3b59-48a6-ae21-c1bca4912242'}, u'_context_tenant': None, u'_context_instance_lock_checked': False, u'_context_project_name': None, u'_context_is_admin': True, u'_context_timestamp': u'2013-06-16T13:13:33.390238', u'_context_project_id': None, u'_context_user': None, u'_unique_id': u'f0d1beeabc194a9c9d5dd70999d6fed2', u'_context_read_deleted': u'no', u'_context_user_id': None, u'method': u'check_token', u'_context_remote_address': None} _safe_log /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py:276 2013-06-16 16:13:33.464 31333 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'read_deleted': u'no', 'project_name': None, 'user_id': None, 'roles': [], 'timestamp': u'2013-06-16T13:13:33.390238', 'auth_token': '<SANITIZED>', 'remote_address': None, 'quota_class': None, 'is_admin': True, 'user': None, 'service_catalog': [], 'request_id': u'req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979', 'instance_lock_checked': False, 'project_id': None, 'user_name': None, 'tenant': None} _safe_log /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py:276 2013-06-16 16:13:33.465 AUDIT nova.consoleauth.manager [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] Checking Token: 82abe089-3b59-48a6-ae21-c1bca4912242, True) 2013-06-16 16:13:33.465 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] Making synchronous call on conductor ... multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:583 2013-06-16 16:13:33.466 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] MSG_ID is b005a263100c42349978b0961bad9c08 multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:586 2013-06-16 16:13:33.466 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 5ed13596fcf24b13a5f4eec81d017f27. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337 2013-06-16 16:13:33.550 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] Making synchronous call on compute.fqdn ... multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:583 2013-06-16 16:13:33.550 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] MSG_ID is 49166d532ade4662ba1a15ed0fe590c0 multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:586 2013-06-16 16:13:33.550 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 8ea538661dbf499dac815f1dc4cbbef5. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337 2013-06-16 16:13:33.592 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 3d98e53ffdff4e6c9e2bb372a62c769c. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337 2013-06-16 16:13:33.599 DEBUG nova.openstack.common.rpc.amqp [req-0d3c6c17-aa5d-47a9-bd0e-dbfda3024979 None None] UNIQUE_ID is 9754f66d880a48b98779fdc99f499eb8. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:337
The issue is not related to selinux - when setting to permissive it still does not work.
Zombies are attacking!: 1 root 20 0 19228 1540 1264 S 0.0 0.1 0:00.79 /sbin/init 5332 nova 20 0 292M 29564 5212 S 0.0 1.5 0:00.32 ├─ python /usr/bin/nova-novncproxy --web /usr/share/novnc/ 5345 nova 20 0 390M 30152 2916 S 0.0 1.6 0:00.07 │ ├─ python /usr/bin/nova-novncproxy --web /usr/share/novnc/ 5344 nova 20 0 0 0 0 Z 0.0 0.0 0:00.00 │ ├─ python 5343 nova 20 0 0 0 0 Z 0.0 0.0 0:00.00 │ ├─ python 5342 nova 20 0 0 0 0 Z 0.0 0.0 0:00.00 │ ├─ python 5341 nova 20 0 0 0 0 Z 0.0 0.0 0:00.00 │ ├─ python 5340 nova 20 0 0 0 0 Z 0.0 0.0 0:00.00 │ ├─ python 5339 nova 20 0 0 0 0 Z 0.0 0.0 0:00.00 │ └─ python
What's the conclusion here? Patch "Fix novnc fail due to unpatched sockets for eventlet" has a change in nova-novncproxy: +import eventlet; eventlet.monkey_patch() and nova-dist.conf added in openstack-nova-novncproxy initscript.
*** Bug 975249 has been marked as a duplicate of this bug. ***
The issue here was twofold: * nova-novncproxy script that we ship didn't do eventlet.monkey_patch() and this was causing it to block on the rpc call to authenticate the token (which uses eventlet). This only became apparent after we have added the fix for 971565. The reason this was moves to the novnc component is that openstack-nova-novncproxy is currently a subpackage of novnce. * In addition, we were not supplying the proper --config-file params in the openstack-nova-novncproxy.init script which would cause it to not pick up the options set in nova-dist.conf In order to test this - just simply make sure that you can boot an instance and get a vnc console working in your browser (exactly as per the reporter's notes)
Moving back to Assigned. There is additional work necessary here.
Nikola, could you also please check that the Doc Text I created for this bug is correct. I am not sure about "NoVNC Console" (comment 0) and just "VNC console" (comment 11). Are these the same thing? Thx.
Created attachment 763403 [details] monkey patch in separate process
Description of the problem is in the provided patch. Bruce - the text looks fine
Patch looks pretty reasonable to me
The reason this failed QA (which was not described in detail) is that after the last fix was applied novnc would work for a single server but would start to misbehave with more parallel connections. Solution seems to be to avoid monkey patching in the process that does proxying and do validation in a separate process where monkey-patching can happen. To confirm that this is indeed not happening any more - please start at least two instances and try to connect to them using vnc as described in the report. Make sure that you can use the consoles and refresh the web page. For bonus points - make sure that after they are closed - there are no wayward processes left on the system.
Verified NVR: novnc-0.4-6.el6ost.noarch Verification Steps: =================== 1. Repeated the reproduction steps in Comment #0 (installed with a Cloud Controller and 2 Compute nodes). 2. Tested via 'get-vnc-console' 3. Tested via horizon 4. Tested with 2 running instances Result: ======= The NoVNC Console is working fine now.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0968.html