Bug 1476946

Summary: instance vnc console not working in latest RHOSP 10 patch
Product: Red Hat OpenStack Reporter: bigswitch <rhosp-bugs-internal>
Component: openstack-novaAssignee: Ollie Walsh <owalsh>
Status: CLOSED WORKSFORME QA Contact: Joe H. Rahme <jhakimra>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: berrange, dasmith, eglynn, kchamart, owalsh, rhosp-bugs-internal, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-21 17:27:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
templates none

Description bigswitch 2017-07-31 21:40:23 UTC 
Description of problem:
Default RHOSP 10 deployment , VNC console for instance is not working.. 

RHOSP Patch 06-15-2017

Version-Release number of selected component (if applicable):
RHOSP 10

How reproducible:
Deploy RHOSP 10 
create instance 
login to instance using console

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

COmpute node nova.conf

[DEFAULT]
force_raw_images=True
use_neutron=True
log_dir=/var/log/nova
notify_api_faults=False
state_path=/var/lib/nova
report_interval=10
image_service=nova.image.glance.GlanceImageService
heal_instance_info_cache_interval=60
notify_on_state_change=vm_and_task_state
firewall_driver=nova.virt.firewall.NoopFirewallDriver
instance_usage_audit=True
use_ipv6=False
reserved_host_memory_mb=2048
vif_plugging_is_fatal=True
service_down_time=60
host=overcloud-compute-1.localdomain
instance_usage_audit_period=hour
auth_strategy=keystone
compute_driver=libvirt.LibvirtDriver
rootwrap_config=/etc/nova/rootwrap.conf
rpc_backend=rabbit
vif_plugging_timeout=300
dhcp_domain=novalocal
allow_resize_to_same_host=False
[api_database]
connection=mysql+pymysql://nova_api:j2WVcwr9fJ4xAcEcMPTdZtmD6.0.17/nova_api
[oslo_messaging_rabbit]
rabbit_userid=guest
rabbit_password=3PnGyaaewds46ts9bW9CT6GfG
heartbeat_timeout_threshold=60
rabbit_use_ssl=False
rabbit_hosts=172.17.0.13:5672
[libvirt]
inject_partition=-2
inject_password=False
live_migration_uri=qemu+ssh://nova_migration@%s/system?keyfile=/etc/nova/migration/identity
cpu_mode=host-model
virt_type=kvm
inject_key=False
[oslo_messaging_notifications]
driver=messagingv2
[glance]
api_servers=http://172.18.0.18:9292
[cinder]
catalog_info=volumev2:cinderv2:internalURL
[vnc]
vncserver_proxyclient_address=172.17.0.20
vncserver_listen=127.0.0.1
keymap=en-us
enabled=True
novncproxy_base_url=http://10.8.65.207:6080/vnc_auto.html

api3005: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.20  netmask 255.255.255.0  broadcast 172.17.0.255
        inet6 fe80::acce:67ff:fed8:6bf4  prefixlen 64  scopeid 0x20<link>
        ether ae:ce:67:d8:6b:f4  txqueuelen 1000  (Ethernet)
        RX packets 148874  bytes 16069197 (15.3 MiB)
        RX errors 0  dropped 3358  overruns 0  frame 0
        TX packets 74778  bytes 21020424 (20.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@overcloud-compute-1 log]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.0.2.1       0.0.0.0         UG    0      0        0 eno3
169.254.169.254 192.0.2.1       255.255.255.255 UGH   0      0        0 eno3
172.17.0.0      0.0.0.0         255.255.255.0   U     0      0        0 api3005
172.18.0.0      0.0.0.0         255.255.255.0   U     0      0        0 storage3006
192.0.2.0       0.0.0.0         255.255.255.0   U     0      0        0 eno3

[root@overcloud-compute-1 log]# ping 10.8.65.207
PING 10.8.65.207 (10.8.65.207) 56(84) bytes of data.
64 bytes from 10.8.65.207: icmp_seq=1 ttl=62 time=1.32 ms

Controller Config:

/etc/nova/nova.conf
[vnc]
novncproxy_port=6080
novncproxy_host=172.17.0.13
novncproxy_base_url=http://10.8.65.207:6080/vnc_auto.html
Comment 1 bigswitch 2017-07-31 21:41:05 UTC 
-rw-r--r--. 1 root root         23 Jun 15 12:30 version-10.0-20170615.1.el7ost.txt
lrwxrwxrwx. 1 root root         67 Jul  6 23:30 version.txt -> /usr/share/rhosp-director-images/version-10.0-20170615.1.el7ost.txt
lrwxrwxrwx. 1 root root         74 Jul  6 23:30 overcloud-full-latest-10.0.tar -> /usr/share/rhosp-director-images/overcloud-full-10.0-20170615.1.el7ost.tar
lrwxrwxrwx. 1 root root         74 Jul  6 23:30 overcloud-full.tar -> /usr/share/rhosp-director-images/overcloud-full-10.0-20170615.1.el7ost.tar
lrwxrwxrwx. 1 root root         79 Jul  6 23:30 ironic-python-agent-latest-10.0.tar -> /usr/share/rhosp-director-images/ironic-python-agent-10.0-20170615.1.el7ost.tar
lrwxrwxrwx. 1 root root         79 Jul  6 23:30 ironic-python-agent.tar -> /usr/share/rhosp-director-images/ironic-python-agent-10.0-20170615.1.el7ost.tar
Comment 2 bigswitch 2017-07-31 21:41:49 UTC 
-rw-r--r--. 1 root root         23 Jun 15 12:30 version-10.0-20170615.1.el7ost.txt
lrwxrwxrwx. 1 root root         67 Jul  6 23:30 version.txt -> /usr/share/rhosp-director-images/version-10.0-20170615.1.el7ost.txt
lrwxrwxrwx. 1 root root         74 Jul  6 23:30 overcloud-full-latest-10.0.tar -> /usr/share/rhosp-director-images/overcloud-full-10.0-20170615.1.el7ost.tar
lrwxrwxrwx. 1 root root         74 Jul  6 23:30 overcloud-full.tar -> /usr/share/rhosp-director-images/overcloud-full-10.0-20170615.1.el7ost.tar
lrwxrwxrwx. 1 root root         79 Jul  6 23:30 ironic-python-agent-latest-10.0.tar -> /usr/share/rhosp-director-images/ironic-python-agent-10.0-20170615.1.el7ost.tar
lrwxrwxrwx. 1 root root         79 Jul  6 23:30 ironic-python-agent.tar -> /usr/share/rhosp-director-images/ironic-python-agent-10.0-20170615.1.el7ost.tar
Comment 3 bigswitch 2017-07-31 21:42:16 UTC 
Previous RHOSP 10 dated :version-10.0-20170201.1.el7ost.txt working fine
Comment 4 bigswitch 2017-08-02 17:22:03 UTC 
There is also following info in controller novncconsole (handler exception)

2017-08-01 18:12:37.360 927120 INFO nova.console.websocketproxy [req-8c4a203d-6851-4121-9795-564b54213631 - - - - -]  41: connect info: {u'instance_uuid': u'c9c93c77-cdde-4ea7-ba45-d4592223530f', u'internal_access_path': None, u'last_activity_at': 1501611157.125692, u'console_type': u'novnc', u'host': u'172.17.0.20', u'token': u'5463ee11-9399-460b-9536-675e0960d871', u'access_url': u'http://10.8.65.207:6080/vnc_auto.html?token=5463ee11-9399-460b-9536-675e0960d871', u'port': u'5900'}
2017-08-01 18:12:37.360 927120 INFO nova.console.websocketproxy [req-8c4a203d-6851-4121-9795-564b54213631 - - - - -]  41: connecting to: 172.17.0.20:5900
2017-08-01 18:12:37.366 927120 INFO nova.console.websocketproxy [req-8c4a203d-6851-4121-9795-564b54213631 - - - - -] handler exception: [Errno 111] ECONNREFUSED
2017-08-01 18:12:43.393 926743 INFO nova.console.websocketproxy [-] 172.17.0.13: ignoring empty handshake
2017-08-01 18:12:43.393 926742 INFO nova.console.websocketproxy [-] 172.17.0.13: ignoring empty handshake
2017-08-01 18:12:43.394 926739 INFO nova.console.websocketproxy [-] 172.17.0.13: ignoring empty handshake
2017-08-01 18:12:43.393 926740 INFO nova.console.websocketproxy [-] 172.17.0.13: ignoring empty handshake
2017-08-01 18:12:50.575 927119 INFO nova.console.websocketproxy [-] 172.17.0.13: ignoring empty handshake
Comment 5 Artom Lifshitz 2017-08-09 15:54:08 UTC 
> 2017-08-01 18:12:37.366 927120 INFO nova.console.websocketproxy
> [req-8c4a203d-6851-4121-9795-564b54213631 - - - - -] handler exception:
> [Errno 111] ECONNREFUSED

This is indicating that websocketproxy was unable to connect to the VM's VNC console on the compute host. In order to better understand what's going on, would it be possible to reproduce this with debug enabled on the controller and compute nodes, and attach controller and compute logs to this bz? Full sosreports would be even better, as they include all the configuration files as well.
Comment 6 bigswitch 2017-08-09 15:56:26 UTC 
Is debug enabled on novaconsole? . is there any place debug needs to be enabled?
Comment 7 Artom Lifshitz 2017-08-11 00:39:34 UTC 
> Is debug enabled on novaconsole? . is there any place debug needs to be
> enabled?

I'm fairly certain that just setting debug=True in nova.conf on the compute will suffice. If not it's possible to pass the -d flag to nova-novncproxy to get it to start in debug mode.
Comment 8 Artom Lifshitz 2017-08-11 11:57:25 UTC 
> I'm fairly certain that just setting debug=True in nova.conf on the compute
> will suffice. If not it's possible to pass the -d flag to nova-novncproxy to
> get it to start in debug mode.

Sorry, got confused. nova-nonvcproxy is normally running on the controller, not the compute.
Comment 9 Artom Lifshitz 2017-08-18 14:24:41 UTC 
Hello,

I'm wondering if you've managed to reproduce this with debug enabled? Or perhaps my suggestions that websocketproxy was unable to connect to the VM's VNC console on the compute host helped you figure out what was going on? In either case, please let us know in the bz - either by attaching logs so we can move forward with troubleshooting, or so we can close this bz altogether.

Cheers!
Comment 10 bigswitch 2017-08-18 15:28:39 UTC 
NOw we are hitting the following 1482186 in latest RHOSP 10 , due to this we cannot get instance running at all.
Comment 11 bigswitch 2017-08-21 21:16:40 UTC 
We are seeing this in nova-novncproxy.log when attempting to access its console:


2017-08-21 14:13:38.091 729608 INFO nova.console.websocketproxy [-] 172.17.0.17 - - [21/Aug/2017 14:13:38] 172.17.0.17: Plain non-SSL (ws://) WebSocket connection
2017-08-21 14:13:38.092 729608 INFO nova.console.websocketproxy [-] 172.17.0.17 - - [21/Aug/2017 14:13:38] 172.17.0.17: Version hybi-13, base64: 'False'
2017-08-21 14:13:38.092 729608 INFO nova.console.websocketproxy [-] 172.17.0.17 - - [21/Aug/2017 14:13:38] 172.17.0.17: Path: '/websockify?token=2a3251c3-d2ee-421c-83e7-687105fd6879'
2017-08-21 14:13:38.096 729608 WARNING oslo_config.cfg [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2017-08-21 14:13:38.096 729608 WARNING oslo_config.cfg [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2017-08-21 14:13:38.097 729608 WARNING oslo_config.cfg [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2017-08-21 14:13:38.156 729608 DEBUG oslo_messaging._drivers.amqpdriver [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -] CALL msg_id: 7a5145cafa6e45698b215a82cf81c4d9 exchange 'nova' topic 'consoleauth' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:495
2017-08-21 14:13:38.218 729608 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 7a5145cafa6e45698b215a82cf81c4d9 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:343
2017-08-21 14:13:38.219 729608 INFO nova.console.websocketproxy [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -]  46: connect info: {u'instance_uuid': u'faacc00b-8384-4efb-8eb0-a2f947082149', u'internal_access_path': None, u'last_activity_at': 1503350017.925391, u'console_type': u'novnc', u'host': u'172.17.0.22', u'token': u'2a3251c3-d2ee-421c-83e7-687105fd6879', u'access_url': u'http://10.9.28.26:6080/vnc_auto.html?token=2a3251c3-d2ee-421c-83e7-687105fd6879', u'port': u'5900'}
2017-08-21 14:13:38.219 729608 INFO nova.console.websocketproxy [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -]  46: connecting to: 172.17.0.22:5900
2017-08-21 14:13:38.226 729608 INFO nova.console.websocketproxy [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -] handler exception: [Errno 111] ECONNREFUSED
2017-08-21 14:13:38.226 729608 DEBUG nova.console.websocketproxy [req-f6ed2381-da8e-43cb-8e3a-2dfa3cc09ea3 - - - - -] exception vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:875
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy Traceback (most recent call last):
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 930, in top_new_client
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     client = self.do_handshake(startsock, address)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 860, in do_handshake
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     self.RequestHandlerClass(retsock, address, self)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 164, in __init__
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 114, in __init__
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     SimpleHTTPRequestHandler.__init__(self, req, addr, server)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     self.handle()
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 581, in handle
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     SimpleHTTPRequestHandler.handle(self)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     self.handle_one_request()
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     method()
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 543, in do_GET
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     if not self.handle_websocket():
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 531, in handle_websocket
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     self.new_websocket_client()
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 135, in new_websocket_client
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     tsock = self.socket(host, port, connect=True)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 167, in socket
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     return websockify.WebSocketServer.socket(*args, **kwargs)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 736, in socket
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     sock.connect(addrs[0][4])
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 242, in connect
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     socket_checkerr(fd)
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy     raise socket.error(err, errno.errorcode[err])
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy error: [Errno 111] ECONNREFUSED
2017-08-21 14:13:38.226 729608 ERROR nova.console.websocketproxy
Comment 12 Artom Lifshitz 2017-08-25 14:06:13 UTC 
This is telling us a couple of things:

1. novncproxy is attempting to connect to 172.17.0.22 for faacc00b-8384-4efb-8eb0-a2f947082149's console. Is 172.17.0.22 the correct IP address for the host running the instance?

2. I don't think nova-compute logs from that host would help us, but maybe the qemu logs would tell us more. Specifically, we're trying to find out whether there is something listening on the host for VNC connections for that instance. One way, as I said, is qemu logs, another would be to check the output of 'sudo netstat -apn | grep LISTENING'.

Could you check those two things and come back to this bz? Thanks!
Comment 13 bigswitch 2017-08-25 16:33:35 UTC 
Let me give you the output from the new setup..

controller is able to ping to 172.17.0.19 

[root@overcloud-controller-0 heat-admin]# ping 172.17.0.19
PING 172.17.0.19 (172.17.0.19) 56(84) bytes of data.
64 bytes from 172.17.0.19: icmp_seq=1 ttl=64 time=0.162 ms
64 bytes from 172.17.0.19: icmp_seq=2 ttl=64 time=0.140 ms

2017-08-25 09:23:52.038 892580 INFO nova.console.websocketproxy [req-999f1a31-2755-42bf-b401-c3955a3204d9 - - - - -]  10: connect info: {u'instance_uuid': u'51a24036-b745-4e6d-a980-548089478c61', u'i1}
2017-08-25 09:23:52.038 892580 INFO nova.console.websocketproxy [req-999f1a31-2755-42bf-b401-c3955a3204d9 - - - - -]  10: connecting to: 172.17.0.19:5900
2017-08-25 09:23:52.047 892580 INFO nova.console.websocketproxy [req-999f1a31-2755-42bf-b401-c3955a3204d9 - - - - -] handler exception: [Errno 111] ECONNREFUSED
2017-08-25 09:24:39.832 892579 INFO nova.console.websocketproxy [-] 172.17.0.10: ignoring empty handshake

1. [root@overcloud-compute-0 heat-admin]# ifconfig vlan2
vlan2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.19  netmask 255.255.255.0  broadcast 172.17.0.255
        inet6 fe80::7cee:31ff:fead:f74e  prefixlen 64  scopeid 0x20<link>
        ether 7e:ee:31:ad:f7:4e  txqueuelen 1000  (Ethernet)
        RX packets 12010  bytes 2063804 (1.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12423  bytes 3566032 (3.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


[root@overcloud-compute-0 heat-admin]# sudo netstat -apn | grep LISTENING
unix  2      [ ACC ]     STREAM     LISTENING     30720    1951/libvirtd        /var/run/libvirt/libvirt-admin-sock
unix  2      [ ACC ]     STREAM     LISTENING     13569    1/systemd            /run/systemd/journal/stdout
unix  2      [ ACC ]     STREAM     LISTENING     18182    1050/ovs-vswitchd    /var/run/openvswitch/ovs-vswitchd.1050.ctl
unix  2      [ ACC ]     STREAM     LISTENING     1806     1/systemd            /run/lvm/lvmetad.socket
unix  2      [ ACC ]     SEQPACKET  LISTENING     1809     1/systemd            /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     36979    2559/master          private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     36982    2559/master          private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     36988    2559/master          private/trace
unix  2      [ ACC ]     STREAM     LISTENING     36991    2559/master          private/verify
unix  2      [ ACC ]     STREAM     LISTENING     36997    2559/master          private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     24857    1050/ovs-vswitchd    /var/run/openvswitch/br-api.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     37000    2559/master          private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     37003    2559/master          private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     21807    1050/ovs-vswitchd    /var/run/openvswitch/br-ex.mgmt
unix  2      [ ACC ]     STREAM     LISTENING     21809    1050/ovs-vswitchd    /var/run/openvswitch/br-ex.snoop
unix  2      [ ACC ]     STREAM     LISTENING     22837    1009/ovsdb-server    /var/run/openvswitch/db.sock
unix  2      [ ACC ]     STREAM     LISTENING     22839    1009/ovsdb-server    /var/run/openvswitch/ovsdb-server.1009.ctl
unix  2      [ ACC ]     STREAM     LISTENING     1848     1/systemd            /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     STREAM     LISTENING     16184    1/systemd            /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     31173    1887/snmpd           /var/agentx/master
unix  2      [ ACC ]     STREAM     LISTENING     25665    1000/NetworkManager  /var/run/NetworkManager/private-dhcp
unix  2      [ ACC ]     STREAM     LISTENING     22626    1/systemd            @ISCSID_UIP_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     22613    1/systemd            /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     22617    1/systemd            /var/run/libvirt/virtlockd-sock
unix  2      [ ACC ]     STREAM     LISTENING     22620    1/systemd            /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     22623    1/systemd            /var/run/libvirt/virtlogd-sock
unix  2      [ ACC ]     STREAM     LISTENING     36969    2559/master          public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     36972    2559/master          public/qmgr
unix  2      [ ACC ]     STREAM     LISTENING     36994    2559/master          public/flush
unix  2      [ ACC ]     STREAM     LISTENING     27776    1050/ovs-vswitchd    /var/run/openvswitch/br-api.snoop
unix  2      [ ACC ]     STREAM     LISTENING     39286    3403/python2         /tmp/rootwrap-LUD9ll/rootwrap.sock
unix  2      [ ACC ]     STREAM     LISTENING     180179   7432/qemu-kvm        /var/lib/libvirt/qemu/domain-1-instance-00000001/monitor.sock
unix  2      [ ACC ]     STREAM     LISTENING     33943    1987/ivs             /var/run/ivs-openflow.ivs.sock
unix  2      [ ACC ]     STREAM     LISTENING     33945    1987/ivs             /var/run/ivs-ucli.ivs.sock
unix  2      [ ACC ]     STREAM     LISTENING     33947    1987/ivs             /var/run/ivs-packet-trace.ivs.sock
unix  2      [ ACC ]     STREAM     LISTENING     18075    926/gssproxy         /run/gssproxy.sock
unix  2      [ ACC ]     STREAM     LISTENING     37006    2559/master          private/relay
unix  2      [ ACC ]     STREAM     LISTENING     37012    2559/master          private/error
unix  2      [ ACC ]     STREAM     LISTENING     37015    2559/master          private/retry
unix  2      [ ACC ]     STREAM     LISTENING     37018    2559/master          private/discard
unix  2      [ ACC ]     STREAM     LISTENING     37021    2559/master          private/local
unix  2      [ ACC ]     STREAM     LISTENING     37024    2559/master          private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     37027    2559/master          private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     36985    2559/master          private/defer
unix  2      [ ACC ]     STREAM     LISTENING     37030    2559/master          private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     18074    926/gssproxy         /var/lib/gssproxy/default.sock
unix  2      [ ACC ]     STREAM     LISTENING     37033    2559/master          private/scache
unix  2      [ ACC ]     STREAM     LISTENING     36976    2559/master          private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     22625    1/systemd            @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     37009    2559/master          public/showq
unix  2      [ ACC ]     STREAM     LISTENING     36962    2559/master          public/pickup
unix  2      [ ACC ]     STREAM     LISTENING     30716    1951/libvirtd        /var/run/libvirt/libvirt-sock
unix  2      [ ACC ]     STREAM     LISTENING     30718    1951/libvirtd        /var/run/libvirt/libvirt-sock-ro


qemu log


2017-08-25 16:19:43.822+0000: starting up libvirt version: 2.0.0, package: 10.el7_3.9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2017-05-04-06:48:37, x86-034.build.eng.bos.redhat.com), qemu version: 2.6.0 (qemu-kvm-rhev-2.6.0-28.el7_3.9), hostname: overcloud-compute-0.bigswitch.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=instance-00000001,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Broadwell,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+f16c,+rdrand,+arat,+tsc_adjust,+xsaveopt,+pdpe1gb,+abm,+rtm,+hle -m 2096 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 51a24036-b745-4e6d-a980-548089478c61 -smbios 'type=1,manufacturer=Red Hat,product=OpenStack Compute,version=14.0.6-2.el7ost,serial=38873f97-1eae-4dc9-a74a-71b078bb59c9,uuid=51a24036-b745-4e6d-a980-548089478c61,family=Virtual Machine' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-instance-00000001/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/51a24036-b745-4e6d-a980-548089478c61/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -add-fd set=0,fd=28 -chardev file,id=charserial0,path=/dev/fdset/0,append=on -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device vfio-pci,host=05:00.5,id=hostdev0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial1)
warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
warning: host doesn't support requested feature: CPUID.01H:EDX.acpi [bit 22]
warning: host doesn't support requested feature: CPUID.01H:EDX.ht [bit 28]
warning: host doesn't support requested feature: CPUID.01H:EDX.tm [bit 29]
warning: host doesn't support requested feature: CPUID.01H:EDX.pbe [bit 31]
warning: host doesn't support requested feature: CPUID.01H:ECX.dtes64 [bit 2]
warning: host doesn't support requested feature: CPUID.01H:ECX.monitor [bit 3]
warning: host doesn't support requested feature: CPUID.01H:ECX.ds_cpl [bit 4]
warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
warning: host doesn't support requested feature: CPUID.01H:ECX.smx [bit 6]
warning: host doesn't support requested feature: CPUID.01H:ECX.est [bit 7]
warning: host doesn't support requested feature: CPUID.01H:ECX.tm2 [bit 8]
warning: host doesn't support requested feature: CPUID.01H:ECX.xtpr [bit 14]
warning: host doesn't support requested feature: CPUID.01H:ECX.pdcm [bit 15]
warning: host doesn't support requested feature: CPUID.01H:ECX.dca [bit 18]
warning: host doesn't support requested feature: CPUID.01H:ECX.osxsave [bit 27]
Comment 14 bigswitch 2017-08-25 16:57:10 UTC 
[root@overcloud-controller-0 nova]# tcpdump -nni em3 port 5900
tcpdump: WARNING: em3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em3, link-type EN10MB (Ethernet), capture size 65535 bytes
09:54:14.261485 IP 172.17.0.10.46772 > 172.17.0.19.5900: Flags [S], seq 3002318097, win 29200, options [mss 1460,sackOK,TS val 60489594 ecr 0,nop,wscale 7], length 0
09:54:14.261617 IP 172.17.0.19.5900 > 172.17.0.10.46772: Flags [R.], seq 0, ack 3002318098, win 0, length 0
09:55:41.149907 IP 172.17.0.10.52042 > 172.17.0.19.5900: Flags [S], seq 1502480943, win 29200, options [mss 1460,sackOK,TS val 60576482 ecr 0,nop,wscale 7], length 0
09:55:41.150040 IP 172.17.0.19.5900 > 172.17.0.10.52042: Flags [R.], seq 0, ack 1502480944, win 0, length 0
Comment 15 Artom Lifshitz 2017-08-25 18:36:04 UTC 
Here's an interesting bit from the qemu instance command line:

-vnc 127.0.0.1:0

Obviously that means nothing outside of localhost will be able to connect to the instance' VNC console, not even nonvcproxy (since it's running on a different host).

What does nova.conf look like on the compute?
Comment 16 bigswitch 2017-08-25 18:40:40 UTC 
[vnc]
vncserver_proxyclient_address=172.17.0.19
vncserver_listen=127.0.0.1
keymap=en-us
enabled=True
novncproxy_base_url=http://10.8.63.15:6080/vnc_auto.html
[cache]
Comment 17 Artom Lifshitz 2017-08-25 19:04:59 UTC 
This started after the 'RHOSP Patch 06-15-2017' (to quote the original bug description) was installed, correct? So VNC consoles were accessible before that patch installation, and all VNC consoles now fail to connect?

I'm assuming nova.conf hasn't been modified by the patch, but it'd be good to check, if that's possible. Specifically, whether the vncserver_listen option was changed.

Also, do you have the exact version of the packages that were installed as part of 'RHOSP Patch 06-15-2017'? They would look something like 'openstack-nova-14.0.7-9.el7ost'.

Thanks!
Comment 18 bigswitch 2017-08-25 19:54:41 UTC 
When we pull the latest RHOSP 10 0615 , problem is seen , Yes previous RHOSP 10 images were good. 

It is not that i have upgraded RHOSP 10 OSP cluster , I just updated the RHOSP overcloud qcow image and deploy a new cluster.

We dont have any setup which is still old RHOSP 10 image though
Comment 19 bigswitch 2017-08-25 20:48:38 UTC 
In the working setup i see the local IP used 

[root@rhosp10-compute-1 heat-admin]# sudo netstat -apn | grep 5900
tcp        0      0 172.17.0.19:5900        0.0.0.0:*               LISTEN      31870/qemu-kvm
tcp        0      0 172.17.0.19:5900        172.17.0.22:47122       ESTABLISHED 31870/qemu-kvm

In the not working setup

[root@overcloud-compute-0 nova]#  sudo netstat -apn | grep 5900
tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      7432/qemu-kvm
Comment 20 bigswitch 2017-08-25 20:58:28 UTC 
Guys , Found the issue it is selinux setting enforcing is causing the issue

If i make the selinux setting to permissive , it is working..
Comment 21 bigswitch 2017-08-28 18:30:52 UTC 
There is also a one more setting between the relaases 

Now all the test deployment , i see the nova.conf has the following setting

[vnc]
vncserver_proxyclient_address=172.17.0.10
vncserver_listen=127.0.0.1
keymap=en-us
enabled=True
novncproxy_base_url=http://10.8.65.209:6080/vnc_auto.html
[cache]


If we make changes to the vncserver_listen to following

[vnc]
vncserver_proxyclient_address=172.17.0.10
vncserver_listen=172.17.0.10
keymap=en-us
enabled=True
novncproxy_base_url=http://10.8.65.209:6080/vnc_auto.html
[cache]

and then reboot the system , and lauch an instance ,

[root@overcloud-compute-0 heat-admin]# sudo netstat -apn | grep 5900
tcp        0      0 172.17.0.10:5900        0.0.0.0:*               LISTEN      3613/qemu-kvm

Instance VNC console is working
Comment 22 bigswitch 2017-09-05 16:39:43 UTC 
There is one more solution for this 

in compute /etc/nova/nova.conf 

[vnc]
vncserver_proxyclient_address=172.20.0.20
vncserver_listen=0.0.0.0 ------ use 0.0.0.0 instaed of 127.0.0.1
keymap=en-us
enabled=True
novncproxy_base_url=http://10.11.2.10:6080/vnc_auto.html
[cache]


Not sure who sets the 127.0.0.1 , we need to make this change during deployment , THis is very critical , we have to let all of our customer to know about this , and need to change manually on all compute nodes..
Comment 23 Ollie Walsh 2017-09-06 16:16:48 UTC 
vncserver_listen should be set to the IP on the internal api network.

The fact that it is defaulting to 127.0.0.1 suggest an issue with the RHOSP director network config. e.g NovaLibvirtNetwork is missing from the ServiceNetMap.

Are any custom environment files used? And if so do they override the ServiceNetMap parameter.
Comment 24 bigswitch 2017-09-06 17:09:27 UTC 
Not sure , How these setting gets override , We have used following deployment command 

#!/bin/bash

source /home/stack/stackrc

openstack overcloud deploy --templates  -r /home/stack/templates/roles_data.yaml -e /home/stack/templates/node-info.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/
network-isolation.yaml -e /home/stack/templates/network-environment-ivs.yaml -e /home/stack/templates/bigswitch-config-pv.yaml  --neutron-disable-tunneling  --ntp-server 10.8.29.9 -
-timeout 150

Not sure if any of the above settings change these,
Comment 25 Ollie Walsh 2017-09-06 18:39:34 UTC 
All of the -e options are environment files. Is it possible to provide these files or at least check the contents for ServiceNetMap?
Comment 26 bigswitch 2017-09-06 18:53:32 UTC 
Created attachment 1322783 [details]
templates
Comment 27 Ollie Walsh 2017-09-06 19:26:24 UTC 
Ok, ServiceNetMap is not being overridden in any of the environment files.

Could you also provide the output from following commands please:

Undercloud:
rpm -q openstack-tripleo-heat-templates

Compute node:
rpm -q puppet-tripleo
rpm -q puppet-nova
hiera -c /etc/puppet/hiera.yaml 'nova::compute::libvirt::vncserver_listen'
Comment 28 bigswitch 2017-09-06 19:31:45 UTC 
[stack@undercloud ~]$ rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-5.2.0-25.el7ost.noarch

I deleted the deployment , i can start and get the output
Comment 29 bigswitch 2017-09-06 19:41:38 UTC 
i deployed new RHOSP director and overcloud deployement with 1 controller and one compute node , in this deployment , vnc_server listen seems to be correctly set


[stack@rhosp10 ~]$ ssh heat-admin.2.10
Last login: Tue Sep  5 17:24:17 2017 from 192.0.2.1
[heat-admin@overcloud-compute-0 ~]$ sudo su
[root@overcloud-compute-0 heat-admin]# rpm -q puppet-tripleo
puppet-tripleo-5.6.0-4.el7ost.noarch
[root@overcloud-compute-0 heat-admin]# rpm -q puppet-nova
puppet-nova-9.5.0-6.el7ost.noarch
[root@overcloud-compute-0 heat-admin]# hiera -c /etc/puppet/hiera.yaml 'nova::compute::libvirt::vncserver_listen'
172.17.0.16

Not sure if there is any update on the templates..
Comment 30 Ollie Walsh 2017-09-06 20:24:08 UTC 
Ok, thanks. Do you know of any difference that may explain why it didn't work previously? E.g was than an update of an existing deployment while this is a fresh install.
Comment 31 bigswitch 2017-09-06 20:29:51 UTC 
Yes , there was an update to exsisting RHOSP 10 undercloud deployment previously..
Comment 32 Ollie Walsh 2017-09-07 19:48:44 UTC 
Ok, thanks. If you can provide any info that may help reproduce that it would be very much appreciated as I've not managed to reproduce this issue when upgrading from an OSP10 GA release e.g:
Initial version deployed.
Initial deploy command line, environment files etc...
Version upgraded to.
Upgrade commands run etc...
Comment 33 Artom Lifshitz 2017-09-21 17:27:45 UTC 
I'm going to close this bug for now since it seems no one can reproduce it anymore. If you're able to reproduce it, please re-open and reply with the information requested by Ollie in comment #32.
Comment 34 Red Hat Bugzilla 2023-09-15 00:03:14 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days