Description of problem: Compute nodes need to have open the port 5900/tcp so nova-novncconsole Version-Release number of selected component (if applicable): PUDDLE UPDATED FRIDAY 21st [stack@undercloud ~]$ sudo rpm -qa |grep openstack python-openstackclient-3.2.0-2.el7ost.noarch openstack-utils-2016.1-1.el7ost.noarch openstack-selinux-0.7.11-1.el7ost.noarch openstack-nova-common-14.0.1-3.el7ost.noarch openstack-neutron-9.0.0-1.4.el7ost.noarch openstack-heat-api-cfn-7.0.0-3.el7ost.noarch openstack-heat-templates-0.0.1-0.20161011152629.40a4ed0.el7ost.noarch openstack-tripleo-image-elements-5.0.0-1.el7ost.noarch openstack-tripleo-common-5.3.0-1.el7ost.noarch openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch openstack-mistral-api-3.0.1-0.20161013000829.6356bce.el7ost.noarch openstack-nova-api-14.0.1-3.el7ost.noarch openstack-swift-account-2.10.1-0.20161011234731.3349016.el7ost.noarch openstack-tripleo-ui-1.0.4-1.el7ost.noarch openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch openstack-zaqar-3.0.0-2.el7ost.noarch openstack-heat-common-7.0.0-3.el7ost.noarch openstack-mistral-engine-3.0.1-0.20161013000829.6356bce.el7ost.noarch openstack-nova-cert-14.0.1-3.el7ost.noarch openstack-swift-container-2.10.1-0.20161011234731.3349016.el7ost.noarch python-openstack-mistral-3.0.1-0.20161013000829.6356bce.el7ost.noarch openstack-ironic-common-6.2.2-0.20161012001047.574a836.el7ost.noarch openstack-ironic-api-6.2.2-0.20161012001047.574a836.el7ost.noarch openstack-mistral-executor-3.0.1-0.20161013000829.6356bce.el7ost.noarch openstack-nova-compute-14.0.1-3.el7ost.noarch openstack-heat-api-7.0.0-3.el7ost.noarch openstack-swift-object-2.10.1-0.20161011234731.3349016.el7ost.noarch puppet-openstack_extras-9.4.0-1.el7ost.noarch python-openstacksdk-0.9.5-1.el7ost.noarch openstack-tripleo-validations-5.1.1-0.20161003173643.3652f12.el7ost.noarch openstack-ironic-inspector-4.2.1-0.20161005144819.9a079eb.el7ost.noarch openstack-mistral-common-3.0.1-0.20161013000829.6356bce.el7ost.noarch openstack-keystone-10.0.0-2.el7ost.noarch openstack-nova-scheduler-14.0.1-3.el7ost.noarch openstack-swift-proxy-2.10.1-0.20161011234731.3349016.el7ost.noarch puppet-openstacklib-9.4.0-0.20161004171440.0e58c86.el7ost.noarch openstack-neutron-openvswitch-9.0.0-1.4.el7ost.noarch openstack-neutron-ml2-9.0.0-1.4.el7ost.noarch openstack-heat-engine-7.0.0-3.el7ost.noarch openstack-glance-13.0.0-1.el7ost.noarch openstack-tripleo-puppet-elements-5.0.0-0.20161003213431.200d011.el7ost.noarch openstack-neutron-common-9.0.0-1.4.el7ost.noarch openstack-ironic-conductor-6.2.2-0.20161012001047.574a836.el7ost.noarch openstack-nova-conductor-14.0.1-3.el7ost.noarch openstack-tempest-13.0.1-0.20161018143400.bafe630.1.el7ost.noarch openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch How reproducible: Deploy with regular templates, not much fancy stuff. It was working with puddles prior Oct 18th. The Oct 21th seems to have some regressions (I also found something with StorageMgmt Predictable VIPs, maybe related) Steps to Reproduce: 1. Deploy opentstack, it finishes OK 2. Boot a VM, it boots OK 3. From Horizon, NovaVNC cannot connect (Failed to connect to server (code: 1006) Actual results: (on the CONSOLE NODE that has the main VIP, the one that I use for horizon), we see the 2016-10-22 20:30:34.041 29351 INFO nova.console.websocketproxy [req-4575bf20-312b-4fa8-8227-3662ed584d98 - - - - -] 33: connecting to: 172.16.3.25:5900 2016-10-22 20:30:34.047 29351 INFO nova.console.websocketproxy [req-4575bf20-312b-4fa8-8227-3662ed584d98 - - - - -] handler exception: [Errno 113] EHOSTUNREACH [root@overcloud-compute-1 ~]# cat /etc/sysconfig/iptables # Generated by iptables-save v1.4.21 on Sat Oct 22 18:24:04 2016 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m comment --comment "000 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -m comment --comment "001 accept all icmp" -m state --state NEW -j ACCEPT -A INPUT -i lo -m comment --comment "002 accept all to lo interface" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "003 accept ssh" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 123 -m comment --comment "105 ntp" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 4789 -m comment --comment "118 neutron vxlan networks" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 161 -m comment --comment "127 snmp" -m state --state NEW -j ACCEPT -A INPUT -p gre -m comment --comment "136 neutron gre networks" -m state --state NEW -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT -m comment --comment "998 log all" -j LOG -A INPUT -m comment --comment "999 drop all" -m state --state NEW -j DROP -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Sat Oct 22 18:24:04 2016 Expected results: Compute nodes should have the port 5900 open so the VNC proxy in the controller can connect to the qemu-kvm process (that listens on 5900 on every compute) Additional info:
This fixes the problem when applied on EVERY compute sudo iptables -I INPUT 4 -p tcp -m multiport --dports 5900 -m comment --comment "999 manually adding 5900 for vnc and qemu-kvm" -m state --state NEW -j ACCEPT The rule position #4 is arbitrary, it's just to put it before the REJECT line BTW I did try re-applying the overcloud deploy again, to ensure a clean puppet pass. It did pass without errors. But I have to apply the iptables fix manually otherwise VNC doesn't work
http://docs.openstack.org/juno/config-reference/content/firewalls-default-ports.html
*** Bug 1391125 has been marked as a duplicate of this bug. ***
Puppet-nova is puppet-nova-9.4.0-1.el7ost.noarch [root@undercloud ~]# rpm -qa |grep puppet puppet-tempest-9.4.0-1.el7ost.noarch puppet-firewall-1.8.1-2.e70157egit.el7ost.noarch puppet-xinetd-2.0.0-2.f9d6e18git.el7ost.noarch puppet-kafka-2.1.0-3.061ef74git.el7ost.noarch puppet-aodh-9.4.0-2.el7ost.noarch puppet-mongodb-0.14.0-0.20161012180542.cf57011.el7ost.noarch puppet-gnocchi-9.4.0-2.el7ost.noarch puppet-oslo-9.4.0-1.el7ost.noarch puppet-git-0.4.0-1.5e86224git.el7ost.noarch puppet-nssdb-1.0.1-1.el7ost.noarch puppet-mistral-9.4.0-1.el7ost.noarch puppet-vlan-0.1.0-1.el7ost.noarch puppet-inifile-1.6.0-2.el7ost.noarch puppet-ceph-2.2.1-2.el7ost.noarch puppet-redis-1.2.3-0.20161016000604.9711564.el7ost.noarch puppet-3.8.7-2.el7.noarch puppet-datacat-0.6.2-1.10f6ddegit.el7ost.noarch puppet-kibana3-0.0.4-1.6ca9631git.el7ost.noarch puppet-cinder-9.4.1-1.el7ost.noarch puppet-keepalived-0.0.2-0.20161004174022.bbca37a.el7ost.noarch puppet-zookeeper-0.6.1-1.3bc30fcgit.el7ost.noarch puppet-trove-9.4.0-1.el7ost.noarch puppet-snmp-3.6.0-1.7d4c97cgit.el7ost.noarch puppet-n1k-vsm-0.0.2-0.20161003153532.91772fa.el7ost.noarch puppet-remote-0.0.1-1.el7ost.noarch openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch puppet-ceilometer-9.4.0-2.el7ost.noarch puppet-elasticsearch-0.14.0-0.20161012195339.b23bab1.el7ost.noarch puppet-vcsrepo-1.4.0-2.cd6c3bcgit.el7ost.noarch puppet-vswitch-5.4.0-1.el7ost.noarch puppet-glance-9.4.0-1.el7ost.noarch puppet-heat-9.4.1-1.el7ost.noarch puppet-midonet-2015.06.9-0.20161003154558.bafa9e9.el7ost.noarch puppet-ironic-9.4.0-1.el7ost.noarch puppet-timezone-3.3.0-1.cf62f1bgit.el7ost.noarch puppet-zaqar-9.4.0-1.el7ost.noarch puppet-horizon-9.4.0-1.el7ost.noarch puppet-apache-1.10.0-0.20161015235625.cf2ff7e.el7ost.noarch puppet-tomcat-1.5.0-0.20161011204918.bbdbf65.el7ost.noarch puppet-neutron-9.4.0-2.el7ost.noarch puppet-mysql-3.9.0-0.20161017182819.669ece6.el7ost.noarch puppet-corosync-5.0.0-0.20161013095720.950324c.el7ost.noarch puppet-keystone-9.4.0-1.el7ost.noarch puppet-openstack_extras-9.4.0-1.el7ost.noarch puppet-certmonger-1.1.1-0.20161009144218.1157a7e.el7ost.noarch puppet-sysctl-0.0.11-1.el7ost.noarch puppet-ssh-2.9.1-1.el7ost.noarch puppet-contrail-1.0.0-0.20161003175205.c0f7cde.el7ost.noarch puppet-kmod-2.1.1-0.20161003155007.0d69a96.el7ost.noarch puppet-module-data-0.0.4-1.28dafcegit.el7ost.noarch puppet-concat-2.2.0-0.20161012002654.c70d77c.el7ost.noarch puppet-tripleo-5.3.0-1.el7ost.noarch puppet-fluentd-0.7.0-0.20161012220912.0441f39.el7ost.noarch puppet-barbican-9.4.0-2.el7ost.noarch puppet-staging-1.0.4-1.b466d93git.el7ost.noarch puppet-openstacklib-9.4.0-0.20161004171440.0e58c86.el7ost.noarch puppet-nova-9.4.0-1.el7ost.noarch puppet-uchiwa-1.0.1-1.64ce619git.el7ost.noarch puppet-sahara-9.4.0-1.el7ost.noarch puppet-cassandra-2.0.2-0.20161015225641.782ccbc.el7ost.noarch puppet-java-1.6.0-3.2b0bd48git.el7ost.noarch puppet-swift-9.4.1-3.el7ost.noarch puppet-collectd-5.1.0-0.20161018180615.b26caaa.el7ost.noarch puppet-rabbitmq-5.5.0-1.837d556git.el7ost.noarch puppet-memcached-2.8.1-1.bfa64e0git.el7ost.noarch puppet-opendaylight-3.7.0-1.b2d8d9dgit.el7ost.noarch puppet-manila-9.4.0-1.el7ost.noarch puppet-ntp-4.2.0-1.d93d4b6git.el7ost.noarch
In OSP8, this iptables setting was done here: /usr/share/instack-undercloud/puppet-stack-config/os-refresh-config/post-configure.d/10-iptables:add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT
sorry I meant this one tripleo-image-elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables:add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT
In OSP10, the setting also exists in the same tripleo-image-elements folder [root@undercloud share]# cat /usr/share/tripleo-image-elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables #!/bin/bash set -eu # open default port for nova-novncproxy connections add-rule INPUT -p tcp -m multiport --dports 6080 -j ACCEPT add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT But this script has only been executed in the CONTROLLERs, not in the COMPUTE NODEs, It should have been executed on BOTH (We are also looking into this one; https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-libvirt.yaml)
Maybe just editing /usr/share/tripleo-image-elements/nova-compute/element-deps to include the novnc settings, or even better, add the firewall (5900:5999) to ./tripleo-image-elements/nova-compute/os-refresh-config/post-configure.d/80-nova-compute like we do already with ./tripleo-image-elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables: add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT
why do we have 2 disparate sources of firewall rules? bash script in os-refresh-config and puppet/hiera
We would really prefer to see these firewall settings for tcp/5900-5999 (compute node) in hiera/puppet , like it is done for other ports here https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-libvirt.yaml I reeally don't like the previous approach, via https://github.com/openstack/tripleo-image-elements/blob/master/elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables These ports must be open on the compute node (qemu-kvm process), not on the controller (which requires port 6080 for novncproxy). more info: http://docs.openstack.org/juno/config-reference/content/firewalls-default-ports.html
(In reply to Jon Schlueter from comment #9) > why do we have 2 disparate sources of firewall rules? bash script in > os-refresh-config and puppet/hiera we don't. 98-nova-novncproxy-fedora-iptables is not used.
eglynn> slagle: novnc itself in on the UI plate, but arguably the firewall port range is specific to the nova usage of novnc and should be on the compute plate
Merged in stable/newton. Next rebased build will include it.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html