Bug 1387914 - Firewall rules for nova vnc and qemu-kvm not set
Summary: Firewall rules for nova vnc and qemu-kvm not set
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: All
OS: Unspecified
unspecified
high
Target Milestone: rc
: 10.0 (Newton)
Assignee: Sven Anderson
QA Contact: Gabriel Szasz
URL:
Whiteboard:
: 1391125 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-23 14:54 UTC by Marcos Garcia
Modified: 2016-12-14 16:24 UTC (History)
12 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-1.4.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:24:34 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 392898 None None None 2016-11-02 17:41:04 UTC
OpenStack gerrit 394019 None None None 2016-11-05 15:53:07 UTC

Description Marcos Garcia 2016-10-23 14:54:56 UTC
Description of problem:
Compute nodes need to have open the port 5900/tcp so nova-novncconsole

Version-Release number of selected component (if applicable):
PUDDLE UPDATED FRIDAY 21st
[stack@undercloud ~]$ sudo rpm -qa |grep openstack
python-openstackclient-3.2.0-2.el7ost.noarch
openstack-utils-2016.1-1.el7ost.noarch
openstack-selinux-0.7.11-1.el7ost.noarch
openstack-nova-common-14.0.1-3.el7ost.noarch
openstack-neutron-9.0.0-1.4.el7ost.noarch
openstack-heat-api-cfn-7.0.0-3.el7ost.noarch
openstack-heat-templates-0.0.1-0.20161011152629.40a4ed0.el7ost.noarch
openstack-tripleo-image-elements-5.0.0-1.el7ost.noarch
openstack-tripleo-common-5.3.0-1.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-mistral-api-3.0.1-0.20161013000829.6356bce.el7ost.noarch
openstack-nova-api-14.0.1-3.el7ost.noarch
openstack-swift-account-2.10.1-0.20161011234731.3349016.el7ost.noarch
openstack-tripleo-ui-1.0.4-1.el7ost.noarch
openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch
openstack-zaqar-3.0.0-2.el7ost.noarch
openstack-heat-common-7.0.0-3.el7ost.noarch
openstack-mistral-engine-3.0.1-0.20161013000829.6356bce.el7ost.noarch
openstack-nova-cert-14.0.1-3.el7ost.noarch
openstack-swift-container-2.10.1-0.20161011234731.3349016.el7ost.noarch
python-openstack-mistral-3.0.1-0.20161013000829.6356bce.el7ost.noarch
openstack-ironic-common-6.2.2-0.20161012001047.574a836.el7ost.noarch
openstack-ironic-api-6.2.2-0.20161012001047.574a836.el7ost.noarch
openstack-mistral-executor-3.0.1-0.20161013000829.6356bce.el7ost.noarch
openstack-nova-compute-14.0.1-3.el7ost.noarch
openstack-heat-api-7.0.0-3.el7ost.noarch
openstack-swift-object-2.10.1-0.20161011234731.3349016.el7ost.noarch
puppet-openstack_extras-9.4.0-1.el7ost.noarch
python-openstacksdk-0.9.5-1.el7ost.noarch
openstack-tripleo-validations-5.1.1-0.20161003173643.3652f12.el7ost.noarch
openstack-ironic-inspector-4.2.1-0.20161005144819.9a079eb.el7ost.noarch
openstack-mistral-common-3.0.1-0.20161013000829.6356bce.el7ost.noarch
openstack-keystone-10.0.0-2.el7ost.noarch
openstack-nova-scheduler-14.0.1-3.el7ost.noarch
openstack-swift-proxy-2.10.1-0.20161011234731.3349016.el7ost.noarch
puppet-openstacklib-9.4.0-0.20161004171440.0e58c86.el7ost.noarch
openstack-neutron-openvswitch-9.0.0-1.4.el7ost.noarch
openstack-neutron-ml2-9.0.0-1.4.el7ost.noarch
openstack-heat-engine-7.0.0-3.el7ost.noarch
openstack-glance-13.0.0-1.el7ost.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20161003213431.200d011.el7ost.noarch
openstack-neutron-common-9.0.0-1.4.el7ost.noarch
openstack-ironic-conductor-6.2.2-0.20161012001047.574a836.el7ost.noarch
openstack-nova-conductor-14.0.1-3.el7ost.noarch
openstack-tempest-13.0.1-0.20161018143400.bafe630.1.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch



How reproducible:
Deploy with regular templates, not much fancy stuff. It was working with puddles prior Oct 18th. The Oct 21th seems to have some regressions (I also found something with StorageMgmt Predictable VIPs, maybe related)


Steps to Reproduce:
1. Deploy opentstack, it finishes OK
2. Boot a VM, it boots OK
3. From Horizon, NovaVNC cannot connect (Failed to connect to server (code: 1006)

Actual results:
(on the CONSOLE NODE that has the main VIP, the one that I use for horizon), we see the  
2016-10-22 20:30:34.041 29351 INFO nova.console.websocketproxy [req-4575bf20-312b-4fa8-8227-3662ed584d98 - - - - -]  33: connecting to: 172.16.3.25:5900
2016-10-22 20:30:34.047 29351 INFO nova.console.websocketproxy [req-4575bf20-312b-4fa8-8227-3662ed584d98 - - - - -] handler exception: [Errno 113] EHOSTUNREACH

[root@overcloud-compute-1 ~]# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.21 on Sat Oct 22 18:24:04 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m comment --comment "000 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -m comment --comment "001 accept all icmp" -m state --state NEW -j ACCEPT
-A INPUT -i lo -m comment --comment "002 accept all to lo interface" -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "003 accept ssh" -m state --state NEW -j ACCEPT
-A INPUT -p udp -m multiport --dports 123 -m comment --comment "105 ntp" -m state --state NEW -j ACCEPT
-A INPUT -p udp -m multiport --dports 4789 -m comment --comment "118 neutron vxlan networks" -m state --state NEW -j ACCEPT
-A INPUT -p udp -m multiport --dports 161 -m comment --comment "127 snmp" -m state --state NEW -j ACCEPT
-A INPUT -p gre -m comment --comment "136 neutron gre networks" -m state --state NEW -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -m comment --comment "998 log all" -j LOG
-A INPUT -m comment --comment "999 drop all" -m state --state NEW -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Sat Oct 22 18:24:04 2016


Expected results:
Compute nodes should have the port 5900 open so the VNC proxy in the controller can connect to the qemu-kvm process (that listens on 5900 on every compute) 


Additional info:

Comment 1 Marcos Garcia 2016-10-23 15:15:33 UTC
This fixes the problem when applied on EVERY compute

sudo iptables -I INPUT 4 -p tcp -m multiport --dports 5900 -m comment --comment "999 manually adding 5900 for vnc and qemu-kvm" -m state --state NEW -j ACCEPT

The rule position #4 is arbitrary, it's just to put it before the REJECT line

BTW I did try re-applying the overcloud deploy again, to ensure a clean puppet pass. It did pass without errors. But I have to apply the iptables fix manually otherwise VNC doesn't work

Comment 3 James Slagle 2016-11-02 15:55:43 UTC
*** Bug 1391125 has been marked as a duplicate of this bug. ***

Comment 4 Marcos Garcia 2016-11-02 15:59:32 UTC
Puppet-nova is puppet-nova-9.4.0-1.el7ost.noarch

[root@undercloud ~]# rpm -qa |grep puppet
puppet-tempest-9.4.0-1.el7ost.noarch
puppet-firewall-1.8.1-2.e70157egit.el7ost.noarch
puppet-xinetd-2.0.0-2.f9d6e18git.el7ost.noarch
puppet-kafka-2.1.0-3.061ef74git.el7ost.noarch
puppet-aodh-9.4.0-2.el7ost.noarch
puppet-mongodb-0.14.0-0.20161012180542.cf57011.el7ost.noarch
puppet-gnocchi-9.4.0-2.el7ost.noarch
puppet-oslo-9.4.0-1.el7ost.noarch
puppet-git-0.4.0-1.5e86224git.el7ost.noarch
puppet-nssdb-1.0.1-1.el7ost.noarch
puppet-mistral-9.4.0-1.el7ost.noarch
puppet-vlan-0.1.0-1.el7ost.noarch
puppet-inifile-1.6.0-2.el7ost.noarch
puppet-ceph-2.2.1-2.el7ost.noarch
puppet-redis-1.2.3-0.20161016000604.9711564.el7ost.noarch
puppet-3.8.7-2.el7.noarch
puppet-datacat-0.6.2-1.10f6ddegit.el7ost.noarch
puppet-kibana3-0.0.4-1.6ca9631git.el7ost.noarch
puppet-cinder-9.4.1-1.el7ost.noarch
puppet-keepalived-0.0.2-0.20161004174022.bbca37a.el7ost.noarch
puppet-zookeeper-0.6.1-1.3bc30fcgit.el7ost.noarch
puppet-trove-9.4.0-1.el7ost.noarch
puppet-snmp-3.6.0-1.7d4c97cgit.el7ost.noarch
puppet-n1k-vsm-0.0.2-0.20161003153532.91772fa.el7ost.noarch
puppet-remote-0.0.1-1.el7ost.noarch
openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch
puppet-ceilometer-9.4.0-2.el7ost.noarch
puppet-elasticsearch-0.14.0-0.20161012195339.b23bab1.el7ost.noarch
puppet-vcsrepo-1.4.0-2.cd6c3bcgit.el7ost.noarch
puppet-vswitch-5.4.0-1.el7ost.noarch
puppet-glance-9.4.0-1.el7ost.noarch
puppet-heat-9.4.1-1.el7ost.noarch
puppet-midonet-2015.06.9-0.20161003154558.bafa9e9.el7ost.noarch
puppet-ironic-9.4.0-1.el7ost.noarch
puppet-timezone-3.3.0-1.cf62f1bgit.el7ost.noarch
puppet-zaqar-9.4.0-1.el7ost.noarch
puppet-horizon-9.4.0-1.el7ost.noarch
puppet-apache-1.10.0-0.20161015235625.cf2ff7e.el7ost.noarch
puppet-tomcat-1.5.0-0.20161011204918.bbdbf65.el7ost.noarch
puppet-neutron-9.4.0-2.el7ost.noarch
puppet-mysql-3.9.0-0.20161017182819.669ece6.el7ost.noarch
puppet-corosync-5.0.0-0.20161013095720.950324c.el7ost.noarch
puppet-keystone-9.4.0-1.el7ost.noarch
puppet-openstack_extras-9.4.0-1.el7ost.noarch
puppet-certmonger-1.1.1-0.20161009144218.1157a7e.el7ost.noarch
puppet-sysctl-0.0.11-1.el7ost.noarch
puppet-ssh-2.9.1-1.el7ost.noarch
puppet-contrail-1.0.0-0.20161003175205.c0f7cde.el7ost.noarch
puppet-kmod-2.1.1-0.20161003155007.0d69a96.el7ost.noarch
puppet-module-data-0.0.4-1.28dafcegit.el7ost.noarch
puppet-concat-2.2.0-0.20161012002654.c70d77c.el7ost.noarch
puppet-tripleo-5.3.0-1.el7ost.noarch
puppet-fluentd-0.7.0-0.20161012220912.0441f39.el7ost.noarch
puppet-barbican-9.4.0-2.el7ost.noarch
puppet-staging-1.0.4-1.b466d93git.el7ost.noarch
puppet-openstacklib-9.4.0-0.20161004171440.0e58c86.el7ost.noarch
puppet-nova-9.4.0-1.el7ost.noarch
puppet-uchiwa-1.0.1-1.64ce619git.el7ost.noarch
puppet-sahara-9.4.0-1.el7ost.noarch
puppet-cassandra-2.0.2-0.20161015225641.782ccbc.el7ost.noarch
puppet-java-1.6.0-3.2b0bd48git.el7ost.noarch
puppet-swift-9.4.1-3.el7ost.noarch
puppet-collectd-5.1.0-0.20161018180615.b26caaa.el7ost.noarch
puppet-rabbitmq-5.5.0-1.837d556git.el7ost.noarch
puppet-memcached-2.8.1-1.bfa64e0git.el7ost.noarch
puppet-opendaylight-3.7.0-1.b2d8d9dgit.el7ost.noarch
puppet-manila-9.4.0-1.el7ost.noarch
puppet-ntp-4.2.0-1.d93d4b6git.el7ost.noarch

Comment 5 Marcos Garcia 2016-11-02 16:28:29 UTC
In OSP8, this iptables setting was done here:
/usr/share/instack-undercloud/puppet-stack-config/os-refresh-config/post-configure.d/10-iptables:add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT

Comment 6 Marcos Garcia 2016-11-02 16:32:49 UTC
sorry I meant this one
tripleo-image-elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables:add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT

Comment 7 Marcos Garcia 2016-11-02 16:44:50 UTC
In OSP10, the setting also exists in the same tripleo-image-elements folder
[root@undercloud share]# cat /usr/share/tripleo-image-elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables
#!/bin/bash
set -eu

# open default port for nova-novncproxy connections
add-rule INPUT -p tcp -m multiport --dports 6080 -j ACCEPT
add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT

But this script has only been executed in the CONTROLLERs, not in the COMPUTE NODEs,
It should have been executed on BOTH


(We are also looking into this one; https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-libvirt.yaml)

Comment 8 Marcos Garcia 2016-11-02 16:57:33 UTC
Maybe just editing
/usr/share/tripleo-image-elements/nova-compute/element-deps
to include the novnc settings, or even better, add the firewall (5900:5999) to 
./tripleo-image-elements/nova-compute/os-refresh-config/post-configure.d/80-nova-compute 

like we do already with ./tripleo-image-elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables:
add-rule INPUT -p tcp -m multiport --dports 5900:5999 -j ACCEPT

Comment 9 Jon Schlueter 2016-11-02 17:01:24 UTC
why do we have 2 disparate sources of firewall rules? bash script in os-refresh-config and puppet/hiera

Comment 10 Marcos Garcia 2016-11-02 17:10:51 UTC
We would really prefer to see these firewall settings for tcp/5900-5999 (compute node) in hiera/puppet , like it is done for other ports here
https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-libvirt.yaml

I reeally don't like the previous approach, via https://github.com/openstack/tripleo-image-elements/blob/master/elements/nova-novncproxy/os-refresh-config/pre-configure.d/98-nova-novncproxy-fedora-iptables

These ports must be open on the compute node (qemu-kvm process), not on the controller (which requires port 6080 for novncproxy).

more info:
http://docs.openstack.org/juno/config-reference/content/firewalls-default-ports.html

Comment 12 James Slagle 2016-11-02 17:30:01 UTC
(In reply to Jon Schlueter from comment #9)
> why do we have 2 disparate sources of firewall rules? bash script in
> os-refresh-config and puppet/hiera

we don't. 98-nova-novncproxy-fedora-iptables is not used.

Comment 13 James Slagle 2016-11-02 17:32:03 UTC
eglynn> slagle: novnc itself in on the UI plate, but arguably the firewall port range is specific to the nova usage of novnc and should be on the compute plate

Comment 16 Sven Anderson 2016-11-07 16:30:54 UTC
Merged in stable/newton. Next rebased build will include it.

Comment 22 errata-xmlrpc 2016-12-14 16:24:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.