Updating the VXLAN firewall rules (and perhaps all firewall rules) with Director will instruct Director to save all iptables rules into /etc/sysconfig/iptables (e.g. run a command similar to `iptables-save > /etc/sysconfig/iptables`). The result is that /etc/sysconfig/iptables is populated with neutron rules that existed at the time when Director ran a stack update. This bug goes unnoticed by the administrator until later. E.g., one way to see the negative consequences of this is to delete a neutron port if this port's security group was implemented via an ipset. An iptables rule in /etc/sysconfig/iptables points to this ipset, and as a consequence, `systemctl restart iptables` does not work which will also break any future updates with Director. Reproducing this in a lab: Before the update, baseline: ~~~ [stack@undercloud-7 ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | 5fb73246-5532-430a-b4a9-529b0f92262f | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.19 | | 50a2c310-2473-46c3-b2c0-15ad07c8eb44 | overcloud-compute-1 | ACTIVE | - | Running | ctlplane=192.0.2.13 | | d68b514f-30e6-4bae-bc5b-f5218bd476e1 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.7 | | b7edaf0e-9df5-42da-b658-f830ae8ef2a7 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.6 | | fe87549b-4cdb-4ae2-8e34-d7751edd4c04 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.16 | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ [stack@undercloud-7 ~]$ ssh heat-admin.2.13 sudo cat /etc/sysconfig/iptables # Generated by iptables-save v1.4.21 on Wed Jan 31 20:57:17 2018 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m comment --comment "000 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -m comment --comment "001 accept all icmp" -m state --state NEW -j ACCEPT -A INPUT -i lo -m comment --comment "002 accept all to lo interface" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "003 accept ssh" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 123 -m comment --comment "105 ntp" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 4789 -m comment --comment "118 neutron vxlan networks" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 161 -m comment --comment "127 snmp" -m state --state NEW -j ACCEPT -A INPUT -p gre -m comment --comment "136 neutron gre networks" -j ACCEPT -A INPUT -p tcp -m multiport --dports 16514,49152:49215,5900:5999 -m comment --comment "200 nova_libvirt" -m state --state NEW -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT -m comment --comment "998 log all" -m state --state NEW -j LOG -A INPUT -m comment --comment "999 drop all" -m state --state NEW -j DROP -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Wed Jan 31 20:57:17 2018 [stack@undercloud-7 ~]$ ssh heat-admin.2.7 sudo cat /etc/sysconfig/iptables # Generated by iptables-save v1.4.21 on Wed Jan 31 20:59:38 2018 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m comment --comment "000 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -m comment --comment "001 accept all icmp" -m state --state NEW -j ACCEPT -A INPUT -i lo -m comment --comment "002 accept all to lo interface" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "003 accept ssh" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8042 -m comment --comment "100 aodh_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13042 -m comment --comment "100 aodh_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8777 -m comment --comment "100 ceilometer_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13777 -m comment --comment "100 ceilometer_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8776 -m comment --comment "100 cinder_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13776 -m comment --comment "100 cinder_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 9292 -m comment --comment "100 glance_api_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13292 -m comment --comment "100 glance_api_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 9191 -m comment --comment "100 glance_registry_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8041 -m comment --comment "100 gnocchi_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13041 -m comment --comment "100 gnocchi_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8004 -m comment --comment "100 heat_api_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13004 -m comment --comment "100 heat_api_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8000 -m comment --comment "100 heat_cfn_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13005 -m comment --comment "100 heat_cfn_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8003 -m comment --comment "100 heat_cloudwatch_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13003 -m comment --comment "100 heat_cloudwatch_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 35357 -m comment --comment "100 keystone_admin_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 5000 -m comment --comment "100 keystone_public_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13000 -m comment --comment "100 keystone_public_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 9696 -m comment --comment "100 neutron_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13696 -m comment --comment "100 neutron_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8775 -m comment --comment "100 nova_metadata_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 6080 -m comment --comment "100 nova_novncproxy_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13080 -m comment --comment "100 nova_novncproxy_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8774 -m comment --comment "100 nova_osapi_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13774 -m comment --comment "100 nova_osapi_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8080 -m comment --comment "100 swift_proxy_server_haproxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 13808 -m comment --comment "100 swift_proxy_server_haproxy_ssl" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 27019 -m comment --comment "101 mongodb_config" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 27018 -m comment --comment "102 mongodb_sharding" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 27017 -m comment --comment "103 mongod" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 873,3306,4444,4567,4568,9200 -m comment --comment "104 mysql galera" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 123 -m comment --comment "105 ntp" -m state --state NEW -j ACCEPT -A INPUT -p vrrp -m comment --comment "106 keepalived vrrp" -m state --state NEW -j ACCEPT -A INPUT -p vrrp -m comment --comment "106 neutron_l3 vrrp" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 1993 -m comment --comment "107 haproxy stats" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 6379,26379 -m comment --comment "108 redis" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 4369,5672,25672 -m comment --comment "109 rabbitmq" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 5000,13000,35357,13357 -m comment --comment "111 keystone" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 9292,13292 -m comment --comment "112 glance_api" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 9191 -m comment --comment "112 glance_registry" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8773,3773,8774,13774,8775 -m comment --comment "113 nova_api" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 9696,13696 -m comment --comment "114 neutron api" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 67 -m comment --comment "115 neutron dhcp input" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 4789 -m comment --comment "118 neutron vxlan networks" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8776,13776 -m comment --comment "119 cinder" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 3260 -m comment --comment "120 iscsi initiator" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 11211 -m comment --comment "121 memcached" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8080,13808 -m comment --comment "122 swift proxy" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 873,6000,6001,6002 -m comment --comment "123 swift storage" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8777,13777 -m comment --comment "124 ceilometer" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8004,13004 -m comment --comment "125 heat_api" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8000,13800 -m comment --comment "125 heat_cfn" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8003,13003 -m comment --comment "125 heat_cloudwatch" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 80,443 -m comment --comment "126 horizon" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 161 -m comment --comment "127 snmp" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8042,13042 -m comment --comment "128 aodh-api" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 8041,13041 -m comment --comment "129 gnocchi-api" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 2224,3121,21064 -m comment --comment "130 pacemaker tcp" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 5405 -m comment --comment "131 pacemaker udp" -m state --state NEW -j ACCEPT -A INPUT -p gre -m comment --comment "136 neutron gre networks" -j ACCEPT -A INPUT -p tcp -m multiport --dports 6080,13080 -m comment --comment "137 nova_vnc_proxy" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 8125 -m comment --comment "140 gnocchi-statsd" -m state --state NEW -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT -m comment --comment "998 log all" -m state --state NEW -j LOG -A INPUT -m comment --comment "999 drop all" -m state --state NEW -j DROP -A FORWARD -j REJECT --reject-with icmp-host-prohibited -A OUTPUT -p udp -m multiport --dports 68 -m comment --comment "116 neutron dhcp output" -m state --state NEW -j ACCEPT COMMIT # Completed on Wed Jan 31 20:59:38 2018 [stack@undercloud-7 ~]$ ~~~ Changing the templates: ~~~ [stack@undercloud-7 ~]$ diff templates/network-environment.yaml{.bck,} 51,62c51,62 < #ExtraConfig: < # neutron::agents::ml2::ovs::vxlan_udp_port: 57083 < #controllerExtraConfig: < # tripleo.neutron_ovs_agent.firewall_rules: < # '118 neutron vxlan networks': < # proto: 'udp' < # dport: 57083 < #NovaComputeExtraConfig: < # tripleo.neutron_ovs_agent.firewall_rules: < # '118 neutron vxlan networks': < # proto: 'udp' < # dport: 57083 --- > ExtraConfig: > neutron::agents::ml2::ovs::vxlan_udp_port: 57083 > controllerExtraConfig: > tripleo.neutron_ovs_agent.firewall_rules: > '118 neutron vxlan networks': > proto: 'udp' > dport: 57083 > NovaComputeExtraConfig: > tripleo.neutron_ovs_agent.firewall_rules: > '118 neutron vxlan networks': > proto: 'udp' > dport: 57083 ~~~ Starting the deploy: ~~~ [stack@undercloud-7 ~]$ templates/deploy.sh control_scale=3, compute_scale=2, ceph_scale=0 (...) 2018-02-02 19:17:01Z [AllNodesDeploySteps]: UPDATE_COMPLETE state changed 2018-02-02 19:17:13Z [overcloud]: UPDATE_COMPLETE Stack UPDATE completed successfully Stack overcloud UPDATE_COMPLETE Overcloud Endpoint: http://10.0.0.4:5000/v2.0 Overcloud Deployed ~~~ After the update, we can see that in iptables-save took place which save neutron rules into /etc/sysconfig/iptables ~~~ [stack@undercloud-7 ~]$ ssh heat-admin.2.13 sudo cat /etc/sysconfig/iptables # Generated by iptables-save v1.4.21 on Fri Feb 2 18:55:04 2018 *raw :PREROUTING ACCEPT [45423:12700384] :OUTPUT ACCEPT [30140:4089099] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-PREROUTING - [0:0] -A PREROUTING -j neutron-openvswi-PREROUTING -A OUTPUT -j neutron-openvswi-OUTPUT -A neutron-openvswi-PREROUTING -m physdev --physdev-in qvbbb9657ae-c7 -m comment --comment "Set zone for 57ae-c79d-477d-8aa9-13b95720534d" -j CT --zone 1 -A neutron-openvswi-PREROUTING -i qvbbb9657ae-c7 -m comment --comment "Set zone for 57ae-c79d-477d-8aa9-13b95720534d" -j CT --zone 1 -A neutron-openvswi-PREROUTING -m physdev --physdev-in tapbb9657ae-c7 -m comment --comment "Set zone for 57ae-c79d-477d-8aa9-13b95720534d" -j CT --zone 1 COMMIT # Completed on Fri Feb 2 18:55:04 2018 # Generated by iptables-save v1.4.21 on Fri Feb 2 18:55:04 2018 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :neutron-filter-top - [0:0] :neutron-openvswi-FORWARD - [0:0] :neutron-openvswi-INPUT - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-ibb9657ae-c - [0:0] :neutron-openvswi-local - [0:0] :neutron-openvswi-obb9657ae-c - [0:0] :neutron-openvswi-sbb9657ae-c - [0:0] :neutron-openvswi-sg-chain - [0:0] :neutron-openvswi-sg-fallback - [0:0] -A INPUT -j neutron-openvswi-INPUT -A INPUT -m comment --comment "000 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -m comment --comment "001 accept all icmp" -m state --state NEW -j ACCEPT -A INPUT -i lo -m comment --comment "002 accept all to lo interface" -m state --state NEW -j ACCEPT -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "003 accept ssh" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 123 -m comment --comment "105 ntp" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 57083 -m comment --comment "118 neutron vxlan networks" -m state --state NEW -j ACCEPT -A INPUT -p udp -m multiport --dports 161 -m comment --comment "127 snmp" -m state --state NEW -j ACCEPT -A INPUT -p gre -m comment --comment "136 neutron gre networks" -j ACCEPT -A INPUT -p tcp -m multiport --dports 16514,49152:49215,5900:5999 -m comment --comment "200 nova_libvirt" -m state --state NEW -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT -m comment --comment "998 log all" -m state --state NEW -j LOG -A INPUT -m comment --comment "999 drop all" -m state --state NEW -j DROP -A FORWARD -j neutron-filter-top -A FORWARD -j neutron-openvswi-FORWARD -A FORWARD -j REJECT --reject-with icmp-host-prohibited -A OUTPUT -j neutron-filter-top -A OUTPUT -j neutron-openvswi-OUTPUT -A neutron-filter-top -j neutron-openvswi-local -A neutron-openvswi-FORWARD -m physdev --physdev-out tapbb9657ae-c7 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-openvswi-sg-chain -A neutron-openvswi-FORWARD -m physdev --physdev-in tapbb9657ae-c7 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-openvswi-sg-chain -A neutron-openvswi-INPUT -m physdev --physdev-in tapbb9657ae-c7 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-openvswi-obb9657ae-c -A neutron-openvswi-ibb9657ae-c -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-openvswi-ibb9657ae-c -s 192.168.0.4/32 -p udp -m udp --sport 67 -m udp --dport 68 -j RETURN -A neutron-openvswi-ibb9657ae-c -s 192.168.0.2/32 -p udp -m udp --sport 67 -m udp --dport 68 -j RETURN -A neutron-openvswi-ibb9657ae-c -s 192.168.0.3/32 -p udp -m udp --sport 67 -m udp --dport 68 -j RETURN -A neutron-openvswi-ibb9657ae-c -p tcp -m tcp -m multiport --dports 1:65535 -j RETURN -A neutron-openvswi-ibb9657ae-c -m set --match-set NIPv42c5747fb-cb6f-4cec-acf1- src -j RETURN -A neutron-openvswi-ibb9657ae-c -p udp -m udp -m multiport --dports 1:65535 -j RETURN -A neutron-openvswi-ibb9657ae-c -p icmp -j RETURN -A neutron-openvswi-ibb9657ae-c -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-openvswi-ibb9657ae-c -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback -A neutron-openvswi-obb9657ae-c -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-openvswi-obb9657ae-c -j neutron-openvswi-sbb9657ae-c -A neutron-openvswi-obb9657ae-c -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-openvswi-obb9657ae-c -p udp -m udp --sport 67 -m udp --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-openvswi-obb9657ae-c -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-openvswi-obb9657ae-c -j RETURN -A neutron-openvswi-obb9657ae-c -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-openvswi-obb9657ae-c -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback -A neutron-openvswi-sbb9657ae-c -s 192.168.0.11/32 -m mac --mac-source FA:16:3E:F2:96:88 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-openvswi-sbb9657ae-c -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-openvswi-sg-chain -m physdev --physdev-out tapbb9657ae-c7 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-ibb9657ae-c -A neutron-openvswi-sg-chain -m physdev --physdev-in tapbb9657ae-c7 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-obb9657ae-c -A neutron-openvswi-sg-chain -j ACCEPT -A neutron-openvswi-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP COMMIT # Completed on Fri Feb 2 18:55:04 2018 [stack@undercloud-7 ~]$ ~~~ Delayed issues with the above: Look at the saved iptables rules: `-A neutron-openvswi-ibb9657ae-c -m set --match-set NIPv42c5747fb-cb6f-4cec-acf1- src -j RETURN` ~~~ [root@overcloud-compute-1 ~]# ipset list Name: NIPv42c5747fb-cb6f-4cec-acf1- Type: hash:net Revision: 3 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 16816 References: 1 Members: 192.168.0.11 Name: NIPv62c5747fb-cb6f-4cec-acf1- Type: hash:net Revision: 3 Header: family inet6 hashsize 1024 maxelem 65536 Size in memory: 17632 References: 1 Members: 2000:192:168:1:f816:3eff:fef2:9688 ~~~ Deleting the instance: ~~~ [stack@undercloud-7 ~]$ nova delete rhel-test1 Request to delete server rhel-test1 has been accepted. ~~~ That will delete the ipset, but it's still in the saved iptables rules! ~~~ [stack@undercloud-7 ~]$ ssh heat-admin.2.13 Last login: Fri Feb 2 19:26:21 2018 from gateway [heat-admin@overcloud-compute-1 ~]$ sudo -i [root@overcloud-compute-1 ~]# ipset list [root@overcloud-compute-1 ~]# ~~~ iptables cannot be restarted (not on a reboot, and not on a Director stack update): ~~~ [root@overcloud-compute-1 ~]# systemctl restart iptables Job for iptables.service failed because the control process exited with error code. See "systemctl status iptables.service" and "journalctl -xe" for details. [root@overcloud-compute-1 ~]# systemctl status iptables -l ● iptables.service - IPv4 firewall with iptables Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2018-02-02 19:27:40 UTC; 5s ago Process: 290356 ExecStop=/usr/libexec/iptables/iptables.init stop (code=exited, status=0/SUCCESS) Process: 290644 ExecStart=/usr/libexec/iptables/iptables.init start (code=exited, status=1/FAILURE) Main PID: 290644 (code=exited, status=1/FAILURE) Feb 02 19:27:40 overcloud-compute-1 systemd[1]: Starting IPv4 firewall with iptables... Feb 02 19:27:40 overcloud-compute-1 iptables.init[290644]: iptables: Applying firewall rules: iptables-restore v1.4.21: Set NIPv42c5747fb-cb6f-4cec-acf1- doesn't exist. Feb 02 19:27:40 overcloud-compute-1 iptables.init[290644]: Error occurred at line: 60 Feb 02 19:27:40 overcloud-compute-1 iptables.init[290644]: Try `iptables-restore -h' or 'iptables-restore --help' for more information. Feb 02 19:27:40 overcloud-compute-1 iptables.init[290644]: [FAILED] Feb 02 19:27:40 overcloud-compute-1 systemd[1]: iptables.service: main process exited, code=exited, status=1/FAILURE Feb 02 19:27:40 overcloud-compute-1 systemd[1]: Failed to start IPv4 firewall with iptables. Feb 02 19:27:40 overcloud-compute-1 systemd[1]: Unit iptables.service entered failed state. Feb 02 19:27:40 overcloud-compute-1 systemd[1]: iptables.service failed. [root@overcloud-compute-1 ~]# ~~~ One can work around this by deleting the failing line, but this of course is only a workaround. We should never have saved neutron iptables rules into /etc/sysconfig/iptables - Andreas
Could you please confirm PurgeFirewallRules is not set to True in your environment? Looking at the code: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/tripleo-firewall.yaml#L38 https://github.com/openstack/puppet-tripleo/blob/master/manifests/firewall.pp#L57 It's disabled by default, so firewall rules shouldn't be purged by the puppetlabs-firewall module unless I missed something.
Nevermind Andreas. I talked with Alex Schultz and here's our conclusion: - Neutron manages IPtables rules and make them consistent with iptables-save: https://github.com/openstack/neutron/blob/cbee0f9f88ff34f70ff19590471b5405e06ff2a9/neutron/agent/linux/iptables_manager.py#L548 - puppetlabs-firewall manages TripleO firewall rules but it could be any operator managing IPtables persistent rules while Neutron is running, the result would be the same, we would have conflict like it happens in this bug. One workaround could be to override https://github.com/openstack/neutron/blob/cbee0f9f88ff34f70ff19590471b5405e06ff2a9/etc/neutron/rootwrap.d/iptables-firewall.filters#L13 and change the content to be: iptables-save: CommandFilter, /bin/true, root ip6tables-save: CommandFilter, /bin/true, root (not tested though).
Why does tripleo dump iptables-save result into the file? Is it to store rules it created during deployment to give access to API and other services for the next reboot? If so, I don't think iptables-save is the right approach. Instead of blindly dumping all rules there are at a particular moment, we should fill in the sysconfig file with exact rules needed for those services, and nothing else.
To comment on #7, no it's completely wrong to override rootwrap filters like that. It will break neutron firewall because the way we use iptables-save is that we pipe its stdout as stdin into iptables-restore (after modifications). We never dump result to any persisted files. The wrongdoer here is, puppetlabs-firewall, or operator, that decides to save all rules to sysconfig while neutron agents are running. Those rules are not supposed to persist. If you need SOME rules in sysconfig, then explicitly write those rules there, and only those rules.
*** Bug 1528605 has been marked as a duplicate of this bug. ***
I totally agree with Ihar. ~~~ - Neutron manages IPtables rules and make them consistent with iptables-save: https://github.com/openstack/neutron/blob/cbee0f9f88ff34f70ff19590471b5405e06ff2a9/neutron/agent/linux/iptables_manager.py#L548 ~~~ This statement is 100% wrong. neutron makes no rule whatsoever consistent. All rules are written temporarily. On a node reboot, those rules are lost. The code that you are referring to above does this: ~~~ This will create a diff between the rules from the previous runs and replace them with the current set of rules. This happens atomically, thanks to iptables-restore. ~~~ I only half-heartedly read through the code, but to me it doesn't look as if it was saving anything to file. It's using iptables-save / iptables-restore to create and apply a dynamical list of changes: ~~~ if changes: # if there are changes to the table, we put on the header # and footer that iptables-save needs commands += (['# Generated by iptables_manager'] + ['*%s' % table_name] + changes + ['COMMIT', '# Completed by iptables_manager']) if not commands: continue all_commands += commands # always end with a new line commands.append('') args = ['%s-restore' % (cmd,), '-n'] if self.namespace: args = ['ip', 'netns', 'exec', self.namespace] + args err = self._run_restore(args, commands) if err: self._log_restore_err(err, commands) raise err ~~~ What Director (and if the above assumptions are correct, puppetlabs-firewall) does, is go in and blindly assume that it's completely save to take all non-persistent firewall rules from within iptables and to write them into /etc/sysconfig/iptables. And that's absolutely not o.k., because it breaks the way neutron works, which is what I outlined above when I opened this ticket. - Andreas
OMG, I'm tired .. consistent, not persistent. So yes, it makes them consistent with iptables-save / -restore, but it does not make them persistent. We're on the same page then. Sorry for misreading your statement!
Your conclusion however is wrong. The wrongdoer is indeed the firewall module. neutron is doing everything right. So a change here: https://github.com/openstack/neutron/blob/cbee0f9f88ff34f70ff19590471b5405e06ff2a9/etc/neutron/rootwrap.d/iptables-firewall.filters#L13 ...is the wrong approach
(In reply to Ihar Hrachyshka from comment #8) > Why does tripleo dump iptables-save result into the file? Is it to store > rules it created during deployment to give access to API and other services > for the next reboot? If so, I don't think iptables-save is the right > approach. This is what TripleO does: https://github.com/puppetlabs/puppetlabs-firewall/blob/d714f122d19155afa00b2b990a9267b422e5946d/lib/puppet/util/firewall.rb#L168 /sbin/service iptables save > Instead of blindly dumping all rules there are at a particular > moment, we should fill in the sysconfig file with exact rules needed for > those services, and nothing else. How do you do it? We can't prevent operators to add their firewall rules and make them consistent on day 2 operations. > To comment on #7, no it's completely wrong to override rootwrap filters like > that. (...) ack, it makes sense. > The wrongdoer here is, puppetlabs-firewall, or operator, that decides to save > all rules to sysconfig while neutron agents are running. Those rules are not > supposed to persist. If you need SOME rules in sysconfig, then explicitly > write those rules there, and only those rules. Like I said, the puppet module is really not filtering anything now. We would have to think about something like this, but for sure we don't have it now. Also mentioned on IRC by Alex, we can't set PurgeFirewallRules to True because it would kill Neutron rules and we have no mechanism to trigger a neutron restart when firewall rules would be purged, because we run containers and not systemd services on the host.
I disagree the fix will belong to neutron. Neutron has no idea that you dumped no longer valid rules and later will try to restore them when resources they reference to, like ipsets, are long gone because they are not needed from neutron data plane perspective. I would argue "service iptables save" is just not compatible with how neutron uses iptables, and there is nothing we can do about it except documenting it, and making tripleo not do what it does right now - issuing the command in the middle of operation. As for customers doing it with PostConfig, they should get guidance on how to do it properly. I imagine RHEL has better mechanism to persistently configure firewall than just dumping current rules into a file. Whatever this mechanism is, it should be used instead.
To give an example, suggesting neutron must be compatible with 'service iptables save' would be similar to suggesting neutron must continue working correctly while operator adds and deletes random rules by calling to iptables directly. Neither the former nor the latter makes much sense to me, and we should just discourage issuing those tools when neutron agents are deployed. Is there any technical problem with making tripleo persist just the needed iptables rules?
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/manual_installation_procedures/sect-configuring_the_firewall_for_telemtry_service https://access.redhat.com/articles/9014 https://access.redhat.com/solutions/367653 These 3 links recommend to use `service iptables save`. We like it or not, this is how our customers are recommended to do now. For your question about TripleO & "needed" persistent rules: yes for now puppetlabs-firewall isn't able to do that. I think it would be a major change, and I'm even not sure it would be accepted by Puppetlabs. If that happens, we would have to carry some custom functions in puppet-tripleo. Note when we switch to Ansible or whatever tool in the future, we would have to rewrite it in $newframework...
I'll investigate further but I might have found a way to ignore some rules: https://github.com/puppetlabs/puppetlabs-firewall/blob/master/README.markdown#parameters-1 I'll keep digging...
Another option is to overload the utility: https://github.com/puppetlabs/puppetlabs-firewall/blob/d714f122d19155afa00b2b990a9267b422e5946d/lib/puppet/util/firewall.rb#L168 And grep -v "neutron-", so neutron-managed rules would be ignored by Puppet.
> Instead of blindly dumping all rules there are at a particular > moment, we should fill in the sysconfig file with exact rules needed for > those services, and nothing else. How do you do it? We can't prevent operators to add their firewall rules and make them consistent on day 2 operations. +++ Of course we can. We put it into documentation and tell them not to save the rules or edit /etc/sysconfig/iptables manually instead. Indeed, we do not support *any* manual change to a file on an overcloud node, so day 2 firewall changes via iptables, via iptables-save > /etc/sysconfig/iptables or via manual modification of /etc/susconfig/iptables are unsupported and users live with the fact that these will be lost on the next `openstack overcloud deploy`. What's important, though, is that Director doesn't mess up things, and right now it's doing exactly what we have been telling users not to do since OSP 7 at least (probably even prior): saving dynamic neutron firewall rules into /etc/sysconfig/iptables
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/manual_installation_procedures/sect-configuring_the_firewall_for_telemtry_service https://access.redhat.com/articles/9014 https://access.redhat.com/solutions/367653 These 3 links recommend to use `service iptables save`. We like it or not, this is how our customers are recommended to do now. +++ Link 1) is for the manual installation procedure, a procedure that was never supported in any version from 7 to 12 and since 12 the guide is even completely retired. Link 2) is for RHEL, not OSP Link 3) is for RHEL (see the environment section) +++ The solutions for OSP are here https://access.redhat.com/solutions/2678381 and here https://access.redhat.com/solutions/3172881 +++ Additionally we all know that our own documentation may contain bugs from time to time or contradicting instructions. The KCS solutions are all written by our support representatives in the field, and my not always be 100% accurate for as much as we try.
Still looking at how to fix it in puppet. The "ignore" parameter in puppetlabs-firewall is implemented for each chain (e.g. "FORWARD:filter:IPv4", which means chains have to be predictable, which isn't the case in Neutron (e.g. "neutron-openvswi-ibb9657ae-c"). See https://github.com/puppetlabs/puppetlabs-firewall/blob/4d602914b27bffbd050b44f9957ba754761f21ff/lib/puppet/type/firewallchain.rb#L231 Which is where the rule isn't manage if matching the regex in ignore. The problem is that chain has to be managed by Puppet. So we need to figure out how we can tell the puppet module to unmanage for all chains, and not only the ones we declare in the catalog.
I'm unsure if that will work but I'm testing something with: https://review.openstack.org/#/c/541852/ https://review.openstack.org/#/c/541849/ I'll keep this bug updated with my findings.
Update: see https://review.openstack.org/541849 and https://review.openstack.org/542544
I'm having issues to reproduce your dependency error on Queens, see the logs on the compute: https://logs.rdoproject.org/49/541849/20/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z858ae5bb9aea41208c31d072659ef51b/overcloud-novacompute-0/var/log/journal.txt.gz#_Feb_15_07_33_51 I'll try to reproduce on Newton...
ok Alex and I realized https://review.openstack.org/#/c/545110/ is also needed (was not backported to Newton). We'll see if that helps. More update to come.
Verified on puddle 2018-05-09.2 [stack@undercloud-0 ~]$ rpm -q puppet-tripleo puppet-tripleo-5.6.8-3.el7ost.noarch hostlist=`nova list | awk '/ACTIVE/ {print $(NF-1)}' | awk -F '=' '{print $NF}' | tr '\n' ' '` for i in $hostlist ; do ssh heat-admin@$i "hostname; sudo grep -i \"neutron-\" /etc/sysconfig/iptables /etc/sysconfig/ip6tables"; done ceph-0 ceph-1 ceph-2 compute-0 compute-1 controller-0 controller-1 controller-2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1593