Bug 1064058

Summary: attempt to update firewall caused existing rules to be wipped
Product: Red Hat OpenStack Reporter: Steve Reichard <sreichar>
Component: openstack-puppet-modulesAssignee: RHOS Maint <rhos-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Ami Jeain <ajeain>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.0CC: aortega, cpelland, derekh, dkranz, hbrock, jguiditt, mmagr, morazi, rhos-maint, yeylon
Target Milestone: z4   
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-27 10:54:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1040649    
Attachments:
Description Flags
iptables.out (step2)
none
iptables.out (step4)
none
iptables.out (manual addition) none

Description Steve Reichard 2014-02-11 22:16:19 UTC 
Description of problem:

In BZ 1064056, it was noticed additional ports needed to be opened.
When lokkit was used to open the necessary port, the existing entries
were wiped.  My installation was based on the Nova networking host groups.

Of course when puppet ran next, it restored the original values and wiped the added.


Version-Release number of selected component (if applicable):


[root@rhos-foreman ~]#  yum list installed | grep -e foreman -e puppet
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
foreman.noarch                    1.3.0.2-1.el6sat   @rhel-x86_64-server-6-ost-4
foreman-installer.noarch          1:1.3.0-1.el6sat   @rhel-x86_64-server-6-ost-4
foreman-mysql.noarch              1.3.0.2-1.el6sat   @rhel-x86_64-server-6-ost-4
foreman-mysql2.noarch             1.3.0.2-1.el6sat   @rhel-x86_64-server-6-ost-4
foreman-proxy.noarch              1.3.0-3.el6sat     @rhel-x86_64-server-6-ost-4
foreman-selinux.noarch            1.3.0-1.el6sat     @rhel-x86_64-server-6-ost-4
openstack-foreman-installer.noarch
packstack-modules-puppet.noarch   2013.2.1-0.22.dev956.el6ost
puppet.noarch                     3.2.4-3.el6_5      @rhel-x86_64-server-6-ost-4
puppet-server.noarch              3.2.4-3.el6_5      @rhel-x86_64-server-6-ost-4
ruby193-rubygem-foreman_openstack_simplify.noarch
rubygem-foreman_api.noarch        0.1.6-1.el6sat     @rhel-x86_64-server-6-ost-4
[root@rhos-foreman ~]# 



How reproducible:

Unsure

Steps to Reproduce:
1. Install nove netowrking using foreman
2. iptables -nvL
3. on controller issue "lokkit --port 8004:tcp"
3. iptables -nvL

Actual results:


Expected results:


Additional info:
Comment 2 Jason Guiditta 2014-02-14 21:20:56 UTC 
We can add a firewall rule for this
Comment 3 Hugh Brock 2014-03-05 17:46:38 UTC 
Moved this out to A4, SPR says important but not critical for A3.
Comment 4 Martin Magr 2014-03-27 10:54:51 UTC 
So lokkit wipes out firewall rules and it's problem in Puppet? Seriously? I think it's lokkit's issue.
Comment 5 Mike Orazi 2014-03-27 20:04:30 UTC 
Created attachment 879631 [details]
iptables.out (step2)
Comment 6 Mike Orazi 2014-03-27 20:05:52 UTC 
Created attachment 879643 [details]
iptables.out (step4)
Comment 7 Mike Orazi 2014-03-27 20:06:27 UTC 
Created attachment 879644 [details]
iptables.out (manual addition)
Comment 8 Mike Orazi 2014-03-27 20:13:44 UTC 
After this got closed, there was some discussion and I wanted to confirm the present behaviour because I seemed to recall that there was an old issue that involved iptables not persisting correctly that had been fixed.

I follow this procedure:

1. Install nove netowrking using foreman
2. iptables -nvL 
3. on controller issue "lokkit --port 8004:tcp"
4. iptables -nvL 

and noted that the output from iptables was no longer wiped out, but it did NOT contain any mention of 8004 accepting TCP connections.

So I additionally did:
5.  iptables -A INPUT -p tcp --dport 8004 -j ACCEPT
6.  iptables -nvL  

and noted that manually adding the rule did result in the expected rule being added.

I'm going to change the status to CLOSED_CURRENTRELEASE because I do think a change in the puppet modules changed this behavior.  I'm not entirely clear why lokkit failed to add the rule as expected but I'm not super familiar with lokkit so I was happy to just ensure that the rules remained intact & that it was possible to add a rule using iptables manually.
Comment 9 Steve Reichard 2014-03-31 14:04:51 UTC 
My results are not as you describe:

- I display the running iptables
- I cat the save iptables
- I use lokkit to open prot 8004
 + running is only 22,443, and 8004
 + same with files
- do a puppet check-in
 + now original rules are inplace along with 8004

so we have issue, which maybe we can document saying run puppet after lokkit.


[root@spr4 ~(keystone_admin)]$ iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
nova-api-INPUT  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 80,443,3260,3306,5000,35357,5672,8773,8774,8775,8776,8777,9292,6080 /* 001 controller incoming */ 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 8080 /* 001 swift proxy incoming */ 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 443 /* 002 ssl controller incoming */ 
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:53 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:53 
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:67 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:67 
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443 
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
nova-filter-top  all  --  0.0.0.0/0            0.0.0.0/0           
nova-api-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24    state RELATED,ESTABLISHED 
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
nova-filter-top  all  --  0.0.0.0/0            0.0.0.0/0           
nova-api-OUTPUT  all  --  0.0.0.0/0            0.0.0.0/0           

Chain nova-api-FORWARD (1 references)
target     prot opt source               destination         

Chain nova-api-INPUT (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            10.16.139.14        tcp dpt:8775 

Chain nova-api-OUTPUT (1 references)
target     prot opt source               destination         

Chain nova-api-local (1 references)
target     prot opt source               destination         

Chain nova-filter-top (2 references)
target     prot opt source               destination         
nova-api-local  all  --  0.0.0.0/0            0.0.0.0/0           
[root@spr4 ~(keystone_admin)]$ cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Wed Mar 26 19:08:16 2014
*nat
:PREROUTING ACCEPT [3452:1154567]
:POSTROUTING ACCEPT [2314:171157]
:OUTPUT ACCEPT [2314:171157]
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE 
COMMIT
# Completed on Wed Mar 26 19:08:16 2014
# Generated by iptables-save v1.4.7 on Wed Mar 26 19:08:16 2014
*mangle
:PREROUTING ACCEPT [30181:97126630]
:INPUT ACCEPT [28305:96524172]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [24691:10966042]
:POSTROUTING ACCEPT [24691:10966042]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 
COMMIT
# Completed on Wed Mar 26 19:08:16 2014
# Generated by iptables-save v1.4.7 on Wed Mar 26 19:08:16 2014
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:164]
-A INPUT -p tcp -m multiport --dports 80,443,3260,3306,5000,35357,5672,8773,8774,8775,8776,8777,9292,6080 -m comment --comment "001 controller incoming" -j ACCEPT 
-A INPUT -p tcp -m multiport --dports 8080 -m comment --comment "001 swift proxy incoming" -j ACCEPT 
-A INPUT -p tcp -m multiport --dports 443 -m comment --comment "002 ssl controller incoming" -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT 
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Wed Mar 26 19:08:16 2014
[root@spr4 ~(keystone_admin)]$ lokkit -p 8004:tcp
[root@spr4 ~(keystone_admin)]$ cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8004 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
[root@spr4 ~(keystone_admin)]$ iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:8004 
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
[root@spr4 ~(keystone_admin)]$  puppet agent -t
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/network.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hamysql_active_node.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ipa_client_configured.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/netns_support.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
Info: Caching catalog for spr4.cloud.lab.eng.bos.redhat.com
Info: Applying configuration version '1396024271'
Notice: /Firewall[001 swift proxy incoming]/ensure: created
Notice: An attempt has been made below to automatically apply your custom
    settings to mongodb::server. Please verify this works in a safe test
    environment.
Notice: /Stage[main]/Mongodb/Notify[An attempt has been made below to automatically apply your custom
    settings to mongodb::server. Please verify this works in a safe test
    environment.]/message: defined 'message' as 'An attempt has been made below to automatically apply your custom
    settings to mongodb::server. Please verify this works in a safe test
    environment.'
Notice: /Firewall[002 ssl controller incoming]/ensure: created
Notice: /File[/etc/httpd/conf/httpd.conf]/content: 
--- /etc/httpd/conf/httpd.conf	2014-03-28 12:25:13.681934446 -0400
+++ /tmp/puppet-file20140328-11690-z63m8j-0	2014-03-28 12:36:47.968868285 -0400
@@ -45,4 +45,3 @@
 
 Include /etc/httpd/conf.d/*.conf
 
-#Listen 0.0.0.0:80

Info: FileBucket got a duplicate file {md5}b740ae1e515d05c9e75a7a77918b4526
Info: /File[/etc/httpd/conf/httpd.conf]: Filebucketed /etc/httpd/conf/httpd.conf to puppet with sum b740ae1e515d05c9e75a7a77918b4526
Notice: /File[/etc/httpd/conf/httpd.conf]/content: content changed '{md5}b740ae1e515d05c9e75a7a77918b4526' to '{md5}0c71013dfa1d8bca5c7bd49db99854da'
Info: /File[/etc/httpd/conf/httpd.conf]: Scheduling refresh of Class[Apache::Service]
Notice: /Firewall[001 controller incoming]/ensure: created
Notice: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]/value: value changed '63227edc0acc68aaa11bb5474cb54c43' to '%ENCRYPTION_KEY%'
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-engine]
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api-cfn]
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api-cloudwatch]
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api]
Notice: /Stage[main]/Horizon/File_line[httpd_listen_on_bind_address_80]/ensure: created
Info: /Stage[main]/Horizon/File_line[httpd_listen_on_bind_address_80]: Scheduling refresh of Service[httpd]
Notice: /Stage[main]/Quickstack::Controller_common/File_line[undo_httpd_listen_on_bind_address_80]/ensure: created
Info: /Stage[main]/Quickstack::Controller_common/File_line[undo_httpd_listen_on_bind_address_80]: Scheduling refresh of Service[httpd]
Info: Class[Apache::Service]: Scheduling refresh of Service[httpd]
Notice: /Stage[main]/Apache::Service/Service[httpd]: Triggered 'refresh' from 3 events
Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Heat::Api/Service[heat-api]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Heat::Engine/Exec[heat-encryption-key-replacement]/returns: executed successfully
Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 1 events
Notice: Finished catalog run in 59.72 seconds
[root@spr4 ~(keystone_admin)]$  iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 80,443,3260,3306,5000,35357,5672,8773,8774,8775,8776,8777,9292,6080 /* 001 controller incoming */ 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 8080 /* 001 swift proxy incoming */ 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 443 /* 002 ssl controller incoming */ 
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:8004 
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
[root@spr4 ~(keystone_admin)]$ cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Fri Mar 28 12:36:49 2014
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:164]
-A INPUT -p tcp -m multiport --dports 80,443,3260,3306,5000,35357,5672,8773,8774,8775,8776,8777,9292,6080 -m comment --comment "001 controller incoming" -j ACCEPT 
-A INPUT -p tcp -m multiport --dports 8080 -m comment --comment "001 swift proxy incoming" -j ACCEPT 
-A INPUT -p tcp -m multiport --dports 443 -m comment --comment "002 ssl controller incoming" -j ACCEPT 
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8004 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Fri Mar 28 12:36:49 2014
[root@spr4 ~(keystone_admin)]$