Description of problem:

When customizing default firewall rules in RHOSP 17.1, the legacy rules that are replaced by custom rules are not removed from the systems.


Version-Release number of selected component (if applicable):
latest 17.1 as of the date of reporting

How reproducible:
Always reproducible when customizing a default rule

Steps to Reproduce:
1. Add an environment file to customize firewall rule "130 pacemaker tcp":
~~~
[stack.lab ~]$ cat templates/firewall.yaml 
parameter_defaults:
  ControllerParameters:
    ExtraFirewallRules:
      '130 pacemaker tcp':
        dport:
          - 2224
          - 3121
          - 21064
        source: 192.168.24.0/24
~~~
(note that the only difference between this rule and the one added by default is the "source" field, which does not exist in the default rules).

2. Add new template to answers file or deployment script
3. Deploy overcloud

Actual results:

Overcloud controller gets both the custom rules and the default ones:
~~~
[root@overcloud-controller-0 ~]# iptables -L |grep "130 pacemaker"
ACCEPT     tcp  --  192.168.24.0/24      anywhere             tcp dpt:21064 ctstate NEW /* 130 pacemaker tcp ipv4 */
ACCEPT     tcp  --  192.168.24.0/24      anywhere             tcp dpt:pcmk-remote ctstate NEW /* 130 pacemaker tcp ipv4 */
ACCEPT     tcp  --  192.168.24.0/24      anywhere             tcp dpt:efi-mg ctstate NEW /* 130 pacemaker tcp ipv4 */
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:21064 ctstate NEW /* 130 pacemaker tcp ipv4 */
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pcmk-remote ctstate NEW /* 130 pacemaker tcp ipv4 */
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:efi-mg ctstate NEW /* 130 pacemaker tcp ipv4 */
[root@overcloud-controller-0 ~]# 
[root@overcloud-controller-0 ~]# cat -n /etc/sysconfig/iptables | grep "130 pacemaker"
    24	-A INPUT -s 192.168.24.0/24 -p tcp -m tcp --dport 21064 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
    25	-A INPUT -s 192.168.24.0/24 -p tcp -m tcp --dport 3121 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
    26	-A INPUT -s 192.168.24.0/24 -p tcp -m tcp --dport 2224 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
    27	-A INPUT -p tcp -m tcp --dport 21064 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
    28	-A INPUT -p tcp -m tcp --dport 3121 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
    29	-A INPUT -p tcp -m tcp --dport 2224 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
[root@overcloud-controller-0 ~]# 
~~~

Expected results:

Only the first 3 lines should be present.

Additional info:

This seems to be a cleanup issue rather than ansible_firewall passing both sets of rules each time overcloud deploy runs. If I remove lines 27-29 from /etc/sysconfig/iptables and run an overcloud deploy, those lines are not added back. More over, config-download shows that Ansible only has the new set of rules, not the old ones:
~~~
(undercloud) [stack.lab ~]$ egrep -r -A5 "130 pacemaker" config-download/overcloud/
config-download/overcloud/Controller/firewall_rules.yaml:130 pacemaker tcp:
config-download/overcloud/Controller/firewall_rules.yaml-  dport:
config-download/overcloud/Controller/firewall_rules.yaml-  - 2224
config-download/overcloud/Controller/firewall_rules.yaml-  - 3121
config-download/overcloud/Controller/firewall_rules.yaml-  - 21064
config-download/overcloud/Controller/firewall_rules.yaml-  source: 192.168.24.0/24
--
config-download/overcloud/group_vars/Controller:  130 pacemaker tcp:
config-download/overcloud/group_vars/Controller-    dport:
config-download/overcloud/group_vars/Controller-    - 2224
config-download/overcloud/group_vars/Controller-    - 3121
config-download/overcloud/group_vars/Controller-    - 21064
config-download/overcloud/group_vars/Controller-    source: 192.168.24.0/24
(undercloud) [stack.lab ~]$ 
~~~