Bug 1544211

this is fixed in bz1380141. the bug is unfortunately present in the package that is being uninstalled, so the fix will take effect only for packages that are upgraded from version that is the fixed one.

iptables-1.4.21-18.2.el7_4 is actually the fixed package version.
* Mon Sep 18 2017 Phil Sutter - 1.4.21-18.2
- Prevent iptables.service and ip6tables.service from running in parallel
  (RHBZ#1491963)
- Don't restart services upon upgrade (RHBZ#1491961)

Ok moving to Upgrades in TripleO , to see if some workaround can be proposed

Adding a note here: this bug is breaking data plane connectivity during minor update, update succeeds but overcloud instances become unreahable during update.

A minor clarification, it's not the overcloud nodes itself, but the vms created in the overcloud.
The way i noticed was the following:
- deploy overcloud
- create a external network with a range of fips
- create a vm, and attach a fip
- spin a continuous pingtest there
- trigger minor update, you will see that the dataplane connection breaks as soon as you update iptables

I was able to avoid the disruption by executing :
 iptables-save > /etc/sysconfig/iptables

See the content that is saved in compute-0 (where i saved the iptables), vs the content that is on compute-1 (that was not saved)

http://pastebin.test.redhat.com/554976

Hi Yolanda,

(In reply to Yolanda Robla from comment #6)
> I was able to avoid the disruption by executing :
>  iptables-save > /etc/sysconfig/iptables

Note that this will make all temporary iptables rules persist across reboots if iptables service is enabled on the system. Did you make sure this won't upset VM management services during startup due to unexpected iptables rules?

> See the content that is saved in compute-0 (where i saved the iptables), vs
> the content that is on compute-1 (that was not saved)
> 
> http://pastebin.test.redhat.com/554976

Assuming the issue is fixed in current iptables package and your workaround doesn't cause negative side-effects, are you fine with closing this ticket then?

Cheers, Phil

I added that as a hint on how it could be solved, but i don't like to use this workaround, as most of the customers that deployed older OSP will be affected. I'd like that this comment is considered by the upgrade team, and add the iptables-save or any other specific fix, as part of the minor update.

I tried this scenario on my environment and I was able to reproduce the issue:

on compute node iptables got updated:
[root@compute-0 ~]# grep iptables /var/log/yum.log 
Feb 13 09:25:40 Updated: iptables-1.4.21-18.2.el7_4.x86_64
Feb 13 09:26:38 Updated: iptables-services-1.4.21-18.2.el7_4.x86_64

FIP ping results during the entire run of the minor update:

[stack@undercloud-0 ~]$ tail -10 ping_results_201802131435 
[1518554529.301842] 64 bytes from 10.0.0.219: icmp_seq=4022 ttl=63 time=1.05 ms
[1518554530.302780] 64 bytes from 10.0.0.219: icmp_seq=4023 ttl=63 time=0.952 ms
[1518554531.303541] 64 bytes from 10.0.0.219: icmp_seq=4024 ttl=63 time=0.840 ms
[1518554532.303817] 64 bytes from 10.0.0.219: icmp_seq=4025 ttl=63 time=1.01 ms
[1518554533.304779] 64 bytes from 10.0.0.219: icmp_seq=4026 ttl=63 time=0.979 ms
[1518554534.304992] 64 bytes from 10.0.0.219: icmp_seq=4027 ttl=63 time=0.939 ms

--- 10.0.0.219 ping statistics ---
4027 packets transmitted, 3975 received, 1% packet loss, time 4029978ms
rtt min/avg/max/mdev = 0.445/0.929/4.717/0.203 ms

Difference is that I cannot see the 'Stopping IPv4 firewall with iptables' in /var/log/messages only

[root@compute-0 heat-admin]# grep 'IPv4 firewall with iptables' /var/log/messages  
Feb 13 09:53:31 localhost systemd: Starting IPv4 firewall with iptables...
Feb 13 09:53:31 localhost systemd: Started IPv4 firewall with iptables.

(In reply to Marius Cornea from comment #9)
> I tried this scenario on my environment and I was able to reproduce the
> issue:

^ Sorry, I was _not_ able to reproduce the issue.

What is your original iptables package? See mine:

    Actualizado iptables-1.4.21-18.0.1.el7.centos.x86_64          @?base
    Actualizar           1.4.21-18.2.el7_4.x86_64                 @updates

From 18.0.1 to 18.2 , can you check yours?

(In reply to Yolanda Robla from comment #11)
> What is your original iptables package? See mine:
> 
>     Actualizado iptables-1.4.21-18.0.1.el7.centos.x86_64          @?base
>     Actualizar           1.4.21-18.2.el7_4.x86_64                 @updates
> 
> From 18.0.1 to 18.2 , can you check yours?

Apologies for the delay. I was indeed able to reproduce this issue when using an older overcloud image(rhel 7.3):

updating from iptables-1.4.21-17.el7.x86_64 to iptables-1.4.21-18.2.el7_4.x86_64

As mentioned in comment #7 I think iptables-save will also save the temporary rules set by Neutron which we should avoid.

What can be the alternatives? Can we discriminate between the rules we want to save?

Hi Yolanda,

(In reply to Yolanda Robla from comment #8)
> I added that as a hint on how it could be solved, but i don't like to use
> this workaround, as most of the customers that deployed older OSP will be
> affected. I'd like that this comment is considered by the upgrade team, and
> add the iptables-save or any other specific fix, as part of the minor update.

I played around a bit with spec file triggers and it looks like I have a
solution:

| %triggerun -n iptables-services -- iptables-services < 1.4.21-20%{?dist}
| mkdir -p /var/run/iptables
| iptables-save > /var/run/iptables/iptables.save
| ip6tables-save > /var/run/iptables/ip6tables.save
| 
| %triggerpostun -n iptables-services -- iptables-services < 1.4.21-20%{?dist}
| iptables-restore -c -w 10 /var/run/iptables/iptables.save
| ip6tables-restore -c -w 10 /var/run/iptables/ip6tables.save
| rm -rf /var/run/iptables

These triggers run before and after deinstallation of the old
iptables-services package. As you can see I'm using your workaround, namely
saving the old ruleset and restoring it after the assumed service restart.
I've successfully tested this on a local machine for testing.

Given the importance of this ticket, I guess we might still get it into
RHEL7.5, for 7.4 then z-stream process applies of course.

Cheers, Phil

Very early implementation of a workaround during the upgrade based on the ovs one.
It fixes update *and* upgrade.

TODO: as it's rhel related we need to "upport" it to master.  The upgrade is using ansible from ocata on so it's completly different from ocata on.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1593