This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1268413 - security groups iptables can block legitimate traffic as INVALID
security groups iptables can block legitimate traffic as INVALID
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
6.0 (Juno)
x86_64 Linux
high Severity high
: async
: 6.0 (Juno)
Assigned To: Nir Magnezi
Alexander Stafeyev
: ZStream
Depends On:
Blocks: 1269849 1338971
  Show dependency treegraph
 
Reported: 2015-10-02 14:58 EDT by Benjamin Schmaus
Modified: 2016-05-23 14:42 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-neutron-2014.2.3-23.el7ost
Doc Type: Bug Fix
Doc Text:
Cause: The Linux iptables implementation of security groups includes a default rule to drop any INVALID packets. Before this fix, User-defined iptables rules were processed before the INVALID DROP rule. Consequence: Security groups iptables can block legitimate traffic, such as SCTP protocol, as INVALID. Fix: Process user-defined iptables rules before the INVALID DROP rule.
Story Points: ---
Clone Of:
: 1269849 1338971 (view as bug list)
Environment:
Last Closed: 2015-10-29 07:39:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
before_and_after.txt (3.73 KB, text/plain)
2015-10-06 07:41 EDT, Nir Magnezi
no flags Details
verification.txt (2.16 KB, text/plain)
2015-10-19 08:03 EDT, Nir Magnezi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1460741 None None None Never
OpenStack gerrit 218517 None None None Never
OpenStack gerrit 231928 None None None Never

  None (edit)
Description Benjamin Schmaus 2015-10-02 14:58:34 EDT
Description of problem:

The iptables implementation of security groups includes a default rule to drop any INVALID packets (according to the Linux connection state tracking system.) It looks like this:

-A neutron-openvswi-od0518220-e -m state --state INVALID -j DROP

This is placed near the top of the rule stack, before any security group rules added by the user. See:

https://github.com/openstack/neutron/blob/stable/kilo/neutron/agent/linux/iptables_firewall.py#L495
https://github.com/openstack/neutron/blob/stable/kilo/neutron/agent/linux/iptables_firewall.py#L506-L510

However, there are some cases where you would not want traffic marked as INVALID to be dropped here. Specifically, our use case:

SCTP Traffic

I'd like to see a Neutron option to enable/disable the population of this INVALID state rule, so that operators (such as us) can disable it if desired. Obviously it's better in general to keep it in there to drop invalid packets, but there are cases where you would like to not do this.


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:
SCTP traffic dropped

Expected results:
Do not drop SCTP traffic

Additional info:

Manual workaround:

BEFORE

SCTP rule 0 hits, INVALID -j DROP incrementing

[root@host]# ip6tables -nvL neutron-openvswi-o4fe90961-b --line-numbers
Chain neutron-openvswi-o4fe90961-b (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1      294 21168 RETURN     icmpv6    *      *       ::/0                 ::/0               
2        0     0 RETURN     udp      *      *       ::/0                 ::/0                 udp spt:546 dpt:547
3    75156 8322K neutron-openvswi-s4fe90961-b  all      *      *       ::/0                 ::/0               
4        0     0 DROP       udp      *      *       ::/0                 ::/0                 udp spt:547 dpt:546
5       77  8624 DROP       all      *      *       ::/0                 ::/0                 state INVALID
6       39  2813 RETURN     all      *      *       ::/0                 ::/0                 state RELATED,ESTABLISHED
7        0     0 RETURN     sctp     *      *       ::/0                 ::/0               
8     4095  269K RETURN     all      *      *       ::/0                 ::/0               
9        0     0 neutron-openvswi-sg-fallback  all      *      *       ::/0                 ::/0               

[root@host]# ip6tables -D neutron-openvswi-o4fe90961-b 5
[root@host]# ip6tables -I neutron-openvswi-o4fe90961-b 8 -m state --state INVALID -j DROP

AFTER

SCTP rule hits incrementing, traffic leaves VM

[root@host]# ip6tables -nvL neutron-openvswi-o4fe90961-b --line-numbers
Chain neutron-openvswi-o4fe90961-b (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1     1987  143K RETURN     icmpv6    *      *       ::/0                 ::/0               
2        0     0 RETURN     udp      *      *       ::/0                 ::/0                 udp spt:546 dpt:547
3    75606 8372K neutron-openvswi-s4fe90961-b  all      *      *       ::/0                 ::/0               
4        0     0 DROP       udp      *      *       ::/0                 ::/0                 udp spt:547 dpt:546
5       39  2813 RETURN     all      *      *       ::/0                 ::/0                 state RELATED,ESTABLISHED
6      414 46368 RETURN     sctp     *      *       ::/0                 ::/0               
7     4103  269K RETURN     all      *      *       ::/0                 ::/0               
8        0     0 DROP       all      *      *       ::/0                 ::/0                 state INVALID
9        0     0 neutron-openvswi-sg-fallback  all      *      *       ::/0                 ::/0
Comment 2 Benjamin Schmaus 2015-10-02 15:06:21 EDT
Proposed fix:


- --- iptables_firewall.py        2015-10-02 11:17:12.643029791 -0500
+++ iptables_firewall.py.new    2015-10-02 11:18:11.224579502 -0500
@@ -425,7 +425,6 @@
 
     def _convert_sgr_to_iptables_rules(self, security_group_rules):
         iptables_rules = []
- -        self._drop_invalid_packets(iptables_rules)
         self._allow_established(iptables_rules)
         for rule in security_group_rules:
             if self.enable_ipset:
@@ -454,6 +453,7 @@
             args += ['-j RETURN']
             iptables_rules += [' '.join(args)]
 
+        self._drop_invalid_packets(iptables_rules)
         iptables_rules += ['-j $sg-fallback']
 
         return iptables_rules
Comment 3 Assaf Muller 2015-10-03 10:57:27 EDT
Added upstream bug and patch.
Comment 4 Nir Yechiel 2015-10-06 02:55:19 EDT
Hi Assaf,

Assuming this is possible with iptables (and AFAIK it is), why not explicitly add SCTP as a classification option under Neutron's security-group? Although not common on the enterprise, allowing SCTP is a common request on the carrier/NFV space.

Thanks,
Nir
Comment 5 Nir Magnezi 2015-10-06 07:41 EDT
Created attachment 1080203 [details]
before_and_after.txt

(In reply to Nir Yechiel from comment #4)
> Hi Assaf,
> 
> Assuming this is possible with iptables (and AFAIK it is), why not
> explicitly add SCTP as a classification option under Neutron's
> security-group? Although not common on the enterprise, allowing SCTP is a
> common request on the carrier/NFV space.
> 
> Thanks,
> Nir

nyechiel,

The implementation is actually not far from what you suggest.
It is possible to add SCTP rule by protocol number (via CLI), and it will be processed before the INVALID DROP rule.

snipped from commit message: "Process user-defined iptables rules before the INVALID DROP".

see the attached example.

I'm leaving the needinfo as-is in case amuller has something to add.
Comment 6 Benjamin Schmaus 2015-10-06 07:58:32 EDT
@Nir Yechiel - Does that mean there is a potential workaround via the CLI and what would that look like then?
Comment 7 Assaf Muller 2015-10-06 09:23:43 EDT
We need to backport the patch, then you'll be able to add a specific accept rule for your SCTP traffic. Without the patch that won't work because of the way the iptables rules are ordered: First we catch INVALID packets, then we go through the rest of the chains. What the patch does is reorder the rules, so that your SCTP accept rule will be first, and the INVALID rule second.
Comment 12 Alexander Stafeyev 2015-10-18 08:45:10 EDT
http://pastebin.test.redhat.com/320727
Comment 13 Nir Magnezi 2015-10-19 08:03 EDT
Created attachment 1084372 [details]
verification.txt

Adding the content of comment #12 as an attachment, since the paste will eventually expire.
Comment 15 errata-xmlrpc 2015-10-29 07:39:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1950.html

Note You need to log in before you can comment on or make changes to this bug.