Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1268413

Summary: security groups iptables can block legitimate traffic as INVALID
Product: Red Hat OpenStack Reporter: Benjamin Schmaus <bschmaus>
Component: openstack-neutronAssignee: Nir Magnezi <nmagnezi>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: amuller, chrisw, dmaley, jlibosva, lpeer, nyechiel, oblaut, sputhenp, yeylon
Target Milestone: asyncKeywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-2014.2.3-23.el7ost Doc Type: Bug Fix
Doc Text:
Cause: The Linux iptables implementation of security groups includes a default rule to drop any INVALID packets. Before this fix, User-defined iptables rules were processed before the INVALID DROP rule. Consequence: Security groups iptables can block legitimate traffic, such as SCTP protocol, as INVALID. Fix: Process user-defined iptables rules before the INVALID DROP rule.
Story Points: ---
Clone Of:
: 1269849 1338971 (view as bug list) Environment:
Last Closed: 2015-10-29 11:39:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1269849, 1338971    
Attachments:
Description Flags
before_and_after.txt
none
verification.txt none

Description Benjamin Schmaus 2015-10-02 18:58:34 UTC
Description of problem:

The iptables implementation of security groups includes a default rule to drop any INVALID packets (according to the Linux connection state tracking system.) It looks like this:

-A neutron-openvswi-od0518220-e -m state --state INVALID -j DROP

This is placed near the top of the rule stack, before any security group rules added by the user. See:

https://github.com/openstack/neutron/blob/stable/kilo/neutron/agent/linux/iptables_firewall.py#L495
https://github.com/openstack/neutron/blob/stable/kilo/neutron/agent/linux/iptables_firewall.py#L506-L510

However, there are some cases where you would not want traffic marked as INVALID to be dropped here. Specifically, our use case:

SCTP Traffic

I'd like to see a Neutron option to enable/disable the population of this INVALID state rule, so that operators (such as us) can disable it if desired. Obviously it's better in general to keep it in there to drop invalid packets, but there are cases where you would like to not do this.


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:
SCTP traffic dropped

Expected results:
Do not drop SCTP traffic

Additional info:

Manual workaround:

BEFORE

SCTP rule 0 hits, INVALID -j DROP incrementing

[root@host]# ip6tables -nvL neutron-openvswi-o4fe90961-b --line-numbers
Chain neutron-openvswi-o4fe90961-b (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1      294 21168 RETURN     icmpv6    *      *       ::/0                 ::/0               
2        0     0 RETURN     udp      *      *       ::/0                 ::/0                 udp spt:546 dpt:547
3    75156 8322K neutron-openvswi-s4fe90961-b  all      *      *       ::/0                 ::/0               
4        0     0 DROP       udp      *      *       ::/0                 ::/0                 udp spt:547 dpt:546
5       77  8624 DROP       all      *      *       ::/0                 ::/0                 state INVALID
6       39  2813 RETURN     all      *      *       ::/0                 ::/0                 state RELATED,ESTABLISHED
7        0     0 RETURN     sctp     *      *       ::/0                 ::/0               
8     4095  269K RETURN     all      *      *       ::/0                 ::/0               
9        0     0 neutron-openvswi-sg-fallback  all      *      *       ::/0                 ::/0               

[root@host]# ip6tables -D neutron-openvswi-o4fe90961-b 5
[root@host]# ip6tables -I neutron-openvswi-o4fe90961-b 8 -m state --state INVALID -j DROP

AFTER

SCTP rule hits incrementing, traffic leaves VM

[root@host]# ip6tables -nvL neutron-openvswi-o4fe90961-b --line-numbers
Chain neutron-openvswi-o4fe90961-b (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1     1987  143K RETURN     icmpv6    *      *       ::/0                 ::/0               
2        0     0 RETURN     udp      *      *       ::/0                 ::/0                 udp spt:546 dpt:547
3    75606 8372K neutron-openvswi-s4fe90961-b  all      *      *       ::/0                 ::/0               
4        0     0 DROP       udp      *      *       ::/0                 ::/0                 udp spt:547 dpt:546
5       39  2813 RETURN     all      *      *       ::/0                 ::/0                 state RELATED,ESTABLISHED
6      414 46368 RETURN     sctp     *      *       ::/0                 ::/0               
7     4103  269K RETURN     all      *      *       ::/0                 ::/0               
8        0     0 DROP       all      *      *       ::/0                 ::/0                 state INVALID
9        0     0 neutron-openvswi-sg-fallback  all      *      *       ::/0                 ::/0

Comment 2 Benjamin Schmaus 2015-10-02 19:06:21 UTC
Proposed fix:


- --- iptables_firewall.py        2015-10-02 11:17:12.643029791 -0500
+++ iptables_firewall.py.new    2015-10-02 11:18:11.224579502 -0500
@@ -425,7 +425,6 @@
 
     def _convert_sgr_to_iptables_rules(self, security_group_rules):
         iptables_rules = []
- -        self._drop_invalid_packets(iptables_rules)
         self._allow_established(iptables_rules)
         for rule in security_group_rules:
             if self.enable_ipset:
@@ -454,6 +453,7 @@
             args += ['-j RETURN']
             iptables_rules += [' '.join(args)]
 
+        self._drop_invalid_packets(iptables_rules)
         iptables_rules += ['-j $sg-fallback']
 
         return iptables_rules

Comment 3 Assaf Muller 2015-10-03 14:57:27 UTC
Added upstream bug and patch.

Comment 4 Nir Yechiel 2015-10-06 06:55:19 UTC
Hi Assaf,

Assuming this is possible with iptables (and AFAIK it is), why not explicitly add SCTP as a classification option under Neutron's security-group? Although not common on the enterprise, allowing SCTP is a common request on the carrier/NFV space.

Thanks,
Nir

Comment 5 Nir Magnezi 2015-10-06 11:41:38 UTC
Created attachment 1080203 [details]
before_and_after.txt

(In reply to Nir Yechiel from comment #4)
> Hi Assaf,
> 
> Assuming this is possible with iptables (and AFAIK it is), why not
> explicitly add SCTP as a classification option under Neutron's
> security-group? Although not common on the enterprise, allowing SCTP is a
> common request on the carrier/NFV space.
> 
> Thanks,
> Nir

nyechiel,

The implementation is actually not far from what you suggest.
It is possible to add SCTP rule by protocol number (via CLI), and it will be processed before the INVALID DROP rule.

snipped from commit message: "Process user-defined iptables rules before the INVALID DROP".

see the attached example.

I'm leaving the needinfo as-is in case amuller has something to add.

Comment 6 Benjamin Schmaus 2015-10-06 11:58:32 UTC
@Nir Yechiel - Does that mean there is a potential workaround via the CLI and what would that look like then?

Comment 7 Assaf Muller 2015-10-06 13:23:43 UTC
We need to backport the patch, then you'll be able to add a specific accept rule for your SCTP traffic. Without the patch that won't work because of the way the iptables rules are ordered: First we catch INVALID packets, then we go through the rest of the chains. What the patch does is reorder the rules, so that your SCTP accept rule will be first, and the INVALID rule second.

Comment 12 Alexander Stafeyev 2015-10-18 12:45:10 UTC
http://pastebin.test.redhat.com/320727

Comment 13 Nir Magnezi 2015-10-19 12:03:26 UTC
Created attachment 1084372 [details]
verification.txt

Adding the content of comment #12 as an attachment, since the paste will eventually expire.

Comment 15 errata-xmlrpc 2015-10-29 11:39:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1950.html