Bug 212839 - "service iptables stop" fails because ip_conntrack cannot be unloaded.
"service iptables stop" fails because ip_conntrack cannot be unloaded.
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Thomas Graf
Brian Brock
: Regression, Reopened
: 112630 213560 216892 (view as bug list)
Depends On:
Blocks: 222586 222981
  Show dependency treegraph
 
Reported: 2006-10-29 10:42 EST by Jeff Needle
Modified: 2014-06-18 04:29 EDT (History)
13 users (show)

See Also:
Fixed In Version: 5.0.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-01-19 08:18:50 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Add ip_conntrack dependency to x_tables so it's removed when ip_conntrack is unloaded (640 bytes, patch)
2006-12-07 13:24 EST, Brad Hinson
no flags Details | Diff
proposed additional patch (2.95 KB, patch)
2006-12-08 11:09 EST, Thomas Graf
no flags Details | Diff
Updated proposed patch (2.53 KB, patch)
2006-12-19 15:54 EST, Brad Hinson
no flags Details | Diff
i686 (13.24 MB, application/x-rpm)
2007-01-03 17:41 EST, Brad Hinson
no flags Details
x86_64 (15.35 MB, application/x-rpm)
2007-01-03 18:01 EST, Brad Hinson
no flags Details
x86_64 xen (15.14 MB, application/x-rpm)
2007-01-04 12:40 EST, Brad Hinson
no flags Details
contains patches from both comment 25 and comment 27 (15.14 MB, application/x-rpm)
2007-01-12 13:51 EST, Brad Hinson
no flags Details
contains patch from both comment 25 only (15.14 MB, application/x-rpm)
2007-01-12 13:53 EST, Brad Hinson
no flags Details
contains patch from comment 25 only (15.14 MB, application/x-rpm)
2007-01-12 13:54 EST, Brad Hinson
no flags Details
i686 .2: comment 25 patch only (13.24 MB, application/x-rpm)
2007-01-12 19:57 EST, Brad Hinson
no flags Details
/etc/sysconfig/ip6tables (751 bytes, text/plain)
2007-01-17 09:14 EST, Jeff Needle
no flags Details
/etc/sysconfig/ip6tables-config (1.71 KB, text/plain)
2007-01-17 09:18 EST, Jeff Needle
no flags Details

  None (edit)
Description Jeff Needle 2006-10-29 10:42:24 EST
Description of problem:


Version-Release number of selected component (if applicable):
iptables-1.3.5-1.2.1

How reproducible:
100%, from a small sample

Steps to Reproduce:
1.service iptables stop
2.observe
3.scratch head

Actual results:
# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [FAILED]


Expected results:
# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]


Additional info:
sh -x /etc/init.d/iptables stop

...

+ modprobe -r ip_conntrack
+ let ret+=1
+ return 2
+ let ret+=2
+ '[' 2 -eq 0 ']'
+ failure
+ local rc=1
+ '[' color '!=' verbose -a -z '' ']'
+ echo_failure
+ '[' color = color ']'
+ echo -en '\033[60G'
                                                           + echo -n '['
[+ '[' color = color ']'
+ echo -en '\033[0;31m'
+ echo -n FAILED
FAILED+ '[' color = color ']'
+ echo -en '\033[0;39m'

And if you try to remove ip_conntrack by hand:

modprobe -r ip_conntrack
FATAL: Module ip_conntrack is in use.
Comment 2 RHEL Product and Program Management 2006-10-29 11:04:35 EST
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux release.  Product Management has requested further review
of this request by Red Hat Engineering.  This request is not yet committed for
inclusion in release.
Comment 3 Thomas Woerner 2006-10-30 04:39:25 EST
This is no iptables userland problem. The xt_state module can not be unloaded
and is blocking ip_conntrack.

Assigning to kernel.
Comment 4 Linda Wang 2006-11-20 17:33:23 EST
This looks very similiar to the bug 216516.  The patch included in 
that bug, 216516, may resolve this issue as well.
Comment 5 Linda Wang 2006-11-20 17:34:39 EST
similiar issue reported with bug 203959
Comment 6 Jeff Needle 2006-11-22 10:59:52 EST
*** Bug 216892 has been marked as a duplicate of this bug. ***
Comment 7 Bill Nottingham 2006-11-22 14:05:12 EST
*** Bug 213560 has been marked as a duplicate of this bug. ***
Comment 8 Bill Nottingham 2006-11-22 14:05:45 EST
*** Bug 112630 has been marked as a duplicate of this bug. ***
Comment 9 Linda Wang 2006-11-30 16:00:07 EST
*** Bug 216516 has been marked as a duplicate of this bug. ***
Comment 14 David Miller 2006-12-05 16:22:56 EST
One thing to look out for is if the qeth driver does delayed TX
packet freeing in a way that there is no upper bound.  Delaying
TX using hw interrupt mitigation techniques is fine, as long
as there is a timer based small upper bound to the TX freeing.
But there must be that small upper bound, it is critical to correct
operation of the entire networking stack.

For example, if it does, this will break many many things,
not just ip_conntrack module unloading.  It will hurt TCP performance
significantly as well, because until the TX packet is freed up by
the device the socket cannot use that send buffer space for queueing
new packets.

If the qeth driver is doing delayed TX packet freeing without an
appropriate upper bound, this will need to be fixed.
Comment 15 Radek Bíba 2006-12-06 03:04:41 EST
I see a lot of comments about qeth driver here but this issue with failing
'service iptables stop' happens on my box with an e1000 NIC too so it does not
appear to be qeth-specific. See the bugs that have been closed as duplicates of
this one, I don't think *all* those folks are running mainframes.
Comment 16 David Miller 2006-12-06 03:14:59 EST
Does unloading the e1000 module or downing the e1000 interface make
the problem go away as it does in the qeth case?
Comment 17 Radek Bíba 2006-12-06 03:46:00 EST
Well, I actually have a box with a Tigon3 NIC here right now, anyway, 'service
iptables stop' fails for the same reason even if I unload the tg3 module.
Comment 19 Jeff Needle 2006-12-06 10:21:03 EST
I don't think this is adapter specific.  I have a system with a 3c905C-TX and it
fails in the same way whether the driver is loaded or unloaded.
Comment 25 Brad Hinson 2006-12-07 13:24:43 EST
Created attachment 143074 [details]
Add ip_conntrack dependency to x_tables so it's removed when ip_conntrack is unloaded

Give this patch a try, solves iptables restart for me.
Comment 27 Thomas Graf 2006-12-08 11:09:56 EST
Created attachment 143160 [details]
proposed additional patch

This patch is required as well.
Comment 28 Brock Organ 2006-12-18 16:14:12 EST
verified fix with internal RHEL5-Server-20061218.nightly/s390x tree ... dup-ing
this issue to the bug with better tracking ...

*** This bug has been marked as a duplicate of 218798 ***
Comment 29 Linda Wang 2006-12-18 18:35:46 EST
this bug is not duplicated of 218798.  bug 218798 issue is a different issue
related to qeth driver.

This issue is related to netfilter, and x-table, so reopen this issue for tracking.
Comment 30 Brad Hinson 2006-12-19 15:54:38 EST
Created attachment 144042 [details]
Updated proposed patch

Thomas, I stripped out my patch (comment 25) from yours (comment 27) and
verified it still works.  I'm not 100% sure the x_tables conntrack dependency
is necessary given your patch.
Comment 31 David Miller 2006-12-19 23:22:15 EST
This patch looks fine to me, FWIW.
Comment 32 Thomas Graf 2006-12-20 07:13:21 EST
Brad, if you have a chance, could you try without the need_conntrack() chunk
so we can verify that its properly fixed upstream?
Comment 33 Brad Hinson 2006-12-20 14:45:38 EST
With just the patch from comment 30 (need_conntrack chunk removed) I can restart
iptables with and without the interface up.  During shutdown though iptables
still hangs.. but I think this may be s390 specific (see bug 218798).

Can the original reporter test the patch in comment 30?
Comment 34 Jeff Needle 2007-01-03 12:41:10 EST
Can I get a fixed RPM to test?  I'd rather not get into the kernel rebuilding
business to test this.
Comment 36 Brad Hinson 2007-01-03 17:41:41 EST
Created attachment 144747 [details]
i686
Comment 37 Brad Hinson 2007-01-03 18:01:54 EST
Created attachment 144749 [details]
x86_64
Comment 38 Jeff Needle 2007-01-04 07:45:23 EST
Hey Brad, any chance you have an x86_64 kernel-xen rpm around?  That would be
much easier for me to test.
Comment 39 Brad Hinson 2007-01-04 12:40:44 EST
Created attachment 144824 [details]
x86_64 xen

Sure thing
Comment 40 Jay Turner 2007-01-10 10:53:59 EST
Built into 2.6.18-1.3002.el5.
Comment 46 Thomas Graf 2007-01-11 11:29:25 EST
Jeff, does the patch in Comment #25 fix the problem for you?
Comment 47 Jeff Needle 2007-01-11 11:32:14 EST
If you can point me at a kernel with that patch in it, I'd be happy to test it
and let you know.
Comment 48 Brad Hinson 2007-01-11 12:26:14 EST
building one now..
Comment 49 Linda Wang 2007-01-12 09:56:28 EST
Brad, any test results?
Comment 51 Brad Hinson 2007-01-12 13:51:41 EST
Created attachment 145475 [details]
contains patches from both comment 25 and comment 27
Comment 52 Brad Hinson 2007-01-12 13:54:06 EST
Created attachment 145476 [details]
contains patch from both comment 25 only
Comment 53 Brad Hinson 2007-01-12 13:54:18 EST
Created attachment 145477 [details]
contains patch from comment 25 only
Comment 54 Jeff Needle 2007-01-12 13:56:26 EST
Do you happen to have x86 versions?  Yeah, I know, I'm a pain in the butt.
Comment 55 Brad Hinson 2007-01-12 16:25:58 EST
Having some unrelated issues building x86.  Any way you can test the 64 xen?
Comment 56 Jeff Needle 2007-01-12 16:29:50 EST
Sorry, I can't.  For some odd reason, I can't reproduce this in that environment
right now.  I'll keep trying, but can guarantee testing results with an x86
because I have a system reproducing that right now.
Comment 57 Brad Hinson 2007-01-12 19:57:18 EST
Created attachment 145509 [details]
i686 .2: comment 25 patch only
Comment 58 Jeff Needle 2007-01-12 20:14:20 EST
I downloaded the kernel in c#57, installed it, and rebooted the system.  It
never came back up, so further test results will have to wait until Monday.
Comment 59 Jeff Needle 2007-01-12 20:20:39 EST
Ah, spoke too soon.  It was just incredibly slow to come back up for some
reason.  Still getting the same results:

# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [FAILED]

#  modprobe -r ip_conntrack
FATAL: Module ip_conntrack is in use.
Comment 60 Brad Hinson 2007-01-14 14:39:17 EST
Is ip6tables running on this system?  I just noticed that they share some
modules.  On one of the stock kernels, if you stop ip6tables first, does
iptables stop successfully?  If so, we may need to work this out through
initscipts rather than kernel.
Comment 61 Jeff Needle 2007-01-14 14:45:38 EST
Good call.  Stopping ip6tables leads to iptables stopping OK. 

# service ip6tables status
Table: filter


# service ip6tables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading ip6tables modules:                               [  OK  ]

# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]
Comment 62 Brad Hinson 2007-01-14 15:26:48 EST
Changing component to initscripts
Comment 64 Bill Nottingham 2007-01-15 11:21:40 EST
iptables initscripts -> iptables package.
Comment 65 John Jarvis 2007-01-16 11:42:39 EST
Feedback from IBM zSeries:  Well, i don't knwo whether some of the other changes
impact things, but in the previosu code, if ip6tables is stopped first, stopping
iptables still hung afterwards but that was without any of the patches applied
Comment 66 Thomas Graf 2007-01-16 18:44:37 EST
The module xt_state depends on ip_conntrack, ip_tables respectively
ip6_tables depends on xt_state _if_ the firewall configuration involves
connection tracking. This dependency is not shown by lsmod because no
symbols are being referenced by either module.

Therefore, while ip6tables has IPv6 connection tracking configured,
the ip6_tables module will hold a reference on the xt_state module
preventing it from being unloaded. This prevents ip_conntrack from
being unloaded as it requires xt_state to be unloaded first.

This means that as long as either iptables protocol version uses
conntracking, neither of them can be unloaded.

My suggestion is to disable iptables module unloading until a solution
is found.
Comment 68 Thomas Woerner 2007-01-17 05:35:04 EST
Cloned this bug for iptables: 222981

Assigning back to kernel, because this is a kernel netfilter problem.
Comment 69 Thomas Graf 2007-01-17 09:04:47 EST
Jeff, can I please see your ip6tables config? I can't reproduce this issue
on any of the systems I tried. I'm suspecting you might be using ipv4 netfilter
extensions from ip6tables.
Comment 71 Jeff Needle 2007-01-17 09:14:00 EST
Created attachment 145810 [details]
/etc/sysconfig/ip6tables
Comment 72 Jeff Needle 2007-01-17 09:18:23 EST
Created attachment 145811 [details]
/etc/sysconfig/ip6tables-config
Comment 73 Thomas Graf 2007-01-17 09:31:19 EST
In /etc/sysconfig/ip6tables:
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

Sorry but I really had to laugh. Using xt_state for IPpv6 traffic with
ip_conntrack can't possibly work. There is no connection tracking support
in ip_conntrack. Whoever added those rules did never even test them.

Please remove those rules from the default config ASAP.
Comment 74 Thomas Graf 2007-01-17 09:35:57 EST
I meant to say ... there is no IPv6 connection tracking support in ip_conntrack ...
Sorry
Comment 75 Linda Wang 2007-01-17 12:44:21 EST
The config problem is now addressed in bug 222981.  
And this issue is only to address the race condition in netfilter.
Given that the patch has been integrated in 1.3002.el5 build,
move to MODIFIED and VERIFIED.
Comment 76 Jay Turner 2007-01-19 08:18:50 EST
Closing out as 1.3002.el5 was a part of 20070111.1 and 20070112.3.

Note You need to log in before you can comment on or make changes to this bug.