Description of problem: Version-Release number of selected component (if applicable): iptables-1.3.5-1.2.1 How reproducible: 100%, from a small sample Steps to Reproduce: 1.service iptables stop 2.observe 3.scratch head Actual results: # service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [FAILED] Expected results: # service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [ OK ] Additional info: sh -x /etc/init.d/iptables stop ... + modprobe -r ip_conntrack + let ret+=1 + return 2 + let ret+=2 + '[' 2 -eq 0 ']' + failure + local rc=1 + '[' color '!=' verbose -a -z '' ']' + echo_failure + '[' color = color ']' + echo -en '\033[60G' + echo -n '[' [+ '[' color = color ']' + echo -en '\033[0;31m' + echo -n FAILED FAILED+ '[' color = color ']' + echo -en '\033[0;39m' And if you try to remove ip_conntrack by hand: modprobe -r ip_conntrack FATAL: Module ip_conntrack is in use.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering. This request is not yet committed for inclusion in release.
This is no iptables userland problem. The xt_state module can not be unloaded and is blocking ip_conntrack. Assigning to kernel.
This looks very similiar to the bug 216516. The patch included in that bug, 216516, may resolve this issue as well.
similiar issue reported with bug 203959
*** Bug 216892 has been marked as a duplicate of this bug. ***
*** Bug 213560 has been marked as a duplicate of this bug. ***
*** Bug 112630 has been marked as a duplicate of this bug. ***
*** Bug 216516 has been marked as a duplicate of this bug. ***
One thing to look out for is if the qeth driver does delayed TX packet freeing in a way that there is no upper bound. Delaying TX using hw interrupt mitigation techniques is fine, as long as there is a timer based small upper bound to the TX freeing. But there must be that small upper bound, it is critical to correct operation of the entire networking stack. For example, if it does, this will break many many things, not just ip_conntrack module unloading. It will hurt TCP performance significantly as well, because until the TX packet is freed up by the device the socket cannot use that send buffer space for queueing new packets. If the qeth driver is doing delayed TX packet freeing without an appropriate upper bound, this will need to be fixed.
I see a lot of comments about qeth driver here but this issue with failing 'service iptables stop' happens on my box with an e1000 NIC too so it does not appear to be qeth-specific. See the bugs that have been closed as duplicates of this one, I don't think *all* those folks are running mainframes.
Does unloading the e1000 module or downing the e1000 interface make the problem go away as it does in the qeth case?
Well, I actually have a box with a Tigon3 NIC here right now, anyway, 'service iptables stop' fails for the same reason even if I unload the tg3 module.
I don't think this is adapter specific. I have a system with a 3c905C-TX and it fails in the same way whether the driver is loaded or unloaded.
Created attachment 143074 [details] Add ip_conntrack dependency to x_tables so it's removed when ip_conntrack is unloaded Give this patch a try, solves iptables restart for me.
Created attachment 143160 [details] proposed additional patch This patch is required as well.
verified fix with internal RHEL5-Server-20061218.nightly/s390x tree ... dup-ing this issue to the bug with better tracking ... *** This bug has been marked as a duplicate of 218798 ***
this bug is not duplicated of 218798. bug 218798 issue is a different issue related to qeth driver. This issue is related to netfilter, and x-table, so reopen this issue for tracking.
Created attachment 144042 [details] Updated proposed patch Thomas, I stripped out my patch (comment 25) from yours (comment 27) and verified it still works. I'm not 100% sure the x_tables conntrack dependency is necessary given your patch.
This patch looks fine to me, FWIW.
Brad, if you have a chance, could you try without the need_conntrack() chunk so we can verify that its properly fixed upstream?
With just the patch from comment 30 (need_conntrack chunk removed) I can restart iptables with and without the interface up. During shutdown though iptables still hangs.. but I think this may be s390 specific (see bug 218798). Can the original reporter test the patch in comment 30?
Can I get a fixed RPM to test? I'd rather not get into the kernel rebuilding business to test this.
Created attachment 144747 [details] i686
Created attachment 144749 [details] x86_64
Hey Brad, any chance you have an x86_64 kernel-xen rpm around? That would be much easier for me to test.
Created attachment 144824 [details] x86_64 xen Sure thing
Built into 2.6.18-1.3002.el5.
Jeff, does the patch in Comment #25 fix the problem for you?
If you can point me at a kernel with that patch in it, I'd be happy to test it and let you know.
building one now..
Brad, any test results?
Created attachment 145475 [details] contains patches from both comment 25 and comment 27
Created attachment 145476 [details] contains patch from both comment 25 only
Created attachment 145477 [details] contains patch from comment 25 only
Do you happen to have x86 versions? Yeah, I know, I'm a pain in the butt.
Having some unrelated issues building x86. Any way you can test the 64 xen?
Sorry, I can't. For some odd reason, I can't reproduce this in that environment right now. I'll keep trying, but can guarantee testing results with an x86 because I have a system reproducing that right now.
Created attachment 145509 [details] i686 .2: comment 25 patch only
I downloaded the kernel in c#57, installed it, and rebooted the system. It never came back up, so further test results will have to wait until Monday.
Ah, spoke too soon. It was just incredibly slow to come back up for some reason. Still getting the same results: # service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [FAILED] # modprobe -r ip_conntrack FATAL: Module ip_conntrack is in use.
Is ip6tables running on this system? I just noticed that they share some modules. On one of the stock kernels, if you stop ip6tables first, does iptables stop successfully? If so, we may need to work this out through initscipts rather than kernel.
Good call. Stopping ip6tables leads to iptables stopping OK. # service ip6tables status Table: filter # service ip6tables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading ip6tables modules: [ OK ] # service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [ OK ]
Changing component to initscripts
iptables initscripts -> iptables package.
Feedback from IBM zSeries: Well, i don't knwo whether some of the other changes impact things, but in the previosu code, if ip6tables is stopped first, stopping iptables still hung afterwards but that was without any of the patches applied
The module xt_state depends on ip_conntrack, ip_tables respectively ip6_tables depends on xt_state _if_ the firewall configuration involves connection tracking. This dependency is not shown by lsmod because no symbols are being referenced by either module. Therefore, while ip6tables has IPv6 connection tracking configured, the ip6_tables module will hold a reference on the xt_state module preventing it from being unloaded. This prevents ip_conntrack from being unloaded as it requires xt_state to be unloaded first. This means that as long as either iptables protocol version uses conntracking, neither of them can be unloaded. My suggestion is to disable iptables module unloading until a solution is found.
Cloned this bug for iptables: 222981 Assigning back to kernel, because this is a kernel netfilter problem.
Jeff, can I please see your ip6tables config? I can't reproduce this issue on any of the systems I tried. I'm suspecting you might be using ipv4 netfilter extensions from ip6tables.
Created attachment 145810 [details] /etc/sysconfig/ip6tables
Created attachment 145811 [details] /etc/sysconfig/ip6tables-config
In /etc/sysconfig/ip6tables: -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT Sorry but I really had to laugh. Using xt_state for IPpv6 traffic with ip_conntrack can't possibly work. There is no connection tracking support in ip_conntrack. Whoever added those rules did never even test them. Please remove those rules from the default config ASAP.
I meant to say ... there is no IPv6 connection tracking support in ip_conntrack ... Sorry
The config problem is now addressed in bug 222981. And this issue is only to address the race condition in netfilter. Given that the patch has been integrated in 1.3002.el5 build, move to MODIFIED and VERIFIED.
Closing out as 1.3002.el5 was a part of 20070111.1 and 20070112.3.