Bug 448328

Summary: ssh connection hangs when running command producing large text output after running "service iptables restart"
Product: Red Hat Enterprise Linux 5 Reporter: Nick Christenson <npc>
Component: kernelAssignee: Anton Arapov <anton>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: medium Docs Contact:
Priority: low    
Version: 5.2CC: dzickus, gavindscott, igeorgex, k.georgiou, nobody, rlerch
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 20:21:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
proposed patch
none
proposed patch for RHEL5 none

Description Nick Christenson 2008-05-25 23:09:59 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14

Description of problem:
This is a completely up-to-date RHEL 5.2 server.   Producing a great deal of text output after restarting the iptables service will hang an ssh connection in which the command is run and the text output is generated.

Version-Release number of selected component (if applicable):
iptables-1.3.5-4.el5, kernel-2.6.18-92.el5

How reproducible:
Always


Steps to Reproduce:
1.  ssh to the affected machines
2.  su or sudo -s to root
3.  run "service iptables restart"
4.  within a few seconds or so (no realy hurry), run a program that produces a great deal of text output.  E.g., "dmesg".
5.  The ssh session will hang until the TCP connection finally times out.

Actual Results:
The ssh connection into the machine hangs before the "dmesg" output completes.  Eventually the TCP connection times out, the ssh process goes away, and you're dropped to the shell prompt on the connecting machine.

Expected Results:
dmesg should finish and the ssh connection should not hang.

Additional info:
We can reproduce this on servers running RHEL 5.1 and RHEL 4 but not RHEL 3.  We cannot reproduce this on a server running Ubuntu 8.04.  It sounds like it's a 2.6 kernel issue, and something introduced by RedHat.  That should help narrow things down.

If your "dmesg" is short, it may not produce enough output for this bug to manifest, if so, try resetting ip tables again and running "dmesg;dmesg;dmesg" or  a continual loop or cat a huge file or some such to cause the problem to manifest itself.

It's *possible* but unlikely there is some overlap between this and bug 165744.

Let me know if you want/need more information.

Comment 1 Thomas Woerner 2008-05-26 10:39:32 UTC
Do I understand this correct: Your ssh connection only dies if you output lots
of text, and if you do not, it is still up and working?

Comment 2 Gavin Scott 2008-05-26 14:36:04 UTC
I see this problem as well.  Yes, the connection continues to function fine until such time as you run a 
command that produces a sufficient amount of output.  Something like cat /usr/share/dict/words will 
work as well.  Note that the problem affects any other ssh conections that were underway at the time of 
the iptables restart.  Also, I have some evidence that it affects all active tcp/ip connections (i.e. not just 
ssh) that were allowed by the iptables rules and were open at the time of the restart.

Comment 3 Thomas Woerner 2008-05-26 16:24:02 UTC
A firewall restart resets the firewall, unloads all netfilter modules and
configures the firewall again. But this should affect the applications
immediately. Therefore it seems to be a kernel problem.

Assigning to kernel.

Comment 4 Nick Christenson 2008-05-29 01:27:42 UTC
Since the status is in "NEEDINFO" from me, I'll answer Comment #1.

Yes, that supposition is correct.  I'll also point out that the comments
by Gavin Scott are exactly on point, and that his comments should be 
considered interchangeable with my own.

Comment 5 Anton Arapov 2008-06-10 10:58:13 UTC
has been successfully reproduced, investigating...

nice workaround for default iptables rules, for ssh:
--A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
+-A RH-Firewall-1-INPUT -p tcp --dport 22 -j ACCEPT



Comment 6 Anton Arapov 2008-06-11 13:33:34 UTC
just a notice:
vanilla 2.6.18.4, used as a base for RHEL5 kernels has the issue as well. So
that the problem was not introduced by any of Red Hat's patches.

/me investigating...

Comment 7 Anton Arapov 2008-06-13 12:30:50 UTC
Created attachment 309201 [details]
proposed patch

upstream commit #a09113c2c8ec59a5cc228efa5869aade2b8f13f7 solves this issue.

Comment 8 Anton Arapov 2008-06-13 14:35:53 UTC
Created attachment 309218 [details]
proposed patch for RHEL5

backport of upstream patch

Comment 9 RHEL Program Management 2008-07-10 18:02:49 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 11 Don Zickus 2008-08-08 21:08:21 UTC
in kernel-2.6.18-103.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Comment 16 errata-xmlrpc 2009-01-20 20:21:03 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-0225.html