Bug 448328 - ssh connection hangs when running command producing large text output after running "service iptables restart"
ssh connection hangs when running command producing large text output after r...
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
x86_64 Linux
low Severity medium
: rc
: ---
Assigned To: Anton Arapov
Martin Jenner
Depends On:
  Show dependency treegraph
Reported: 2008-05-25 19:09 EDT by Nick Christenson
Modified: 2014-06-18 04:01 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-01-20 15:21:03 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
proposed patch (8.55 KB, patch)
2008-06-13 08:30 EDT, Anton Arapov
no flags Details | Diff
proposed patch for RHEL5 (7.86 KB, patch)
2008-06-13 10:35 EDT, Anton Arapov
no flags Details | Diff

  None (edit)
Description Nick Christenson 2008-05-25 19:09:59 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv: Gecko/20080404 Firefox/

Description of problem:
This is a completely up-to-date RHEL 5.2 server.   Producing a great deal of text output after restarting the iptables service will hang an ssh connection in which the command is run and the text output is generated.

Version-Release number of selected component (if applicable):
iptables-1.3.5-4.el5, kernel-2.6.18-92.el5

How reproducible:

Steps to Reproduce:
1.  ssh to the affected machines
2.  su or sudo -s to root
3.  run "service iptables restart"
4.  within a few seconds or so (no realy hurry), run a program that produces a great deal of text output.  E.g., "dmesg".
5.  The ssh session will hang until the TCP connection finally times out.

Actual Results:
The ssh connection into the machine hangs before the "dmesg" output completes.  Eventually the TCP connection times out, the ssh process goes away, and you're dropped to the shell prompt on the connecting machine.

Expected Results:
dmesg should finish and the ssh connection should not hang.

Additional info:
We can reproduce this on servers running RHEL 5.1 and RHEL 4 but not RHEL 3.  We cannot reproduce this on a server running Ubuntu 8.04.  It sounds like it's a 2.6 kernel issue, and something introduced by RedHat.  That should help narrow things down.

If your "dmesg" is short, it may not produce enough output for this bug to manifest, if so, try resetting ip tables again and running "dmesg;dmesg;dmesg" or  a continual loop or cat a huge file or some such to cause the problem to manifest itself.

It's *possible* but unlikely there is some overlap between this and bug 165744.

Let me know if you want/need more information.
Comment 1 Thomas Woerner 2008-05-26 06:39:32 EDT
Do I understand this correct: Your ssh connection only dies if you output lots
of text, and if you do not, it is still up and working?
Comment 2 Gavin Scott 2008-05-26 10:36:04 EDT
I see this problem as well.  Yes, the connection continues to function fine until such time as you run a 
command that produces a sufficient amount of output.  Something like cat /usr/share/dict/words will 
work as well.  Note that the problem affects any other ssh conections that were underway at the time of 
the iptables restart.  Also, I have some evidence that it affects all active tcp/ip connections (i.e. not just 
ssh) that were allowed by the iptables rules and were open at the time of the restart.
Comment 3 Thomas Woerner 2008-05-26 12:24:02 EDT
A firewall restart resets the firewall, unloads all netfilter modules and
configures the firewall again. But this should affect the applications
immediately. Therefore it seems to be a kernel problem.

Assigning to kernel.
Comment 4 Nick Christenson 2008-05-28 21:27:42 EDT
Since the status is in "NEEDINFO" from me, I'll answer Comment #1.

Yes, that supposition is correct.  I'll also point out that the comments
by Gavin Scott are exactly on point, and that his comments should be 
considered interchangeable with my own.
Comment 5 Anton Arapov 2008-06-10 06:58:13 EDT
has been successfully reproduced, investigating...

nice workaround for default iptables rules, for ssh:
--A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
+-A RH-Firewall-1-INPUT -p tcp --dport 22 -j ACCEPT

Comment 6 Anton Arapov 2008-06-11 09:33:34 EDT
just a notice:
vanilla, used as a base for RHEL5 kernels has the issue as well. So
that the problem was not introduced by any of Red Hat's patches.

/me investigating...
Comment 7 Anton Arapov 2008-06-13 08:30:50 EDT
Created attachment 309201 [details]
proposed patch

upstream commit #a09113c2c8ec59a5cc228efa5869aade2b8f13f7 solves this issue.
Comment 8 Anton Arapov 2008-06-13 10:35:53 EDT
Created attachment 309218 [details]
proposed patch for RHEL5

backport of upstream patch
Comment 9 RHEL Product and Program Management 2008-07-10 14:02:49 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
Comment 11 Don Zickus 2008-08-08 17:08:21 EDT
in kernel-2.6.18-103.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5
Comment 16 errata-xmlrpc 2009-01-20 15:21:03 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.