Red Hat Bugzilla – Bug 448328
ssh connection hangs when running command producing large text output after running "service iptables restart"
Last modified: 2014-06-18 04:01:45 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:184.108.40.206) Gecko/20080404 Firefox/220.127.116.11
Description of problem:
This is a completely up-to-date RHEL 5.2 server. Producing a great deal of text output after restarting the iptables service will hang an ssh connection in which the command is run and the text output is generated.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. ssh to the affected machines
2. su or sudo -s to root
3. run "service iptables restart"
4. within a few seconds or so (no realy hurry), run a program that produces a great deal of text output. E.g., "dmesg".
5. The ssh session will hang until the TCP connection finally times out.
The ssh connection into the machine hangs before the "dmesg" output completes. Eventually the TCP connection times out, the ssh process goes away, and you're dropped to the shell prompt on the connecting machine.
dmesg should finish and the ssh connection should not hang.
We can reproduce this on servers running RHEL 5.1 and RHEL 4 but not RHEL 3. We cannot reproduce this on a server running Ubuntu 8.04. It sounds like it's a 2.6 kernel issue, and something introduced by RedHat. That should help narrow things down.
If your "dmesg" is short, it may not produce enough output for this bug to manifest, if so, try resetting ip tables again and running "dmesg;dmesg;dmesg" or a continual loop or cat a huge file or some such to cause the problem to manifest itself.
It's *possible* but unlikely there is some overlap between this and bug 165744.
Let me know if you want/need more information.
Do I understand this correct: Your ssh connection only dies if you output lots
of text, and if you do not, it is still up and working?
I see this problem as well. Yes, the connection continues to function fine until such time as you run a
command that produces a sufficient amount of output. Something like cat /usr/share/dict/words will
work as well. Note that the problem affects any other ssh conections that were underway at the time of
the iptables restart. Also, I have some evidence that it affects all active tcp/ip connections (i.e. not just
ssh) that were allowed by the iptables rules and were open at the time of the restart.
A firewall restart resets the firewall, unloads all netfilter modules and
configures the firewall again. But this should affect the applications
immediately. Therefore it seems to be a kernel problem.
Assigning to kernel.
Since the status is in "NEEDINFO" from me, I'll answer Comment #1.
Yes, that supposition is correct. I'll also point out that the comments
by Gavin Scott are exactly on point, and that his comments should be
considered interchangeable with my own.
has been successfully reproduced, investigating...
nice workaround for default iptables rules, for ssh:
--A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
+-A RH-Firewall-1-INPUT -p tcp --dport 22 -j ACCEPT
just a notice:
vanilla 18.104.22.168, used as a base for RHEL5 kernels has the issue as well. So
that the problem was not introduced by any of Red Hat's patches.
Created attachment 309201 [details]
upstream commit #a09113c2c8ec59a5cc228efa5869aade2b8f13f7 solves this issue.
Created attachment 309218 [details]
proposed patch for RHEL5
backport of upstream patch
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release. Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products. This request is not yet committed for inclusion in an Update
You can download this test kernel from http://people.redhat.com/dzickus/el5
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.