Bug 182012

Description of problem:

We use an iptables setup which has a rule allowing packets associated with
"ESTABLISHED" connections through, followed by specific rules allowing "NEW"
connections to particular ports, followed by fallthrough rules denying all
traffic.  Very infrequently, packets associated with established connections are
not matched by the early-accept rule; since these packets are not initiating new
connections, they fall through to the reject rules, leading to either spurious
ICMP "prohibited" or TCP reset messages, depending on the particular reject rules.

This behavior occurs with connections that are actively exchanging traffic; idle
timeouts, TCP keepalives, etc. are not at issue.

An example of a problem configuration is
http://www-cse.ucsd.edu/~jbrown/reset/iptables-problem ; a workaround for this
problem is to remove the "-m state --state NEW" qualifier, as seen in
http://www-cse.ucsd.edu/~jbrown/reset/iptables-okay .  With that qualifier
removed, when the "ESTABLISHED" rule fails to match, the packets are still
allowed in by the per-port accept rules.

An example of a spurious TCP reset caused by the problem configuration is
http://www-cse.ucsd.edu/~jbrown/reset/both .  Observing the server end, the
server transmits sequence 4167818615:4167818687, the client responds with
2733189498:2733189546 and ACKs the receipt of the server's packet (4167818687).
 At this point the server sends a spurious RST, followed immediately by a
retransmit of 
4167818615:4167818687 : it's sending data for a connection which it has just
reset, and for which it has already received an ACK.  We surmise that the
ESTABLISHED rule incorrectly failed to match the packet carrying the ACK,
leading to that packet being dropped and an RST being generated by the server's
firewall.  The server's own TCP stack never received the ACK and didn't actually
reset the connection, so it dutifully re-transmitted the data.  At that point,
the client had already received the RST, so it responded with an RST of its own,
and the connection was torn down.

In the default iptables configuration of our RHEL4U2 installation --
http://www-cse.ucsd.edu/~jbrown/reset/iptables-orig -- the occasional failure of
ESTABLISHED to match leads only to a spurious ICMP message, which seems to be
ignored by clients for already-established connections, so no problems are
observed by users.  (We have confirmed via tcpdump that spurious ICMP messages
are in fact generated.)  Our addition of the TCP-reset reject rule led to this
problem impacting users, as their connections would spontaneously be reset.


Version-Release number of selected component (if applicable): RHEL 4 U 2


How reproducible: Unfortunately, this problem does not occur deterministically.
We observe it at varying frequencies from different client networks, with
frequencies ranging from once every few minutes to once every few days.


Steps to Reproduce:
1. Install the given "problem" rule set on a server.
2. SSH in and work away.
3. Probabilistically observe your connection getting dropped.
  
Actual results: "Read from remote host XXXX.SERVER.DOMAIN: Connection reset by peer"

Expected results: (Connections not being reset)


Additional info: kernel version "2.6.9-22.ELsmp #1 SMP", behavior observed both
on single-processor and SMP machines.