Bug 171267

Kernel version kernel-smp-2.6.13-1.1526_FC4 was used by both client and server,
but earlier kernels (both uniprocessor and smp) had similar problems about
10-30% of the time.  The other 70-90% of the time, NFS mounts work correctly
with unchanged iptables configuration.

Is selinux in the picture?

No. Selinux is installed but disabled on the client, and not even installed on
the server.

Mass update to all FC4 bugs:

An update has been released (2.6.14-1.1637_FC4) which rebases to a new upstream
kernel (2.6.13.2). As there were ~3500 changes upstream between this and the
previous kernel, it's possible your bug has been fixed already.

Please retest with this update, and update this bug if necessary.

Thanks.

No improvement at reboot -- NFS still complains "could not read superblock". 
Some improvement at mount from command line after reboot, due to retries.  Here
is the ethereal log of "mount -a -t nfs" from the command line, picking up the
attempt to establish NFS v3 TCP connection after after successful NFS authorization:

Pkt No. Time        Source           Destination      Protocol Detail...
     51 0.025710    Client           Server           TCP      800 > nfs [SYN]
Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=4294947169 TSER=0 WS=2
     52 0.025821    Server           Client           TCP      nfs > 800 [ACK]
Seq=0 Ack=0 Win=16022 Len=0 TSV=191157096 TSER=156374
     53 0.025853    Client           Server           ICMP     Destination
unreachable (Host administratively prohibited)

Note the ICMP rejection -- generated by Client's iptables despite the fact that
the Client originated the TCP connection to the NFS server.  This rejection
doesn't happen when iptables are disabled.

HINT: TCP 3-way handshake should proceed as SYN-SYN-ACK, but based on my
ethereal logs, the Server's response to Client's SYN doesn't set the SYN bit. 
This could be a bug on the NFS server side.  All machines are running the latest
software (kernel 2.6.14-1.1637_FC4 etc.).

The next mount retry also fails 3 seconds later:

     54 3.022518    Client           Server           TCP      800 > nfs [SYN]
Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=4294947919 TSER=0 WS=2
     55 3.022637    Server           Client           TCP      nfs > 800 [ACK]
Seq=529232714 Ack=2998602303 Win=16022 Len=0 TSV=191157845 TSER=156374
     56 3.022678    Client           Server           ICMP     Destination
unreachable (Host administratively prohibited)

The attempt at 9 seconds gets no response:

     61 9.022629    Client           Server           TCP      800 > nfs [SYN]
Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=4294949419 TSER=0 WS=2

The attempt at 21 seconds finally succeeds -- note that this time the Server's
response to Client's SYN is a [SYN,ACK] packet as required by TCP 3-way handshake:

     65 21.022848   Client           Server           TCP      800 > nfs [SYN]
Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=4294952419 TSER=0 WS=2
     66 21.023019   Server           Client           TCP      nfs > 800 [SYN,
ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=191162345 TSER=4294952419 WS=2
     67 21.023074   Client           Server           TCP      800 > nfs [ACK]
Seq=1 Ack=1 Win=5840 Len=0 TSV=4294952419 TSER=191162345
     68 21.029047   Client           Server           NFS      V3 NULL Call
(Reply In 70)

BTW, during the above, the iptables configuration on the Client was quite basic
(default+ssh):

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Finally, SELinux is disabled on both Client and Server.  This looks like a
problem involving iptables and NFS mount only.

Most likely, there is a bug in nfsd where clients reconnecting after reboot do
not get the normal TCP 3-way handshake (SYN-SYN-ACK).  Instead, the server
responds to client's SYN with a pure ACK, so that iptables blocks the unexpected
SYN-ACK sequence.

This is a mass-update to all currently open kernel bugs.

A new kernel update has been released (Version: 2.6.15-1.1830_FC4)
based upon a new upstream kernel release.

Please retest against this new kernel, as a large number of patches
go into each upstream release, possibly including changes that
may address this problem.

This bug has been placed in NEEDINFO_REPORTER state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

If this bug is a problem preventing you from installing the
release this version is filed against, please see bug 169613.

Thank you.

Closing per previous comment.