Bug 479604 - icmp host administratively disabled messages
icmp host administratively disabled messages
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Neil Horman
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-01-11 19:30 EST by Muayyad Alsadi
Modified: 2009-02-25 12:42 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-21 18:57:33 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
small portion of netshark session (in libpcap format) (1.13 KB, application/octet-stream)
2009-01-11 19:30 EST, Muayyad Alsadi
no flags Details
look at the arrow please (144.25 KB, image/png)
2009-01-17 19:10 EST, Muayyad Alsadi
no flags Details
wireshark log of trial2 (5.04 KB, application/octet-stream)
2009-01-20 17:00 EST, Muayyad Alsadi
no flags Details
wireshark log of trial3 (1.34 KB, application/octet-stream)
2009-01-20 17:00 EST, Muayyad Alsadi
no flags Details
screenshot of tplink firewall (118.76 KB, image/png)
2009-01-20 17:01 EST, Muayyad Alsadi
no flags Details

  None (edit)
Description Muayyad Alsadi 2009-01-11 19:30:17 EST
Created attachment 328687 [details]
small portion of netshark session (in libpcap format)

Description of problem:
at random times I no longer can browse the web
(but ping works)

I was advised to use wireshark and a small portion of the report is attached (in libpcap format)

showing the after TCP pockets I get an ICMP pocket telling me that destination is unreachable (host administratively prohibited) (type 3 code 10)

and the checksum is always different by 1

Version-Release number of selected component (if applicable):
kernel-2.6.27.9-159.fc10.i686

How reproducible:
random

Steps to Reproduce:
use the computer for long time
  
Actual results:
can't browse the web or even use wget while the other machine on the LAN can

Expected results:
no network problems

Additional info:
when I reboot I can browse again

I'm using a built-in NIC driven by forcedeth kernel module

# lspci | grep -i eth
00:0f.0 Ethernet controller: nVidia Corporation MCP73 Ethernet (rev a2)
# lspci -n | grep 0f\.0
00:0f.0 0200: 10de:07dc (rev a2)
Comment 1 Neil Horman 2009-01-11 20:33:42 EST
You're having a legitimate network issue.  The ICMP error is being sent from an external source to your system (based on the MAC source and destination in the attached trace).  In fact it looks like its comming from the system that you're tring to access.  As such it would seem clear that the system (or a router on route to the destination) is indicating that your system is not allowed to access the system (it may be a data transfer limitation or something along those lines).  First suggestion would be, if other systems on the lan can access web sites without receiving this error, to swap IP addresses with one of them.  If the ICMP error moves to the machine that you swap with, then you can be fairly confident that someone has a poor firewall configuration somewhere, and you can contact your IT staff for further analysis.
Comment 2 Muayyad Alsadi 2009-01-12 07:54:57 EST
> someone has a poor firewall configuration somewhere,
> and you can contact your IT staff for further analysis

this is my lan at my home, and we have two PCs (one in my room and the other in my brother's room) and a cheap TP-LINK router running DHCP

I tend to blame that router by the other machine is always connected and never had any problem

I thought about blaming the wires (because they are very short about 2m from my PC to the router on the same table)
http://www.broadbandutopia.com/caandcaco.html

but because many things works like IRC and ping pushed me to drop this idea


> IT staff for further analysis.

> First suggestion would be, if other systems on the lan can
> access web sites without receiving this error, to swap IP addresses with one
> of them.

this will be the first thing I'll do when it happens again
when it happens again should I configure the DHCP on the router to give me a different IP or just set a different IP with ifconfig
Comment 3 Neil Horman 2009-01-12 10:35:30 EST
The TPLinux router is the mac oui that is the source of the ICMP messages, so that is almost certainly your culprit.  This could very easily be a per-protocol filter on that router (i.e. allowing all traffic through, but blocking http traffic).  In addition to the ip shifting, you might also try (if you have something available) setting up an ssh tunnel (using stunnel) toy our other machine, or to something beyond your router to see if masking the HTTP bits of the packet in an encryption layer gets around the firewall
Comment 4 Muayyad Alsadi 2009-01-12 17:24:56 EST
I have apache on the other machine and I was able to access it (no need for tunnle)

I'm suspecting forcedeth because when I restart my PC the problem vanishes

I tried to change the IP and the new IP had the same problem
Comment 5 Neil Horman 2009-01-12 18:55:50 EST
I will all but promise you that this isn't a local driver issue.  You don't get ICMP admin prohibited codes from a bad driver, especially when the source mac of those ICMP messages is from the default gateway.

Its possible that the filtering is tied to the mac address itself.  To test, try changing the MAC of your forcedeth card:
ifconfig <interface> hw ether <newmac>

Chose whatever mac you like, but be sure that:
1) it doesnt use the nvidia oui
2) it has the laa bit set
3) its not otherwise used on the local network.

If that doesn't get you through the router (while other hosts on your network make it through, then I'll start to think something is wrong on the system locally)
Comment 6 Muayyad Alsadi 2009-01-17 19:10:09 EST
> especially when the source mac of those ICMP messages is from the default gateway.


I'm not an expert but please look at the arrow of in screenshot I have just attached

I thought it's coming from the router because my computer sent a pocket with bad checksum

is this right ? I mean the bad checksum is in a pocket coming out from my PC and the router is complaining 

or is it the router sending me ICMP pockets but I have something wrong that case the checksum of the ICMP entering my PC to be corrupted

> I will all but promise you that this isn't a local driver issue. 
then could it be a hardware problem in my ethernet card ?


> try changing the MAC of your forcedeth card:
> ifconfig <interface> hw ether <newmac>

I'll try that next time
Comment 7 Muayyad Alsadi 2009-01-17 19:10:43 EST
Created attachment 329294 [details]
look at the arrow please
Comment 8 Neil Horman 2009-01-18 09:42:33 EST
updating summary
Comment 9 Neil Horman 2009-01-18 09:58:39 EST
No, its not right.  You're seeing incorrect checksums in your sent captured packets for exactly the reason that wireshark is telling you.  Forcedeth hardware supports tcp checksum offload, which means that the tcp data that you send has the hardware do all its checksums computation.  This in turn means the software need not touch the tcp layer checksum field.  Since you capture sent packets with wireshark prior to passing them to the hardware, you see garbage in the checksum field.  If you want to eliminate that, you can use ethtool to diable gso/tso/uso.  But thats really a waste of time.  packets with bad checksums are discarded.  Again, an ICMP administratively prohibited message means exactly what it says: The sending host (in this case the TPlink router) has some configuration aspect that tells it that packets from your failing host should not be allowed to send frames.  I looked at the TP-LINK site and most of their SOHO routers seem to support ip address filtering.  This is the feature that prevents access to the internet based on certain LAN ip address or ports.  Go check it and make sure that its marked as disabled.  Also, you may want to make sure its updated to the latest firmware.  I don't know exactly which model you have, but I looked at some of the changelogs and some of the fw packages seem to indicate some bugfixes that might pertain to your situation.
Comment 10 Muayyad Alsadi 2009-01-20 16:58:12 EST
first let me thank you for your attention and great help
you was very responsive

regarding the model it's TL-R402
and I have all the firewalls and filtering disabled

and I tried to change my PC IP and MAC


my eth0 mac is 00:1E:90:60:96:DC
I changed it to
ifconfig eth0 hw ether 00:11:5b:ba:2f:6a
ifconfig eth0 192.168.1.102 up
ifconfig eth0 hw ether 00:11:5b:ba:2f:6a
route add default gw 192.168.1.1

the new mac 00:11:5b:ba:2f:6a is the same mac as my brother's PC but the first byte in the serial is changed from 6f to 6a 
arp gives
Address                  HWtype  HWaddress           Flags Mask            Iface
pc2                      ether   00:11:5b:ba:2f:6f   C                     eth0
192.168.1.1              ether   00:1d:0f:73:23:88   C                     eth0

and it not work so I choose another address and another try

ifconfig eth0 192.168.1.2 up
ifconfig eth0 hw ether 00:11:5b:ba:2f:6a
route add default gw 192.168.1.1
route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0
default         192.168.1.1     0.0.0.0         UG    0      0        0 eth0

and it did not work
and in both cases ping works but browsing don't
I attached a small segment of wire shark showing that
sorry for bothering you, it really could be in the crippled hardware
Comment 11 Muayyad Alsadi 2009-01-20 17:00:02 EST
Created attachment 329512 [details]
wireshark log of trial2
Comment 12 Muayyad Alsadi 2009-01-20 17:00:31 EST
Created attachment 329513 [details]
wireshark log of trial3
Comment 13 Muayyad Alsadi 2009-01-20 17:01:09 EST
Created attachment 329514 [details]
screenshot of tplink firewall
Comment 14 Neil Horman 2009-01-21 06:46:52 EST
I'm sorry, I just realized what I think is going on.  I think I've been reading the tcpdumps backwards. Looking at the origional log, We see a session over http between peers 192.168.1.2 and 74.125.105.35.  Frame 1 shows your system sending a tcp SYN frame, and Frame 2 shows a SYN/ACK response from the remote peer.  However, frame 3 (the icmp message) is sourced from your host, not the default gateway (which would be the typical scenario).  Its not your router with the bad configuration, its your local system thats preventing this from occuring.  Looking at your new captures, its the same thing.  Your http session starts up fine, but as soon as you receive a response to the TCP handshake, you send an ICMP unreachable message and likely drop the frame in the stack.

So the question becomes, why?  Usually ICMP dest unreach messages are sent in response to routing failures, but if you don't have forwarding enabled on your system thats likely not a problem.  Specifically in the case of code 10 (admin prohibited) dest unreachable messages, that usually indicates a firewall rule on the local system that is specifically rejecting these packets.  You likely have an iptables rule matching on these frames that jumps to a REJECT target withthe --reject-with option set to icmp-host-prohibited.  I would suggest turning off the firewall on your local system (service iptables stop; iptables --flush).  If the problem goes away, then you need to look at your iptables config to see what is causing this rule to match, and fix that.

This certainly explains why changing your mac & ip address was having no effect on this behavior.
Comment 15 Muayyad Alsadi 2009-01-21 16:27:45 EST
OK I'll try that next time
my rules are simple

iptables-save 
# Generated by iptables-save v1.4.1.1 on Wed Jan 21 23:27:06 2009
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [54464:4809779]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Wed Jan 21 23:27:06 2009


and here it's 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 

I wonder how those are reached when there are 
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
before them
Comment 16 Neil Horman 2009-01-21 18:57:33 EST
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 

These rules are broken.  The packet thats reject is a SYN/ACK tcp frame  (the second step in the 3 way handshake for tcp on an http connection.  Since your the client there, the dport on the incomming packet is whatever the kernel assigned to you (something > 1024), so you wont match on the second rule above. The connection isn't established yet, and the connection is not RELATED yes (the connection is still in the NEW state), so you wont match on the first rule either.  Since you fall through both of those you reject the packet with a host unreachable message.  Kill the --dport 22 match and it should all work.

I'm closing this as NOTABUG.
Comment 17 Muayyad Alsadi 2009-01-21 20:32:16 EST
thanks, that was useful

but my problem is not yet solved even after I did
"service iptables stop; iptables --flush"

[root@pc1 ~]# service iptables stop; iptables --flush
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@pc1 ~]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

[root@pc1 thwab-2.2]# iptables-save
# Generated by iptables-save v1.4.1.1 on Thu Jan 22 03:22:03 2009
*filter
:INPUT ACCEPT [127:10600]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [99:7216]
COMMIT
# Completed on Thu Jan 22 03:22:03 2009
Comment 18 Thomas Woerner 2009-02-16 06:03:24 EST
Neil: Can you please tell me how "-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT" should match an outgoing http connect at all? Therefore I do not see the point here. And the other question is why this should happen only after some time.
Comment 19 Neil Horman 2009-02-16 07:13:35 EST
I don't mean to kill the entire rule, I mean you should remove the --dport specification only.  The problem is that in your rules, nothing catches http connections while in the NEW state, so you fall through to the icmp reject rule.
Comment 20 Thomas Woerner 2009-02-16 08:01:59 EST
If you remove the --dport option, then you do not need a firewall at all for tcp. 
BTW: If you have a look at comment #17 you will see that even having no firewall rules lead to the same problem.
Comment 21 Muayyad Alsadi 2009-02-16 10:06:14 EST
> you will see that even having no firewall rules lead to the same problem.
to be strict it did stop the problem
ie. I haven't put my PC without a firewall for long enough period of time
specially that the problem happens after long random times

> why this should happen only after some time.
yes, this is very strange, and because 

what I can't understand is "why the problem gets resolved with a restart (even with firewall rules)?"

please if you still think it's a firewall problem move it to s-c-firewall, because I have just accepted the default desktop rules (then I added ssh exception)
Comment 22 Thomas Woerner 2009-02-25 12:25:50 EST
I would say that this could be a buffer problem in the netfilter kernel code. This would explain why it only happens after some time.
Comment 23 Muayyad Alsadi 2009-02-25 12:42:54 EST
>> you will see that even having no firewall rules lead to the same problem.
>to be strict it did stop the problem

I meant to say, it did **not** stop the problem
ie. I haven't put my PC without a firewall for long enough period of time
specially that the problem happens after long random times


> I would say that this could be a buffer problem in the netfilter kernel code.
> This would explain why it only happens after some time.

I strong feeling it has something todo in the kernel
and because it's not a common problem, so I need to know in what way my system is so special to have this problem,

the first thing that come into my mind is to blame nvidia related things

Note You need to log in before you can comment on or make changes to this bug.