Bug 785772 - Logs filled by ICMPv6 RA: ndisc_router_discovery() failed to add default route
Logs filled by ICMPv6 RA: ndisc_router_discovery() failed to add default route
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: NetworkManager (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Dan Williams
Fedora Extras Quality Assurance
: Reopened
: 771130 806170 806932 807896 808203 811187 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-01-30 10:29 EST by Tomasz Torcz
Modified: 2012-10-09 16:43 EDT (History)
24 users (show)

See Also:
Fixed In Version: NetworkManager-0.9.4.0-7.git20120403.fc17
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-02 16:51:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
captured RAs (228 bytes, application/octet-stream)
2012-02-10 02:21 EST, Tomasz Torcz
no flags Details
simple rt stap script (99 bytes, application/octet-stream)
2012-02-10 15:42 EST, Neil Horman
no flags Details
updated stap file (491 bytes, application/octet-stream)
2012-02-13 14:54 EST, Neil Horman
no flags Details
radvd.conf (541 bytes, text/plain)
2012-02-17 03:39 EST, Tomasz Torcz
no flags Details
Syslog of affected client running 3.3-rc5+ (167.80 KB, text/plain)
2012-02-29 23:53 EST, Steven Noonan
no flags Details
Client tcpdump of wlan0 (16.00 KB, text/plain)
2012-02-29 23:53 EST, Steven Noonan
no flags Details
Server/router-side tcpdump of LAN interface eth1 (20.00 KB, text/plain)
2012-02-29 23:53 EST, Steven Noonan
no flags Details
in depth debug patch (6.15 KB, patch)
2012-03-02 16:06 EST, Neil Horman
no flags Details | Diff
dmesg with extra printk patch. (96.70 KB, application/octet-stream)
2012-03-05 21:46 EST, Steven Noonan
no flags Details
better dmesg with extra printks (93.66 KB, application/octet-stream)
2012-03-05 21:51 EST, Steven Noonan
no flags Details
debug patch to announce processes manually adding route from user space (1.23 KB, patch)
2012-03-07 16:33 EST, Neil Horman
no flags Details | Diff

  None (edit)
Description Tomasz Torcz 2012-01-30 10:29:30 EST
Description of problem:
With 3.3.x kernel, dmesg and in result log files, are filled by following message every few seconds:

ICMPv6 RA: ndisc_router_discovery() failed to add default route.

I have radvd running on a router and route seem to be added correctly:

[root@rawphys ~]# ip -6 r | grep defau
default via fe80::223:aeff:feb2:f43b dev p2p1  proto static  metric 1024 

[root@rawphys ~]# rdisc6 p2p1
Soliciting ff02::2 (ff02::2) on p2p1...

Hop limit                 :           64 (      0x40)
Stateful address conf.    :           No
Stateful other conf.      :           No
Router preference         :       medium
Router lifetime           :         1800 (0x00000708) seconds
Reachable time            :  unspecified (0x00000000)
Retransmit time           :  unspecified (0x00000000)
 Prefix                   : 2001:6a0:1c6::/64
  Valid time              :        86400 (0x00015180) seconds
  Pref. time              :        14400 (0x00003840) seconds
 Source link-layer address: 00:23:AE:B2:F4:3B
 from fe80::223:aeff:feb2:f43b



Version-Release number of selected component (if applicable):
kernel-3.3.0-0.rc1.git4.1.fc17.x86_64
Comment 1 Josh Boyer 2012-02-08 11:39:33 EST
I'm not immediately seeing anything that would have fixed this upstream yet, but are you still seeing this in dmesg with the 3.3-rc2-gitX kernels?
Comment 2 Tomasz Torcz 2012-02-09 03:52:54 EST
Yes:

# uname -a; dmesg | grep RA:
Linux lnx-test 3.3.0-0.rc2.git6.1.fc17.x86_64 #1 SMP Wed Feb 8 00:10:43 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[  443.456253] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Comment 3 Neil Horman 2012-02-09 15:15:34 EST
thats not the default route that its looking at, that route advertisement should have added a routes to reach the prefix advertized by that Router, and the router itself via the interface the ra was received on.  If you don't see those, thats the problem.  Can you please post the full contents of your ipv6 routing table, before and after you receive the RA here, and if possible can you include a small tcpdump showing the RA frames you are receving.  My first guess would be that radvd is advertizing frames using a bogus source address, which is causing the failure to add the route, but I won't know until I see it.  Thanks!
Comment 4 Tomasz Torcz 2012-02-10 02:21:41 EST
Created attachment 560793 [details]
captured RAs

Hi,

my configuration.

radvd.conf on router:
interface support
{
	AdvSendAdvert on;
	prefix 2001:6a0:1c6::/64
	{
		AdvOnLink on;
		AdvAutonomous on;
	};
};


As received on affected client:
# rdisc6 em1
Soliciting ff02::2 (ff02::2) on em1...

Hop limit                 :           64 (      0x40)
Stateful address conf.    :           No
Stateful other conf.      :           No
Router preference         :       medium
Router lifetime           :         1800 (0x00000708) seconds
Reachable time            :  unspecified (0x00000000)
Retransmit time           :  unspecified (0x00000000)
 Prefix                   : 2001:6a0:1c6::/64
  Valid time              :        86400 (0x00015180) seconds
  Pref. time              :        14400 (0x00003840) seconds
 Source link-layer address: 00:23:AE:B2:F4:3B
 from fe80::223:aeff:feb2:f43b


ip -6 r after receiving RA frames (:69 is also the router's address)

# ip -6 r
2001:6a0:1c6::69 via 2001:6a0:1c6::69 dev em1  metric 0 
    cache  hoplimit 255
2001:6a0:1c6::/64 dev em1  proto kernel  metric 256  expires 5681sec
unreachable fe80::/64 dev lo  proto kernel  metric 256  error -101
fe80::/64 dev em1  proto kernel  metric 256 
default via fe80::223:aeff:feb2:f43b dev em1  proto static  metric 1024 


captured RAs:
reading from file RAs.tcpdump, link-type EN10MB (Ethernet)
08:16:12.126039 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 8) lnx-test.local > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 8
08:16:12.128338 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::223:aeff:feb2:f43b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
	hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0ms, retrans time 0ms
	  prefix info option (3), length 32 (4): 2001:6a0:1c6::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
	    0x0000:  40c0 0001 5180 0000 3840 0000 0000 2001
	    0x0010:  06a0 01c6 0000 0000 0000 0000 0000
	  source link-address option (1), length 8 (1): 00:23:ae:b2:f4:3b
	    0x0000:  0023 aeb2 f43b

(also attached)


Router is certainly reachable:
# ping6 -I em1 -c 3 fe80::223:aeff:feb2:f43b
PING fe80::223:aeff:feb2:f43b(fe80::223:aeff:feb2:f43b) from fe80::218:8bff:fe15:be85 em1: 56 data bytes
64 bytes from fe80::223:aeff:feb2:f43b: icmp_seq=1 ttl=64 time=0.397 ms
64 bytes from fe80::223:aeff:feb2:f43b: icmp_seq=2 ttl=64 time=0.361 ms
64 bytes from fe80::223:aeff:feb2:f43b: icmp_seq=3 ttl=64 time=0.343 ms

--- fe80::223:aeff:feb2:f43b ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.343/0.367/0.397/0.022 ms

# ping6 -c 3 2001:6a0:1c6::69
PING 2001:6a0:1c6::69(2001:6a0:1c6::69) 56 data bytes
64 bytes from 2001:6a0:1c6::69: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 2001:6a0:1c6::69: icmp_seq=2 ttl=64 time=0.342 ms
64 bytes from 2001:6a0:1c6::69: icmp_seq=3 ttl=64 time=0.309 ms

--- 2001:6a0:1c6::69 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.309/0.560/1.031/0.333 ms





I don't know how to get IP6 route table before receiving RAs, as receive happens during boot up.
Comment 5 Neil Horman 2012-02-10 08:09:51 EST
shoot I apologize, somehow I had it in my head that radvd should be advertizing prefixes using its global address, not its link local address, and of course thats not the case.  That all looks correct above.  I'll have a stap script for you shortly (or a debug kernel that we can use to track down the problem here.  I'm starting to wonder if perhaps you have a second router advertising and its creating a conflict in the router table (i.e. adding a route that already exists against an alternate interface or address.  We'll find out shortly
Comment 6 Neil Horman 2012-02-10 15:42:46 EST
Created attachment 560978 [details]
simple rt stap script

ok, lets start small, this stap script will report the error code from ip6_route_add which must be failing during our gateway add at some subsequent point.  You'll need to install the kernel debuginfo packages for this, but it should tell us why some RA's are resulting in failed gw adds.  If you could run it and report the output back here that will give us a good start to figuring out whats going on here.  Thanks!
Comment 7 Tomasz Torcz 2012-02-13 09:45:58 EST
ip6_route_add returns 0
ip6_route_add returns 0
ip6_route_add returns -113

errno -113 is No route to host. Which is weird, as I have demonstrated in comment #4, gateway is reachable.
I've modified stap script to show $$parms$$ to see what address is being add. Unfortunately .u6_addr8 needs some kind of decoding:

ip6_route_add cfg={.fc_table=254, .fc_metric=0, .fc_dst_len=128, .fc_src_len=0, .fc_ifindex=2, .fc_flags=3, .fc_protocol=4, .fc_dst={.in6_u={.u6_addr8="�g�", .u6_addr16=[26564, ...], .u6_addr32=[13789124, ...]}}, .fc_src={.in6_u={.u6_addr8="", .u6_addr16=[0, ...], .u6_addr32=[0, ...]}}, .fc_prefsrc={.in6_u={.u6_addr8="", .u6_addr16=[0, ...], .u6_addr32=[0, ...]}}, .fc_gateway={.in6_u={.u6_addr8="�g�", .u6_addr16=[26584, ...], .u6_addr32=[13789144, ...]}}, .fc_expires=0, .fc_mx=0x0, .fc_mx_len=0, .fc_nlinfo={.nlh=0xffff8

ip6_route_add returns -113
Comment 8 Neil Horman 2012-02-13 11:07:57 EST
yeah, the gateway is reachable, but I think this is perhaps a different RA than the one you have for the router thats been added.  Converting the u6_addr16 to hex gives us 0x67c4, which doesn't fit as an ipv6 address in any way on any of the ip addresses that gets advitised in the RA you've shown.  Its also interesting that fc_nlinfo->nlh above shows what appears to be a stack address above.  I note that because in rt6_add_dflt_router, which is where we presumably are adding the default gw from ndisc_router_discovery, we explicitly set that value to NULL.  Did this return code repeat periodically, and was it accompanied by the reported "ICMPv6 RA: ndisc_router_discovery() failed to add default route." message?

I also note a few other things
1) fc flags are 0x3, which is (RTF_UP|RTF_GATEWAY).  rt_add_dflt_router sets several other flags.  Not sure how to reconcile that

2) The dst addr appears to have the same ipv6 address as the gateway addr.  Makes me think that somehow we are getting route advertisement from a local system via the network inadvertently.  Can you check your netstat output and proc output to see if any sockets or applications have subscribed, or are listening to the all routers ipv6 multicast address?

While you do that, I'll refine the stap script to get a better view of what the above parameters mean.  Thanks!
Comment 9 Nicolas Mailhot 2012-02-13 13:44:58 EST
If there is something simple people can run to test this, I have this problem too

grep -nr  'ndisc_router_discovery() failed to add default route' /var/log/messages* |wc -l
2700
Comment 10 Neil Horman 2012-02-13 14:54:23 EST
Created attachment 561653 [details]
updated stap file

Small update to the stap script to include the things that were previously discussed.

Additionally it occured to me that, if we do have another route daemon sending frames on this, or any interface in this box, we should be able to detect that pretty easily.  If you could add this rule to your ip6 tables configuration:

ip6tables -t filter -I INPUT 1 -p ipv6-icmp ! --source <link local source address of router> --icmpv6-type router-advertisement -j DROP

If you specify the link layer address of the router in the rule above (noting the negating !), this should drop all RA's that arrive at your box on any interface that has a source address other than that of your specified router.  If the ndisc messages cease, then we can be fairly certain that some system is erroneously sending RA's to your system, and we can start modifying the rule to identify the interface its arriving on, and capture the system in the act.
Comment 11 Tomasz Torcz 2012-02-14 04:28:04 EST
I have problem with this second script. I've corrected two typos at the beginning, but address printing is not good:

Pass 1: parsed user script and 85 library script(s) using 200528virt/23288res/2964shr kb, in 310usr/60sys/583real ms.
semantic error: unresolved arity-1 global array u6_addr32, missing global declaration?: identifier 'u6_addr32' at ra2.stp:14:27
        source: 	a = ntohl($gwaddr->in6_u.u6_addr32[0]);
                	                         ^
semantic error: unresolved arity-1 global array u6_addr32, missing global declaration?: identifier 'u6_addr32' at :15:27
        source: 	b = ntohl($gwaddr->in6_u.u6_addr32[1]);
                	                         ^
semantic error: unresolved arity-1 global array u6_addr32, missing global declaration?: identifier 'u6_addr32' at :16:27
        source: 	c = ntohl($gwaddr->in6_u.u6_addr32[2]);
                	                         ^
semantic error: unresolved arity-1 global array u6_addr32, missing global declaration?: identifier 'u6_addr32' at :17:27
        source: 	b = ntohl($gwaddr->in6_u.u6_addr32[d]);
                	                         ^
Pass 2: analyzed script: 3 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 385560virt/154320res/98012shr kb, in 1370usr/180sys/1698real ms
Comment 12 Neil Horman 2012-02-14 06:51:14 EST
dang it, thats a parser error from systemtap, it has a very hard time dealing with the '.' operator.  I'll talk to the maintainer about how to manage that.

In the interim, can you please try the ip6tables command I mentioned in comment 10?
Comment 13 Tomasz Torcz 2012-02-14 09:27:58 EST
Neil,

I've entered above iptables command but nothing have changed wrt to messages.
After few hours I put different rule in ip6tables -- one ending wiht -j LOG.
Logged entries are all the same:
IN=em1 OUT= MAC=33:33:00:00:00:01:00:23:ae:b2:f4:3b:86:dd SRC=fe80:0000:0000:0000:0223:aeff:feb2:f43b DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=96 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=134 CODE=0 

No other MAC/SRC is sending RAs in my network. (and SRC from those messages is ping6-able).
Comment 14 Neil Horman 2012-02-15 10:43:42 EST
hmm, ok, I'm at a loss then.  I don't really see why this would be happening.  I've gone to looking at my own working setup here to see what might be different, and I did notice one thing.  My route adverts have a second route info option in them, and I vaguely recall having to set this up to get things working.  Can you check your radvd.conf file on your router to see if you have this clause present:

route ::/0
{
  AdvRoutePreference high;
};

If you don't, please add it and see if the problem subsides.  That clause will clearly advertise a default gw, but I need to look through my notes to see why its required.
Comment 15 Tomasz Torcz 2012-02-16 10:41:33 EST
with "route ::/0" in radvd.conf, the problem on client persist.
Comment 16 Neil Horman 2012-02-16 15:04:41 EST
Can you post your radvd.conf file please?
Comment 17 Tomasz Torcz 2012-02-17 03:39:18 EST
Created attachment 563852 [details]
radvd.conf

It's right there in #c4, but I'm including it for completness.
Comment 18 Neil Horman 2012-02-17 11:49:15 EST
Ah, apologies, I should have seen that.

I notice two other discrepancies between our radvd config files:

1) your interface name is odd.  While its perfectly legitimate, I just want to make sure the interface you want you router to advert on is actually named support.

2) I enable AdvRouterAddr on my configuration, whereas you don't. You shouldn't need to enable this, but if you could try, that would give us a lead here

Possibly corresponding to item (2) above, I took a tcpdump of my ra's and the prefix information option on my net has the router address flag set, whereas yours does not.  It seems like they should be controlled separately, but its possible that that option inadvertently allows that bit to be set.
Comment 19 Tomasz Torcz 2012-02-20 03:33:03 EST
Neil,

1) yes, the interface is named "support", because it is connected to Technical Support department network. I used to name network interfaces with their functions.

2) I disabled AdvRouterAddr while trying to fix this issues before, it's a left over. Now I enabled it again, but it doesn't change anything, "ICMPv6 RA: ndisc_router_discovery() failed to add default route" still appears in logs.
Comment 20 Neil Horman 2012-02-20 07:23:39 EST
Would you please attach a sosreport of your router system and your client system?  I'm running out of ideas to specifically check, and would rather not send you running for them every time something occurs to me please.  With the sosreports I can poke around both systems without bothering you for every little file.  Thank you.
Comment 21 Neil Horman 2012-02-20 11:16:40 EST
A fwe things odd jumped out at me while looking at the server sos report

1) your sever has an em1 interface with a link local address of fe80::223:aeff:feb2:f43b/64.  This matches the link local interface address of your 'support' interface.  In fact they have the same mac address.  Is that intentional?  While two interfaces should be able to have the same link local address without issue, I'd be concerned if they were on the same network.  adding secondary ip addresses to a single link is the proscribed method of handling this type of thing.

2) The route table on your client shows this:
2001:6a0:1c6:0:223:aeff:feb2:f410 via 2001:6a0:1c6:0:223:aeff:feb2:f410 dev em1  metric 0     

thats a cached entry to reach your router.  Its odd because it contains the link local base 64 bits of the router link local address(es) noted in 1, but includes a global prefix rather than the link local prefix.  Nothing in the ifconfig output shows that any interface on the router has that global address, so I'm confused as to how (a) we have a route cache entry for that address, (b) how its reachable the system you provided the sosreport for, and (c) why we would try to add a route cache entry for that in the first place.

as I said, I'm not 100% certain that any of the directly relates to the problem, but it strikes me as fairly odd that two physical interfaces are sharing the same MAC and link local address.  

Canyou elaborate on why that link local addr sharing might be occuring?
Comment 22 Tomasz Torcz 2012-02-20 12:03:09 EST
ad 1) oh crap, I forgot about one (important) detail! "support" is in fact bridge interface. em1 is physical interface, and em1.1 (VLAN no 1) is bridged is slave to "support". Other slaves are vnet* interfaces from KVM.

# brctl show support
bridge name	bridge id		STP enabled	interfaces
support		8000.0023aeb2f43b	no		em1.1
							vnet0
							vnet1
							vnet2

I'm sorry, I should've remembered that:(

2) :f410 address is not router, it is another computer in this LAN segment. It is communicating with server (router), thus route is cached.

The router has addresses:  fe80::223:aeff:feb2:f43b/64 and 2001:6a0:1c6::69/64
Comment 23 Neil Horman 2012-02-20 14:20:05 EST
Ah, thank you, that explains alot when you take bridging into account.  I'm sorry, I missed the f410 rather then the f43b on the end of that cached route, and that had me confused. 

Although, looking further at the client, I noticed something else odd.  The default route in the client lists proto static rather than proto boot or proto kernel, as I would nominally expect.  proto static suggests an administratively added route, rather than one picked up via route advertisement.  The sosreport is unfortunately too scrubbed for me to see any more detail about that route.

Going back to some of our earlier work, ip6_route_add seems to be returning EHOSTUNREACH when trying to add the advertised route.  This can occur for one of 3 reasons:

1) If no link local path can be found to the router in the route table as it currently exists. (i.e. we need to be able to look up the link local address of the router and find a single interface that it exists on).

2) If the routing table already indicates that address of the router exists on an interface other than the interface represented by the ifindex provided in the fib_config structure.

2) If the route found in (1) already exists, but does not have the RTF_GATEWAY flag set on it.

Given that the client has only one public interface according to the sosreport, I think we can safely say checks 1 and 2 are passing.  I'm worried about item 3 however.  Its not in the sosreport, so can you please look at /proc/net/ipv6_route.  That will contain all the ipv6 routes for the system, including their flags.  You can either post it here or look for yourself.  The entry representing the default route should have in the second to last column 0x2 or-ed into the value. If it doesn't, then the gateway flag isn't set and we can safely say thats the cause of our problem.  We'll just have to track down whats adding that route prior to our recepit of the RA.
Comment 24 Steven Noonan 2012-02-29 05:36:21 EST
This bug entry has been idle for a while and I just noticed these messages starting to show up in the kernel log when I got 3.3-rc5 up and running. Hopefully I'm not just providing a worthless "me too" -- let me know what I can do to help get this issue resolved.

My configuration:

My router is running kernel 3.2.8.  As far as IPv4 is concerned, eth0 is public side, eth1 is private side. Using 6to4 to provide IPv6 to the LAN-side, with radvd configured thusly:

interface eth1 {
	MinRtrAdvInterval 3;
	MaxRtrAdvInterval 10;
	AdvLinkMTU 1280;
	AdvSendAdvert on;
	prefix 0:0:0:1::/64 {
		AdvOnLink on;
		AdvAutonomous on;
		AdvValidLifetime 86400;
		AdvPreferredLifetime 86400;
		Base6to4Interface eth0;
	};
};

My client is running 3.3-rc5-00102-gfcfcb67 (based on 891003ab plus a few self-maintained patches to get Linux booting under EFI on my two MacBooks).

Here's some [hopefully relevant] output from the client:

$ cat /proc/net/ipv6_route 
200243a1555000010000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 004c0001    wlan0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001    wlan0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 fe80000000000000021b21fffeb070ed 00000400 00000000 00000000 00000003    wlan0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0000000d 00200200       lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000005 80200001       lo
200243a155500001e2f847fffe31bec2 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000005 80200001       lo
fe80000000000000e2f847fffe31bec2 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001       lo
ff020000000000000000000000000001 80 00000000000000000000000000000000 00 ff020000000000000000000000000001 00000000 00000000 0000003b 01000001    wlan0
ff0200000000000000000000000000fb 80 00000000000000000000000000000000 00 ff0200000000000000000000000000fb 00000000 00000000 00000004 01000001    wlan0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001    wlan0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0000000d 00200200       lo

$ ip -6 route
2002:43a1:5550:1::/64 dev wlan0  proto kernel  metric 256  expires 84917sec
fe80::/64 dev wlan0  proto kernel  metric 256 
default via fe80::21b:21ff:feb0:70ed dev wlan0  proto static  metric 1024 

$ ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2002:43a1:5550:1:e2f8:47ff:fe31:bec2/64 scope global dynamic 
       valid_lft 84922sec preferred_lft 84922sec
    inet6 fe80::e2f8:47ff:fe31:bec2/64 scope link 
       valid_lft forever preferred_lft forever

Any more information I can provide?
Comment 25 Neil Horman 2012-02-29 06:20:28 EST
your /var/log/messages file, a sosreport and tcpdump taken on the internal interface of your router while your client boots would all be very helpful if possible.  Thank you.
Comment 26 Steven Noonan 2012-02-29 23:53:07 EST
Created attachment 566740 [details]
Syslog of affected client running 3.3-rc5+
Comment 27 Steven Noonan 2012-02-29 23:53:29 EST
Created attachment 566741 [details]
Client tcpdump of wlan0
Comment 28 Steven Noonan 2012-02-29 23:53:55 EST
Created attachment 566742 [details]
Server/router-side tcpdump of LAN interface eth1
Comment 29 Steven Noonan 2012-02-29 23:57:14 EST
First off, caveat: I'm running Ubuntu, and there doesn't seem to be an obvious sosreport package available anywhere. The build system for it is also broken on Debian. So I gave up on that. If there's something specific you need that sosreport would give you, let me know.

I do have a /var/log/syslog for you, along with client and server side tcpdump outputs (with -vvv).

I did notice something interesting, too. I cannot reproduce the issue on the client when connected via ethernet. It only happens when connected via 802.11. My wireless chipset (BCM4331) is newly supported by the 3.3 kernel, so perhaps there's some issues there too, but it's not really clear.
Comment 30 Tomasz Torcz 2012-03-01 07:54:10 EST
Hi,

sorry for delay, $DAYJOB demands didn't left me time for this. Anyway, I don't see 0x2 ored in any entry in routing tables (tables pasted below). 
In the meantin I created simple script dumping ipv6_routes content into text file during boot, every 0.1 second. It shows new routes added but without much context. Would running "ip route monitor" during boot be any help?

00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001       lo
200106a001c600000223aefffeb2f410 80 00000000000000000000000000000000 00 200106a001c600000223aefffeb2f410 00000000 00000001 0000000c 010c0001      em1
200106a001c600000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 004c0001      em1
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00200200       lo
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001      em1
00000000000000000000000000000000 00 00000000000000000000000000000000 00 fe800000000000000223aefffeb2f43b 00000400 00000000 00000000 00000003      em1
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000393 00200200       lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000000c 80200001       lo
200106a001c6000002188bfffe15be85 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000003cc 80200001       lo
fe8000000000000002188bfffe15be85 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000009e 80200001       lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001      em1
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000393 00200200       l
Comment 31 Neil Horman 2012-03-02 09:44:25 EST
no, thank you.  Looking at your tcpdumps I don't see anything out of place.  At this point I think the best thing to do is whip up a debug patch to figure out what exactly is failing inside ip6_route_add and work backwards from there.  I'll have such a patch later today.
Comment 32 Neil Horman 2012-03-02 16:06:15 EST
Created attachment 567164 [details]
in depth debug patch

Ok, I've instrumented the heck out of ip6_rotue_add, and hopefully this will give us additional insight as to whats going on here  I've attached the patch above, and started a kernel build here:

http://koji.fedoraproject.org/koji/taskinfo?taskID=3847813

if anyone doesn't want to build their own kernel.  Please run this kernel, reproduce the problem, and attach the resultant /var/log/messages file.  Thanks!
Comment 33 Tomasz Torcz 2012-03-05 05:15:12 EST
Neil,

could you please verify this debug patch was included in kernel build? I see no debug messages in dmesg.
Comment 34 Neil Horman 2012-03-05 07:03:08 EST
Hmm, thats odd, Its included in my local build of the rpm here, but not on the version uploaded to koji.  Let me see whats going on.
Comment 35 Neil Horman 2012-03-05 08:23:53 EST
http://koji.fedoraproject.org/koji/taskinfo?taskID=3852812
There you go, apparently my local build had some options set that included additional patches.  the above should work for you.
Comment 36 Steven Noonan 2012-03-05 21:46:46 EST
Created attachment 567803 [details]
dmesg with extra printk patch.

Attached is my log from running the kernel with the extra debug printks. I had to try to capture this log a couple times because a soft lockup occurred within a few seconds after this printout finished. I basically did a quick 'dmesg > log; sync' before I had to hard shutdown the machine.

I was curious why a patch which only adds printks would cause this, so I took a closer look and I noticed this:

-       return __ip6_ins_rt(rt, &cfg->fc_nlinfo);
+       printk(KERN_CRIT "GOING TO INSERT ROUTE\n");
+       err = __ip6_ins_rt(rt, &cfg->fc_nlinfo);
+       printk(KERN_CRIT"_ip6_ins_rt RETURNED %d\n", err);

There's no return, so it just falls through to the error path and then things go totally bonkers. Oops. Hopefully this issue doesn't invalidate the log. I'll rebuild with a 'return err;' added after the 2nd printk there and see if there's a significant difference.
Comment 37 Steven Noonan 2012-03-05 21:51:21 EST
Created attachment 567804 [details]
better dmesg with extra printks

OK, yes, it was definitely adversely affected. Better log attached.
Comment 38 Steven Noonan 2012-03-05 22:13:28 EST
Heh. Looks like __ip6_ins_rt is returning:

include/asm-generic/errno-base.h:#define        EEXIST          17

Probably an error that should just be eaten by ip6_route_add. I'm curious why it isn't, in fact. Thoughts?
Comment 39 Neil Horman 2012-03-06 06:56:58 EST
Thank you steven.  Not sure why I didnt' see that when I ran it here.

Regardless, that error is eaten, its just eaten by the calling function, in this case rt6_add_dflt_router.  So this tells us that:

ip6_route_add fails in rt_add_dflt_router because the route already exists, but that error gets swallowed (thats good)

the return from rt6_add_dflt_router (which is generated by rt6_get_dflt_router) is an error code, which suggests that the route cant be found.


Ok, one more debug kernel comming up shortly
Comment 40 Neil Horman 2012-03-06 11:47:03 EST
Actually, I think I see the problem (or at least part of it).  Heres a snippet frmo the logs:


[   13.503337] IN IP6_ROUTE_ADD
[   13.503367] SRC = ::
[   13.503378] DST = ::
[   13.503389] FLAGS = 3
[   13.503399] DST_LEN = 0
[   13.503418] SRC_LEN = 0
[   13.503429] INDEX = 3
[   13.503439] Checking lengths
[   13.503451] CHECKING INDEX
[   13.503464] FOUND DEVICE
[   13.503475] FOUND IDEV
[   13.503486] SET METRIC to 1024
[   13.503500] TABLE IS ffff8804597ef7e0
[   13.503516] rt = ffff880456584780
[   13.503530] SETTING ROUTE EXPIRATION TO 0
[   13.503546] CURRENT JIFFIES IS 4294881279
[   13.503563] THIS IS A FORWARDING ROUTE
[   13.503579] THIS IS A GATEWAY ROUTE
[   13.503594] GWA_TYPE = 33
[   13.503606] FINALLY, DEV is ffff88044e15d000
[   13.503624] WE STILL HAVE A DEV
[   13.503637] BINDING GW ROUTE TO NEIGHBOR
[   13.503654] rt6_bind_neighbour RETURNED 0
[   13.503671] GOING TO INSERT ROUTE
[   13.503690] _ip6_ins_rt RETURNED 0

[   16.907667] IN IP6_ROUTE_ADD
[   16.907688] SRC = ::
[   16.907699] DST = ::
[   16.907709] FLAGS = 450003
[   16.907721] DST_LEN = 0
[   16.907731] SRC_LEN = 0
[   16.907742] INDEX = 3
[   16.907752] Checking lengths
[   16.907765] CHECKING INDEX
[   16.907777] FOUND DEVICE
[   16.907789] FOUND IDEV
[   16.907799] SET METRIC to 1024
[   16.908912] TABLE IS ffff8804597ef7e0
[   16.910026] rt = ffff88044cfb08c0
[   16.911166] SETTING ROUTE EXPIRATION TO 4294882302
[   16.912262] CURRENT JIFFIES IS 4294882302
[   16.913375] THIS IS A FORWARDING ROUTE
[   16.914492] THIS IS A GATEWAY ROUTE
[   16.914493] GWA_TYPE = 33
[   16.914494] FINALLY, DEV is ffff88044e15d000
[   16.914494] WE STILL HAVE A DEV
[   16.914495] BINDING GW ROUTE TO NEIGHBOR
[   16.914496] rt6_bind_neighbour RETURNED 0
[   16.914497] GOING TO INSERT ROUTE
[   16.914498] _ip6_ins_rt RETURNED -17
[   16.914500] ICMPv6 RA: ndisc_router_discovery() failed to add default route.

That last line represents the first instance of the ICMPv6 error you're seeing.  Inthe message set above, immediately prior to the failing one we see a default gateway being added to the routing table with flags = 0x3 (RTF_UP | RTF_GATEWAY). Nothing in the kernel adds a route with just those flags.  I'm guessing that first (successful) route addition is the result of something in user space doing the add (ip -6 route add default gw  would add a route with these flags).  So if we have a route to destination :: already, when we get the addrconf route advert, we try to add the same route again, but it fails with -EEXIST, which is correct behavior, since we already have this route).  Then rt6_add_deflt_router calls  rt6_get_dflt_router to find the appropriate route entry to return.  Unfortunately rt6_get_dflt_router searches for a route with RTF_ADDRCONF set, as it expected the route to be added by the rotue advert receive routine (ndisc_router_discovery).  Since it wasn't however, we don't find any route at all, and you get the above error.

Now, I still don't understand exactly why moving to the 3.3 kernel might be causing this, but I think the above is pretty clear.

So the questions that remain:
1) Who is adding the non-addrconf produced default route
2) what do we do about it?

(1) Can be determined pretty easily I think, we can just trap on the netlink and ioctl paths that add ipv6 routes and dump out the pid and parent pid of the process executing there, which should give us a good hint as to who is doing this.  (assuming you don't already have a guess as to whats going on there).

(2) This is based on the answer to (1).  If you want to accept the default route provided by the router advert, then we need to follow up on (1), figure out who's adding that route from user space, and stop it.  If you want to let the manual route add proceede, then the solution is this:
echo 0 > /proc/sys/net/ipv6/conf/<if>/accept_ra_defrtr

Which path would you like to follow up on?
Comment 41 Steven Noonan 2012-03-06 18:42:42 EST
Okay. I don't know what could be responsible for #1. Let's follow up on that. If I ever manage to get systemtap to work, it should be easy to trace... Or do you have a patch which can dump that in dmesg?
Comment 42 Neil Horman 2012-03-06 19:56:08 EST
we can go either way.  I'll write up a patch for you in the AM since it sounds like stap is falling over for you
Comment 43 Neil Horman 2012-03-07 16:33:45 EST
Created attachment 568428 [details]
debug patch to announce processes manually adding route from user space

Heres a new debug patch, and its building here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3865771

boot that on your system, and it should dump a bunch of info in your message log any time a process on your system tries to manually add a route.
Comment 44 Steven Noonan 2012-03-07 21:21:57 EST
Well, this shouldn't be much of a surprise:

[   13.046772] SOMEONE IS ADDING AN IPV6 ROUTE VIA NETLINK
[   13.046815] SOURCE is ::
[   13.046831] DST is ::
[   13.046842] PROCESS IS 1176(NetworkManager)
[   13.046860] PARENT PROCESS IS 1(init)

So I guess the question is... should NetworkManager be doing this, or deferring to the kernel when possible?
Comment 45 Tomasz Torcz 2012-03-08 03:36:35 EST
And I got this kernel also, but I'm not sure what I'm seeing. Here's how the surrounding of ICMPv6 message looks like in dmesg:

[  493.728037] SOMEONE IS ADDING AN IPV6 ROUTE VIA NETLINK
[  493.728049] SOURCE is ::
[  493.728054] DST is ::
[  493.728058] PROCESS IS 888(NetworkManager)
[  493.728063] PARENT PROCESS IS 1(systemd)
[  556.816315] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[  622.072519] SOMEONE IS ADDING AN IPV6 ROUTE VIA NETLINK
[  622.072528] SOURCE is ::
[  622.072533] DST is a48c:901::505d:601:0:0
[  622.072536] PROCESS IS 888(NetworkManager)
[  622.072540] PARENT PROCESS IS 1(systemd)

And second occurence:

[  751.122535] SOMEONE IS ADDING AN IPV6 ROUTE VIA NETLINK
[  751.122545] SOURCE is ::
[  751.122548] DST is ::
[  751.122554] PROCESS IS 888(NetworkManager)
[  751.122558] PARENT PROCESS IS 1(systemd)
[  827.553674] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[  879.937303] SOMEONE IS ADDING AN IPV6 ROUTE VIA NETLINK
[  879.937312] SOURCE is ::
[  879.937317] DST is 6423:401::705c:601:0:0
[  879.937321] PROCESS IS 888(NetworkManager)
[  879.937324] PARENT PROCESS IS 1(systemd)

  But default route wasn't added by NM:
# ip -6 r | grep default
default via fe80::223:aeff:feb2:f43b dev em1  proto static  metric 1024 
# dmesg | grep -i aeff
#

So my observations:
1) ICMPv6 RA do not coincident with NM route table manipulations (judging from timestamps)
2) NM is adding some strange routes (6423:401? a48c:901? what kind of prefixes are that?)
Comment 46 Tomasz Torcz 2012-03-08 03:38:24 EST
Also, log file shows some problems on NM side:

NetworkManager[888]: NetworkManager[888] <warn> Failed to add route Netlink Error (errno = No route to host) 
NetworkManager[888]: <warn> Failed to add route Netlink Error (errno = No route to host) 
 kernel: [ 1525.887376] SOMEONE IS ADDING AN IPV6 ROUTE VIA NETLINK 
 kernel: [ 1525.887385] SOURCE is :: 
 kernel: [ 1525.887390] DST is 7464:601::705c:601:0:0 
 kernel: [ 1525.887394] PROCESS IS 888(NetworkManager) 
 kernel: [ 1525.887398] PARENT PROCESS IS 1(systemd) 
NetworkManager[888]: NetworkManager[888] <error> [1331195792.465755] [nm-system.c:595] nm_system_apply_ip6_config(): (em1): failed to set IPv6 route: Netlink Error (errno = No route to host)
Comment 47 Neil Horman 2012-03-08 06:50:14 EST
Its not a question of should NetworkManager be doing this, its a question of why its configured to do this.  Each of you should go go to the NetworkManager applet, click the manage connection button, open the connection you're using, click the ipv6 tab, and look at the routes page for the ipv6 settings.  If you see any routes there, they need to be removed.

Tomasz, I'm not sure whats going on with your wierd addresses or failures in the NM path yet, but lets just start with getting NM to not add routes at all, as it doesn't need to here.
Comment 48 Tomasz Torcz 2012-03-08 07:01:12 EST
NM settings for em1 IPv6 are "Automatic". Routes are empty. NM is using system connection, ifcfg-em1 contains:

# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
DEVICE=em1
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
DEFROUTE=yes
IPV6INIT=yes
NAME="System em1"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6_FAILURE_FATAL=no
HWADDR=00:18:8B:15:BE:85
PEERDNS=yes
PEERROUTES=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
LAST_CONNECT=1301387166
Comment 49 Neil Horman 2012-03-08 13:40:37 EST
Well, thats a problem, since we have evidence that NM is adding routes that, according to your above configuration it shouldn't be.  Adding the NM maintainer to see if he has any insight as to whats going on.

Summary for Dan: We seem to have some fedora systems in which NetworkManager is adding routes to the ipv6 route table despite not being configured to do so.  The result is the autoconfigured routes that match the source and dest values (the default gw in this case), are not adding properly.  Do you have any clue as to whats going on here? comments 43-46 show debug kernel output indicating that NM is adding these routes
Comment 50 Tomasz Torcz 2012-03-20 06:49:02 EDT
The problem seemingly stopped occuring with NetworkManager 0.9.3.995-0.6. It probably related to this entry in Changelog for 0.9.3.995-0.5:

- core: fix IPv6 addressing on newer kernels
Comment 51 Neil Horman 2012-03-20 09:08:35 EDT
Yes, it appears there are several ipv6 fixes in the NetworkManager git tree, and Fedora just did a wholesale update.  Ok, I'll close this as currentrelease then
Comment 52 Trever Adams 2012-04-02 11:54:48 EDT
I am afraid the NM updates done here did NOT fix this problem for me.
Comment 53 Sam Hegarty 2012-04-07 16:27:30 EDT
Also did not fix this problem for me and I'm running NetworkManager 0.9.4-2.git20120403 from Testing.
Comment 54 Pavel Šimerda (pavlix) 2012-04-08 13:23:25 EDT
I'm another person experiencing lots of these messages. I'm using NetworkManager-0.9.4.0-1.git20120328.fc17.x86_64 with kernel-3.3.0-1.fc17.x86_64.
Comment 55 Neil Horman 2012-04-09 06:50:20 EDT
Ok, based on commentary here, and in several other bugs, this is clearly an NM problem.  Its adding ipv6 routes when it shouldn't be.  Reassigning to the NM maintainer.

As a workaround, you can set the ipv6 mode for your interfaces in NM to 'ignore' or 'disable'.  That should prevent NM from doing anything W.R.T to ipv6 on those interfaces at all, allowing SLAAC to operte normally in the kernel
Comment 56 Dan Winship 2012-04-18 15:35:03 EDT
Neil: NM sets the default route itself, because the individual interfaces don't always know the right policy for that. (Eg, you may want your default route to go through a VPN, or you may want to ensure that the default route goes through eth1, even if eth0 comes up first, etc.)

Now, NM could set accept_ra_defrtr on each interface (and maybe it should), but the current kernel behavior still seems broken anyway; setting a default IPv6 route by hand completely breaks router discovery on all accept_ra_defrtr==1 interfaces, because ndisc_router_discovery() bails out after failing to update the default route.

It seems like the code there ought to skip over the default-route-setting code if there is *any* default IPv6 route, rather than only if there's an autoconfigured IPv6 route...
Comment 57 Dan Winship 2012-04-18 15:51:48 EDT
actually... if we set accept_ra_defrtr to 0, and the RA doesn't include any other explicit routes, then is there any way for userland to even find the router address?
Comment 58 Dan Winship 2012-04-19 10:37:42 EDT
*** Bug 808203 has been marked as a duplicate of this bug. ***
Comment 59 Dan Winship 2012-04-19 10:38:47 EDT
*** Bug 753482 has been marked as a duplicate of this bug. ***
Comment 60 Dan Winship 2012-04-19 10:42:27 EDT
NM packages at http://koji.fedoraproject.org/koji/taskinfo?taskID=4005602 should work around this (for users with a single IPv6 interface), by not deleting-and-re-adding the default route if it's already correct.
Comment 61 Neil Horman 2012-04-19 10:57:41 EDT
in response to comment 56, yes, thats exactly what NM should do.  If it plans on setting a static default route, it should prevent the kernel from setting any additional default routes received via route advertisment.  Thats the only way the kernel can know what to do when configuring addresses via SLAAC.  If you want to use SLAAC for automatic ipv6 addressing but still set your default route manually, then the correct thing to do is to clear the accept_ra_defrtr flag on the interface prior to bringing the interface up.
Comment 62 Dan Winship 2012-04-19 14:26:43 EDT
Hm. Setting accept_ra_defrtr won't work because then we have no way of knowing the router's address when we decide we do want to make that the default route.

But I found a better fix anyway; instead of replacing the kernel's default route, we just add an additional default route of our own with a lower metric. Now we've got the default we want, so we're happy, and the kernel still has its autogenerated default, so it's happy.
Comment 63 Neil Horman 2012-04-19 15:11:54 EDT
If you're using SLAAC, then you really don't want to have someone setting a default route other than the one that the router advertisement provides for you.  If you do, for whatever reason, want to have such a route set, then you need to know what that default route should be via other means (pre-shared static configuration, snooping route advertisements, etc).  The solution you've found is a good one for that as it provides a 'default' default route that the kernel can supercede, which should work well
Comment 64 Dan Winship 2012-04-19 17:12:52 EDT
*** Bug 808203 has been marked as a duplicate of this bug. ***
Comment 65 Dan Winship 2012-04-19 17:14:21 EDT
*** Bug 753482 has been marked as a duplicate of this bug. ***
Comment 66 Dan Winship 2012-04-19 17:16:42 EDT
*** Bug 811187 has been marked as a duplicate of this bug. ***
Comment 67 Dan Winship 2012-04-19 17:18:04 EDT
*** Bug 806932 has been marked as a duplicate of this bug. ***
Comment 68 Dan Winship 2012-04-19 17:18:56 EDT
*** Bug 806170 has been marked as a duplicate of this bug. ***
Comment 69 Dan Winship 2012-04-19 17:20:58 EDT
*** Bug 771130 has been marked as a duplicate of this bug. ***
Comment 70 Fedora Update System 2012-04-19 18:02:47 EDT
NetworkManager-0.9.4-5.git20120403.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.4-5.git20120403.fc17
Comment 71 Pavel Šimerda (pavlix) 2012-04-19 18:43:36 EDT
The new version won't update version 1:0.9.4.0-1.git20120328.fc17.
Comment 72 Dave Jones 2012-04-19 18:56:56 EDT
This is also affecting F16 now that we have a 3.3 kernel there.
Comment 73 Pavel Šimerda (pavlix) 2012-04-19 19:29:22 EDT
NetworkManager-0.9.4-5.git20120403.fc17.x86_64 (from koji) seems to work well.
Comment 74 Fedora Update System 2012-04-20 02:03:05 EDT
Package NetworkManager-0.9.4-5.git20120403.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing NetworkManager-0.9.4-5.git20120403.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-6266/NetworkManager-0.9.4-5.git20120403.fc17
then log in and leave karma (feedback).
Comment 75 Fedora Update System 2012-04-20 10:28:52 EDT
NetworkManager-0.9.4.0-6.git20120403.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.4.0-6.git20120403.fc17
Comment 76 Dan Winship 2012-04-20 10:50:30 EDT
(new package is the same as the previous one, it just fixes the version number snafu)
Comment 77 David Woodhouse 2012-04-20 11:05:30 EDT
(In reply to comment #76)
> (new package is the same as the previous one, it just fixes the version number
> snafu)

Doesn't fix it very well for me... it still says 'fc17' not 'fc16'...
Comment 78 Pavel Šimerda (pavlix) 2012-04-20 12:52:03 EDT
Thanks!

@David: I was actually complainig about the fc17 version and I thought the fc16 version works fine (but I can't test).
Comment 79 Bill C. Riemers 2012-04-20 14:41:44 EDT
Hmmm.  I'm not sure how I ended-up on the Cc for this bug.  Did it use to have another title.  In any case in regards to Fedora 16, I see the following in my log files o:

[1226]: <info> -----------------------------------------
Apr 15 20:56:43 briemersw NetworkManager[1226]: <warn> could not spawn process '/etc/init.d/nscd condrestart': Failed to execute child process "/etc/init.d/nscd" (No such file or directory)
Apr 15 20:56:43 briemersw NetworkManager[1226]: NetworkManager[1226]: <warn> could not spawn process '/etc/init.d/nscd condrestart': Failed to execute child process "/etc/init.d/nscd" (No such file or directory)

As such nscd, what ever that is never runs, Sometimes this message occurs repeatedly, sometimes it only occurs once every ten minutes or so.  Either way it is not frequent enough to be the primary message filling up my log file.
Comment 80 Pavel Šimerda (pavlix) 2012-04-20 14:52:37 EDT
Bill: Bug 753482 was marked as a duplicate of this bug for a short while. This is how you got here :).
Comment 81 Fedora Update System 2012-04-20 15:10:10 EDT
NetworkManager-0.9.4-3.git20120403.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.4-3.git20120403.fc16
Comment 82 David Woodhouse 2012-04-20 18:06:54 EDT
NetworkManager-0.9.4-3.git20120403.fc16 still seems to fail here, although not *every* time the RA lifetime expires.

Apr 20 22:39:53 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 22:39:53 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 22:39:56 shinybook dnsmasq[12780]: reading /etc/resolv.conf
Apr 20 22:39:56 shinybook dnsmasq[12780]: using nameserver 2001:8b0:10b:1:21d:7dff:fe04:dbe2#53
Apr 20 22:39:56 shinybook dnsmasq[12780]: using nameserver 90.155.92.214#53
Apr 20 22:39:56 shinybook dnsmasq[12780]: using nameserver 90.155.92.209#53
Apr 20 22:49:54 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 22:49:54 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 22:49:56 shinybook dnsmasq[12780]: reading /etc/resolv.conf
Apr 20 22:49:56 shinybook dnsmasq[12780]: using nameserver 2001:8b0:10b:1:21d:7dff:fe04:dbe2#53
Apr 20 22:49:56 shinybook dnsmasq[12780]: using nameserver 90.155.92.214#53
Apr 20 22:49:56 shinybook dnsmasq[12780]: using nameserver 90.155.92.209#53
Apr 20 22:54:54 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 22:54:54 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 22:54:56 shinybook dnsmasq[12780]: reading /etc/resolv.conf
Apr 20 22:54:56 shinybook dnsmasq[12780]: using nameserver 2001:8b0:10b:1:21d:7dff:fe04:dbe2#53
Apr 20 22:54:56 shinybook dnsmasq[12780]: using nameserver 90.155.92.214#53
Apr 20 22:54:56 shinybook dnsmasq[12780]: using nameserver 90.155.92.209#53
Apr 20 22:57:21 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 22:57:21 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 22:57:21 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 22:57:21 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 22:57:22 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 22:57:22 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 22:57:23 shinybook dnsmasq[12780]: reading /etc/resolv.conf
Apr 20 22:57:23 shinybook dnsmasq[12780]: using nameserver 2001:8b0:10b:1:21d:7dff:fe04:dbe2#53
Apr 20 22:57:23 shinybook dnsmasq[12780]: using nameserver 90.155.92.214#53
Apr 20 22:57:23 shinybook dnsmasq[12780]: using nameserver 90.155.92.209#53
Apr 20 23:03:25 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv4 routing and DNS.
Apr 20 23:03:25 shinybook NetworkManager[32007]: <info> Policy set 'Auto Baythorne Wavelan' (wlan0) as default for IPv6 routing and DNS.
Apr 20 23:04:29 shinybook dnsmasq[12780]: reading /etc/resolv.conf
Apr 20 23:04:29 shinybook dnsmasq[12780]: using nameserver 2001:8b0:10b:1:21d:7dff:fe04:dbe2#53
Apr 20 23:04:29 shinybook dnsmasq[12780]: using nameserver 90.155.92.214#53
Apr 20 23:04:29 shinybook dnsmasq[12780]: using nameserver 90.155.92.209#53
Apr 20 23:04:41 shinybook NetworkManager[32007]: <info> (wlan0): device state change: activated -> failed (reason 'ip-config-unavailable') [100 120 5]
Apr 20 23:04:41 shinybook NetworkManager[32007]: <warn> Activation (wlan0) failed for access point (Baythorne Wavelan)
Comment 83 Fedora Update System 2012-04-23 18:41:58 EDT
NetworkManager-0.9.4.0-7.git20120403.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.4.0-7.git20120403.fc17
Comment 84 Fedora Update System 2012-04-27 01:57:56 EDT
NetworkManager-0.9.4-3.git20120403.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 85 Jirka Klimes 2012-05-02 02:35:37 EDT
*** Bug 807896 has been marked as a duplicate of this bug. ***
Comment 86 Trever Adams 2012-05-02 04:56:05 EDT
The latest NetworkManager in FC16 seems to work. In FC17, I am not sure it works, but my wireless connection never connects until service NetworkManager restart.

FC17 is NOT solved.
Comment 87 Pavel Šimerda (pavlix) 2012-05-02 08:54:00 EDT
F17 works for me.

NetworkManager-0.9.4.0-6.git20120403.fc17.x86_64
kernel-3.3.4-1.fc17.x86_64

Lenovo Thinkpad x61

00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03)
03:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection (rev 02)
Comment 88 Dan Winship 2012-05-02 09:22:54 EDT
(In reply to comment #86)
> The latest NetworkManager in FC16 seems to work. In FC17, I am not sure it
> works, but my wireless connection never connects until service NetworkManager
> restart.

That's a different bug, and a fix for that should be coming down the pipeline soon.
Comment 89 Fedora Update System 2012-05-02 16:51:20 EDT
NetworkManager-0.9.4.0-7.git20120403.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 90 Kyle Brantley 2012-06-03 22:59:57 EDT
I don't think that this is NM related. I intentionally uninstalled NM as I require some very involved /etc/sysconfig/network-scripts/ifcfg-* configuration that NM doesn't support, and I'm being destroyed with these messages.

[root@home-router ~]# cat /etc/fedora-release
Fedora release 17 (Beefy Miracle)
[root@home-router ~]# uname -a
Linux home-router.<redacted>.com 3.3.7-1.fc17.x86_64 #1 SMP Mon May 21 22:32:19 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@home-router ~]# rpm -qa | grep Network
[root@home-router ~]#
[root@home-router ~]# tail /var/log/messages
Jun  3 20:52:37 home-router kernel: [24918.472045] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:40 home-router kernel: [24921.440603] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:43 home-router kernel: [24924.409137] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:46 home-router kernel: [24927.377695] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:49 home-router kernel: [24930.346235] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:52 home-router kernel: [24933.315656] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:56 home-router kernel: [24937.094194] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:52:59 home-router kernel: [24940.073743] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:53:02 home-router kernel: [24943.031296] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
Jun  3 20:53:05 home-router kernel: [24946.159889] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[root@home-router ~]# date
Sun Jun  3 20:53:08 MDT 2012
Comment 91 Neil Horman 2012-06-04 07:00:46 EDT
The problem with network manager was that it was, IIRC, trying to add static routes for ipv6 while not having disabled SLAAC on interfaces it was managing.  If you're static setup adds static routes and doesn't disable SLAAC via /proc, then you'll get the same behavior, even without NM installed.

Note You need to log in before you can comment on or make changes to this bug.