Bug 1008104 - NetworkManager 0.9.9.0-11.git20130913.fc21 doesn't stay connected on ipv6 network
NetworkManager 0.9.9.0-11.git20130913.fc21 doesn't stay connected on ipv6 net...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: NetworkManager (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Pavel Šimerda (pavlix)
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-14 15:12 EDT by Kevin Fenzi
Modified: 2013-10-09 10:52 EDT (History)
4 users (show)

See Also:
Fixed In Version: NetworkManager-0.9.9.0-14.git20131003.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-09 10:52:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
journalctl -b -u NetworkManager (54.67 KB, text/plain)
2013-09-20 19:22 EDT, Kevin Fenzi
no flags Details
debug enabled logs (189.85 KB, text/plain)
2013-09-24 15:16 EDT, Kevin Fenzi
no flags Details
nmcli con show conf antidisestablishmentarianism (2.21 KB, text/plain)
2013-09-24 15:17 EDT, Kevin Fenzi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
GNOME Desktop 709230 None None None Never

  None (edit)
Description Kevin Fenzi 2013-09-14 15:12:01 EDT
With 0.9.9.0-11.git20130913.fc21 I see nm-applet show that it's trying to connect, I see it connect to my ipv4 wireless, but then it disconnects and just cycles endlessly.

If I change the ipv6 setting from 'automatic' to 'ignore' it connects fine to ipv4 and works. 

These seem to be possibly related: 

Sep 14 12:57:32 jelerak.scrye.com NetworkManager[15333]: <info> Activation (wlp3s0) Stage 5 of 5 (IP
v6 Commit) complete.
Sep 14 12:57:32 jelerak.scrye.com NetworkManager[15333]: (NetworkManager:15333): GLib-CRITICAL **: g_variant_is_object_path: assertion 'string != NULL' failed
Sep 14 12:57:32 jelerak.scrye.com NetworkManager[15333]: marshal_object_path: assertion 'g_variant_is_object_path (path)' failed
Sep 14 12:57:32 jelerak.scrye.com NetworkManager[15333]: failed to marshal parameter 1 for signal PropertiesChanged
Sep 14 12:57:32 jelerak.scrye.com NetworkManager[15333]: <info> (wlp3s0): device state change: failed -> disconnected (reason 'none') [120 30 0]

Happy to attach full journal or gather other information.
Comment 1 Jirka Klimes 2013-09-17 13:29:03 EDT
It might help if you attach the logs showing the whole connection activation attempt.
Also, can you list the details of the connection you are activating:
$ nmcli con show conf "name of your connection"
Comment 2 Kevin Fenzi 2013-09-20 19:22:11 EDT
Created attachment 800802 [details]
journalctl -b -u NetworkManager

Here's journalctl -b -u NetworkManager, let me know if you would like any more/further.
Comment 3 Jirka Klimes 2013-09-23 13:00:00 EDT
The problematic lines are:
Sep 20 17:15:00 jelerak.scrye.com NetworkManager[1040]: <error> [1379718900.433122] [rdisc/nm-lndp-rdisc.c:184] send_rs(): (wlp3s0): cannot send router solicitation: -99.
and
Sep 20 17:15:17 jelerak.scrye.com NetworkManager[1040]: refresh_object: assertion 'kernel_object' failed

-99 is an errno meaning:
#define EADDRNOTAVAIL   99      /* Cannot assign requested address */

It would help if you can switch on DEBUG mode for NM, so that we can see more verbose logs:
$ sudo nmcli g l level DEBUG
and when you are done, set back to normal
$ sudo nmcli g l level INFO

Please grab the output of
$ nmcli con show conf antidisestablishmentarianism

What are the versions of kernel and libnl3 ?
Comment 4 Kevin Fenzi 2013-09-24 15:16:25 EDT
Created attachment 802432 [details]
debug enabled logs

Here's the logs after a boot where I set debugging.
Comment 5 Kevin Fenzi 2013-09-24 15:17:06 EDT
Created attachment 802433 [details]
nmcli con show conf antidisestablishmentarianism
Comment 6 Kevin Fenzi 2013-09-24 15:17:38 EDT
kernel-3.12.0-0.rc2.git0.1.fc21.x86_64
libnl3-3.2.22-2.fc21.x86_64
Comment 7 Dan Williams 2013-09-26 14:03:06 EDT
I ran across somebody on IRC the other day with this same problem.  It turned out his radvd options were the culprit, I think:

(12:49:09 PM) prahal: dcbw: sorted out the issue from yesterday thanks to your debug patch . On the radvd box I add prefix <...>::1/64 (ending up as needle) instead of prefix <...>::/64 (which was what was in the haystack), thanks

The problem is that the NDP-calculated route that NetworkManager adds to the kernel gets changed somehow by the kernel or libnl3, and thus after adding the route when NetworkManager asks the kernel for the version the kernel is actually using, they don't match (the refresh_object() problem).

Could people with this problem check what their IPv6 router is actually broadcasting for a prefix?
Comment 8 Kevin Fenzi 2013-09-26 16:15:06 EDT
Bingo. Exactly the problem here. ;) 

ipv6 now works fine with the new NM. 

i am getting "Could not start connection: wrong plugin state 3" from my vpn plugins now, but thats a seperate issue. ;)
Comment 9 Dan Williams 2013-09-27 17:05:07 EDT
(In reply to Kevin Fenzi from comment #8)
> Bingo. Exactly the problem here. ;) 
> 
> ipv6 now works fine with the new NM. 

Excellent, nice to know, I think I'll keep this bug open though as it appears that people are hitting it pretty often.  It's odd that radvd doesn't appear to care about a supposedly invalid configuration.

> i am getting "Could not start connection: wrong plugin state 3" from my vpn
> plugins now, but thats a seperate issue. ;)

That's something we screwed up recently and subsequently fixed upstream, see https://bugzilla.gnome.org/show_bug.cgi?id=708255 .
Comment 10 Dan Williams 2013-09-27 17:13:34 EDT
Pavel, any thoughts on this?  Can we somehow fix this up in the platform so that we match whatever fixup the kernel is doing?

Could it be that in src/rdisc/nm-lndp-rdisc.c here:

		/* Device route */
		memset (&route, 0, sizeof (route));
--->		route.network = *ndp_msg_opt_prefix (msg, offset);
		route.plen = ndp_msg_opt_prefix_len (msg, offset);
		route.timestamp = now;

NetworkManager needs to mask off the lower <prefix> bits to ensure that mis-configured prefixes are fixed up to only represent the network bits?
Comment 11 Pavel Šimerda (pavlix) 2013-09-30 05:53:48 EDT
(In reply to Dan Williams from comment #10)
> Pavel, any thoughts on this?  Can we somehow fix this up in the platform so
> that we match whatever fixup the kernel is doing?
> 
> Could it be that in src/rdisc/nm-lndp-rdisc.c here:
> 
> 		/* Device route */
> 		memset (&route, 0, sizeof (route));
> --->		route.network = *ndp_msg_opt_prefix (msg, offset);
> 		route.plen = ndp_msg_opt_prefix_len (msg, offset);
> 		route.timestamp = now;
> 
> NetworkManager needs to mask off the lower <prefix> bits to ensure that
> mis-configured prefixes are fixed up to only represent the network bits?

I'm ok with masking off the route network right in the nm-lndp-rdisc code and possibly adding a simple check to nm-platform.
Comment 12 Dan Williams 2013-10-01 16:21:19 EDT
Pavel, any chance you could do a patch for that?
Comment 13 Pavel Šimerda (pavlix) 2013-10-01 16:34:34 EDT
(In reply to Dan Williams from comment #12)
> Pavel, any chance you could do a patch for that?

Started an upstream bug. I'll prepare a patch but I don't yet know when. Assiging to myself as it's a Fedora bug and I'll be working on the solution.
Comment 14 Thomas Haller 2013-10-02 06:14:01 EDT
For reference: the upstream bug is https://bugzilla.gnome.org/show_bug.cgi?id=709230
Comment 15 Pavel Šimerda (pavlix) 2013-10-02 06:18:23 EDT
(In reply to Thomas Haller from comment #14)
> For reference: the upstream bug is
> https://bugzilla.gnome.org/show_bug.cgi?id=709230

It's actually in the 'External Trackers' section at the top of this bugzilla page.
Comment 16 Dan Williams 2013-10-03 11:13:24 EDT
Upstream branch dcbw/rh1008104-bgo709230-rdisc-prefix-mask for review.
Comment 17 Fedora Update System 2013-10-03 15:51:39 EDT
NetworkManager-0.9.9.0-14.git20131003.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.9.0-14.git20131003.fc20
Comment 18 Fedora Update System 2013-10-04 21:43:13 EDT
Package NetworkManager-0.9.9.0-14.git20131003.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing NetworkManager-0.9.9.0-14.git20131003.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18316/NetworkManager-0.9.9.0-14.git20131003.fc20
then log in and leave karma (feedback).
Comment 19 Fedora Update System 2013-10-09 10:52:31 EDT
NetworkManager-0.9.9.0-14.git20131003.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.