fwiw...

# rpm -qa NetworkManager\*
NetworkManager-vpnc-0.9.8.2-2.fc20.x86_64
NetworkManager-glib-devel-0.9.9.0-28.git20131003.fc20.x86_64
NetworkManager-devel-0.9.9.0-28.git20131003.fc20.x86_64
NetworkManager-0.9.9.0-28.git20131003.fc20.x86_64
NetworkManager-vpnc-gnome-0.9.8.2-2.fc20.x86_64
NetworkManager-glib-0.9.9.0-28.git20131003.fc20.x86_64
NetworkManager-glib-0.9.9.0-28.git20131003.fc20.i686
NetworkManager-openconnect-0.9.8.0-2.fc20.x86_64
NetworkManager-pptp-gnome-0.9.8.2-3.fc20.x86_64
NetworkManager-openvpn-gnome-0.9.9.0-0.1.git20140128.fc20.x86_64
NetworkManager-openvpn-0.9.9.0-0.1.git20140128.fc20.x86_64
NetworkManager-pptp-0.9.8.2-3.fc20.x86_64

Logs say this:

Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (br0): carrier is OFF
Feb 10 06:57:31 tlielax NetworkManager[880]: <error> [1392033451.208423] [platform/nm-linux-platform.c:1338] sysctl_get(): error reading /sys/class/net/br0/phys_port_id: Failed to read from file '/sys/class/net/br0/phys_port_id': Operation not supported
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (br0): new Bridge device (driver: 'bridge' ifindex: 3)
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (br0): exported as /org/freedesktop/NetworkManager/Devices/1
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (br0): No existing connection detected.
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (br0): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (br0): bringing up device.
Feb 10 06:57:31 tlielax NetworkManager[880]: <warn> (br0): device not up after timeout!
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (em1): carrier is OFF
Feb 10 06:57:31 tlielax NetworkManager[880]: <error> [1392033451.249225] [platform/nm-linux-platform.c:1338] sysctl_get(): error reading /sys/class/net/em1/phys_port_id: Failed to read from file '/sys/class/net/em1/phys_port_id': Operation not supported
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (em1): new Ethernet device (driver: 'e1000e' ifindex: 2)
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (em1): exported as /org/freedesktop/NetworkManager/Devices/2
Feb 10 06:57:31 tlielax dbus[926]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (em1): No existing connection detected.
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (em1): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
Feb 10 06:57:31 tlielax NetworkManager[880]: <info> (em1): bringing up device.

...and then a little later:

Feb 10 06:57:35 tlielax kernel: [   35.230530] e1000e: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Feb 10 06:57:35 tlielax kernel: [   35.230726] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready
Feb 10 06:57:35 tlielax NetworkManager[880]: <info> (em1): link connected
Feb 10 06:57:35 tlielax NetworkManager[880]: <info> (em1): device state change: unavailable -> disconnected (reason 'carrier-changed') [20 30 40]
Feb 10 06:57:35 tlielax NetworkManager[880]: <info> Auto-activating connection 'br0sl1'.
Feb 10 06:57:35 tlielax NetworkManager[880]: <info> Connection 'br0sl1' auto-activation failed: (1) No compatible disconnected device found for master connection 92ab87e2-4af7-41af-a963-a5fcb8954d34.
Feb 10 06:57:35 tlielax NetworkManager[880]: <info> startup complete

I'm not sure, but problem seems to be that em1 is picking up an IPv6 address when it should stay unconfigured and slaved to the bridge?

I'm really not sure what changed and caused this to start happening though.

I'm seeing a similar problem with the updates-testing version of NetworkManager (1:0.9.9.0-29.git20140131.fc20).

The behavior I see is that NetworkManager stays in the "twirling" animation. If I mouse over it, it says:

    Requesting address for 'virbr0'...

If I tell NetworkManager to disconnect from virbr0, it takes the virbr0 interface down entirely, which makes it non-functional for virtual guests:

2014-02-10T13:56:58.985749-05:00 localhost dnsmasq-dhcp[2031]: DHCP packet received on virbr0 which has no address

If I then try to bring the interface back up via NetworkManger, I receive this error dialog:

    Connection activation failed

    (1) Creating object for path '/org/freedesktop/NetworkManager/ActiveConnection/2' failed in libnm-glib.

So it looks like the updates-testing version of NetworkManager has issues with bridge devices, including virbr0.

If I revert to 1:0.9.9.0-28.git20131003.fc20, NetworkManager doesn't acknowledge that virbr0 exists: it doesn't appear in the "Network Connections" list.

I do note that there is no /etc/sysconfig/network-scripts/ifcfg-virbr0 script on my system. I wonder if creating it and adding "NM_CONTROLLED=no" to it will prevent git20140131 from [mis]managing virbr0.

(Even if it does, though, I don't think that's a realistic option: even if libvirtd is wrong for not telling NM to not manage virbr0, breaking everyone who is running libvirtd isn't acceptable.)

That problem sounds different, but it may be due to similar causes. My bridge is set up to come up whether libvirtd is running or not (though I have it running to provide a bridge for guests to attach to). The problem you're having sounds more like some sort of strange interaction between libvirtd and NM. It may be best to open your problem as a separate bug since I'm not convinced they're the same issue. The bug owner can always collapse them together if they do turn out to be related.

FYI, you might try downgrading libnl3 to 3.2.21-2 -- my bridge started working again with the older libnl, so there's an incompatibility there someplace.

James - I saw some weirdness with virbr0 and NM until I restarted NM.  I'm not sure it is a good idea to display virbr0, but at least now the status shows as up/connected.

I can reproduce this issue.

having the following two connections:


br0:[connection]
br0:id=br0
br0:uuid=7ecb25ca-f89e-44a9-869b-0c66ddba0cb8
br0:interface-name=br0
br0:type=bridge
br0:autoconnect=false
br0:
br0:[ipv6]
br0:method=auto
br0:
br0:[ipv4]
br0:method=auto
br0:
br0:[bridge]
br0:interface-name=br0
br0-em1:[connection]
br0-em1:id=br0-em1
br0-em1:uuid=bf8ec57d-c917-499b-b33f-5bb46581efc7
br0-em1:interface-name=em1
br0-em1:type=ethernet
br0-em1:autoconnect=false
br0-em1:master=br0
br0-em1:slave-type=bridge



The error happens when I `nmcli connection up br0-em1` and the bridge br0 does not exist yet.





Interestingly, the error does not happen with

  - NetworkManager-current-master
  - libnl3-3.2.22-2.fc20.x86_64
  - kernel-3.12.10-300.fc20.x86_64

In that case, I can up the bridge and down it successfully. Only afterwards, the em1 device has NO-CARRIER and state DOWN for about 10 minutes and the interface is effectively unusable, neither warm-reboot or rmmod helps. Only waiting or cold-reboot makes em1 usable again. I think, this is an unrelated bug kernel bug that I might file later.



The error with
<warn> (br0): device not up after timeout!
does however happen with

  - NetworkManager-current-master
  - libnl3-3.2.24-1.fc20.x86_64
  - kernel-3.12.10-300.fc20.x86_64

so, it seems the difference is libnl3, which just last week was pushed to F20/stable. Reassign bug to libnl3...

(In reply to Scott Shambarger from comment #6)
> FYI, you might try downgrading libnl3 to 3.2.21-2 -- my bridge started
> working again with the older libnl, so there's an incompatibility there
> someplace.

Yep, thanks! Downgrading to libnl3-3.2.21-2.fc20 fixed the issue for me.

Working bridge (libnl3-3.2.21-2):

<info> (br0): bringing up device.
<debug> [1392067683.902405] [platform/nm-platform.c:767] nm_platform_link_set_up(): link: setting up 'br0' (5)
...
<debug> [1392067683.905160] [platform/nm-linux-platform.c:993] check_cache_items(): cache 0x7f1f28839600 object 0x7f1f2883a690
<debug> [1392067683.905190] [platform/nm-platform.c:2091] log_link(): signal: link changed: br0 (5)

Non-working bridge (libnl3-3.2.24-1):

<info> (br0): bringing up device.
<debug> [1392065455.670394] [platform/nm-platform.c:767] nm_platform_link_set_up(): link: setting up 'br0' (4)
<debug> [1392065455.670948] [platform/nm-platform.c:2091] log_link(): signal: link added: br0 (4)
<warn> (br0): device not up after timeout!

Note the difference.... "link changed" works, "link added" doesn't - could be the new netlink has changes how things are reported?

Downgrading to libnl3-3.2.21-2.fc20 doesn't change the behavior of NetworkManager-0.9.9.0-29.git20140131.fc20 in beating on the virbr0 interface, so I agree that this is probably a separate issue. I'll open a separate bug report.

@Orion (comment 7): I tried both restarting NetworkManager, and performing a full system reboot. NetworkManager's behavior was still broken.

One observation is that the waiting in http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/devices/nm-device.c#n5330 is broken since commit http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=5074898591ae99f79b0cc4872c3c7cf018abee82. The reason is, that nm_device_is_up() looks into the cache (without reloading it from the kernel), and since the main thread gets blocked, the state will not change.

This should be fixed, by calling something like nl_cache_refill() while waiting.

>> reassigning to NetworkManager.

Another question is, why it worked before, but breaks with newer libnl. Maybe we can also add an workaround to libnl...

(In reply to Thomas Haller from comment #13)
> One observation is that the waiting in
> http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/devices/
> nm-device.c#n5330 is broken since commit
> http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/
> ?id=5074898591ae99f79b0cc4872c3c7cf018abee82. The reason is, that
> nm_device_is_up() looks into the cache (without reloading it from the
> kernel), and since the main thread gets blocked, the state will not change.
> 
> This should be fixed, by calling something like nl_cache_refill() while
> waiting.

Please don't ever do that as it breaks the whole caching workflow. We've had problems with it before. If you do need to periodically check whether the device is IFF_UP/IFF_LOWER_UP, it's sufficient to call nm_platform_link_set_up.

In most cases the active waiting should be avoided. And it would be good to add comments with specific use cases and reasons why waiting is needed.

In the logfile (setting NLDBG environment variable), I can see with libnl3-3.2.24-1 the lines

DBG<4>            object.c:207  nl_object_get: New reference to object 0x19c34b0, total 2
DBG<5>        route/link.c:895  link_keygen: link 0x19c34b0 key (dev 9 fam 7) keysz 8, hash 0x2b2
DBG<2>         hashtable.c:127  nl_hash_table_add: Warning: Add to hashtable found duplicate...
DBG<4>            object.c:221  nl_object_put: Returned object reference 0x19c34b0, 1 remaining
NetworkManager[17745]: <error> [1392114373.475432] [platform/nm-linux-platform.c:1328] event_notification(): netlink cache error: Object exists

whereas they are not present with the (working) libnl3-3.2.22-2.

So, this seems to be https://bugzilla.gnome.org/show_bug.cgi?id=719905 and this bug was already present before (it only happens more easily with newer libnl3).

I think, the relevant change in libnl is commit https://github.com/thom311/libnl/commit/fdd1ba220dd7b780400e9d0652cde80e59f63572


I pushed branch th/bgo719905_platform_caching for review, which fixes the issue AFAIS.

(In reply to Pavel Šimerda (pavlix) from comment #14)
> (In reply to Thomas Haller from comment #13)
> > One observation is that the waiting in
> > http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/devices/
> > nm-device.c#n5330 is broken since commit
> > http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/
> > ?id=5074898591ae99f79b0cc4872c3c7cf018abee82. The reason is, that
> > nm_device_is_up() looks into the cache (without reloading it from the
> > kernel), and since the main thread gets blocked, the state will not change.
> > 
> > This should be fixed, by calling something like nl_cache_refill() while
> > waiting.
> 
> Please don't ever do that as it breaks the whole caching workflow. We've had
> problems with it before. If you do need to periodically check whether the
> device is IFF_UP/IFF_LOWER_UP, it's sufficient to call
> nm_platform_link_set_up.
> 
> In most cases the active waiting should be avoided. And it would be good to
> add comments with specific use cases and reasons why waiting is needed.

Btw. whenever a netlink event hits, the object gets refetched synchronously and there are other places, where this happens too (e.g via refresh_object). So, obviously it is possible to refetch data without breaking the whole caching workflow.

*** Bug 1063800 has been marked as a duplicate of this bug. ***

This is my winning combination:
kernel-3.14.0-0.rc2.git0.10.fc21.x86_64
NetworkManager-0.9.9.0-27.git20140131.fc21.x86_64
libnl3-3.2.22-2.fc21.x86_64

Thanks.

(In reply to Thomas Haller from comment #16)
> Btw. whenever a netlink event hits, the object gets refetched synchronously
> and there are other places, where this happens too (e.g via refresh_object).
> So, obviously it is possible to refetch data without breaking the whole
> caching workflow.

It obviously is. It just cannot be done using nl_cache_refill and is not very useful. The only busy waiting in NM is on link state and can be either removed or nm_platform_link_set_up can be used as a temporary workaround to trigger the refetch.

We had a discussion on IRC, so just summarizing...

My intention was always to get rid of the busy waiting and never expose any cache operations from nm-platform except via 'push' actions (i.e. NM-requested changes of kernel configuration). Apart from the push actions, nm-platform collects events from the kernel to only notify NM core once as well as to distinguish changes coming from NM and from outside. The caching code is rather fragile, not in the way it's written, but in the functionality it's achieving. A relatively small change can trigger unexpected behavior.

Resurrection of now defunct nm-platform tests would also help to keep nm-platform working properly. I would guess it warrants one or more upstream bugzilla tickets.

Downgrading from 3.2.24-1.fc20 to 3.2.21-2.fc20 and restarting NetworkManager allowed my bridging to work again. I had symptoms very close to Scott Shambarger's report.

Created attachment 863284 [details]
journalctl -u NetworkManager -u libvirtd

The same here with:
NetworkManager-glib-0.9.9.0-28.git20131003.fc20.x86_64
NetworkManager-pptp-gnome-0.9.8.2-3.fc20.x86_64
NetworkManager-0.9.9.0-28.git20131003.fc20.x86_64
NetworkManager-openconnect-0.9.8.0-2.fc20.x86_64
NetworkManager-pptp-0.9.8.2-3.fc20.x86_64
NetworkManager-vpnc-0.9.8.2-2.fc20.x86_64
NetworkManager-openvpn-gnome-0.9.9.0-0.1.git20140128.fc20.x86_64
NetworkManager-glib-0.9.9.0-28.git20131003.fc20.i686
NetworkManager-openvpn-0.9.9.0-0.1.git20140128.fc20.x86_64
NetworkManager-vpnc-gnome-0.9.8.2-2.fc20.x86_64

When I reboot, I see this:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          8000.000000000000       no
virbr0          8000.525400ed5541       yes             virbr0-nic

# systemctl restart NetworkManager
# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          0080.f46d0461abb4       yes             p6p1
virbr0          8000.525400ed5541       yes             virbr0-nic

NetworkManager-0.9.9.0-30.git20131003.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.9.0-30.git20131003.fc20

Package NetworkManager-0.9.9.0-30.git20131003.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing NetworkManager-0.9.9.0-30.git20131003.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-2651/NetworkManager-0.9.9.0-30.git20131003.fc20
then log in and leave karma (feedback).

Will the Rawhide get an update likewise?

Version 0.9.9.0-30.git20131003.fc20 apparently does not fix this problem.

My next try will be to downgrade libnl3. Will keep you posted.

Downgrading to libnl3-cli-3.2.21-2.fc20.x86_64 along with NetworkManager-0.9.9.0-28.git20131003.fc20.x86_64 still doesn't work.

I'm disabling bridging for a while. Will have to live without it until it is fixed.

(In reply to Avi Alkalay from comment #26)
> Downgrading to libnl3-cli-3.2.21-2.fc20.x86_64 along with
> NetworkManager-0.9.9.0-28.git20131003.fc20.x86_64 still doesn't work.
> 
> I'm disabling bridging for a while. Will have to live without it until it is
> fixed.

Version 0.9.9.0-30.git20131003.fc20 fixes the problem for me. You probably have another issue, but without more information that is hard to tell

Also tried NM-0.9.9.0-30 the libnl3-3.2.24-1 and bridging came up normally.

Of course, the NM version is quite old, and is missing many bugs fixed in -29 (and the later git base), so it's a mixed blessing...

Confirmed for me. NetworkManager-0.9.9.0-30.git20131003.fc20.x86_64 along with the later libnl3 seem to work fine.

I tried the updates-testing RPM with no success. Maybe the problem is the way I'm configuring bridging, which uses the NetworkManager UI as documented here:

http://avi.alkalay.net/2014/01/fedora-20-virtualization-networkmanager-native-bridge.html

I just received a comment from that blog post from somebody saying it is not working for him too.

Can anybody have a look on the method on the URL and tell if it is right?

Note that the how-to goes further and configures libvirt for bridging, but my problem is before libvirt. The host's bridge1 interface doesn't get an IP from DHCP and this is where I get stuck.

Thank you for your help

Avi, given that the originally reported bug is fixed, I suggest that you open a new bug for the problem you're seeing.

(In reply to poma from comment #24)
> Will the Rawhide get an update likewise?

Update for Rawhide is ready too...

NetworkManager-0.9.9.0-30.git20131003.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

(In reply to Thomas Haller from comment #32)
> (In reply to poma from comment #24)
> > Will the Rawhide get an update likewise?
> 
> Update for Rawhide is ready too...

Works equally well in Rawhide & Humpty Dumpty.
A bridged virtio_nets is the only n' lonely what doesn't work, via static leases.
But that's another story. :)

Thanks!