Bug 1166199

Any update on this? It's been close to 2 months. I just performed a fedup to 21 and I hit the same issue. My em1 device remains disconnected and virbr0 is shown connected but of course I can't get internet through it. Good that at least wifi works though.

looks like re-occurrence of bug 1063545 and bug 1064441
I'm changing product and version to fedora21 as it looks like a network manager issue.

Created attachment 978433 [details]
log from systemctl restart NetworkManager

I've got rid of virbr0 device by putting into /etc/NetworkManager/NetworkManager.conf the following content:

> [keyfile]
> unmanaged-devices=interface-name:virbr0

It also should support unmanaged-devices=mac:...

But I still can't get em1 to connect. Attaching what I get from /var/log/messages on NetworkManager service restart.

Forgot to say that running dhcp manually brings up the interface just fine:
> $ sudo dhclient -d em1
> Internet Systems Consortium DHCP Client 4.3.1
> Copyright 2004-2014 Internet Systems Consortium.
> All rights reserved.
> For info, please visit https://www.isc.org/software/dhcp/
> 
> Listening on LPF/em1/3c:97:0e:19:30:f9
> Sending on   LPF/em1/3c:97:0e:19:30:f9
> Sending on   Socket/fallback
> DHCPREQUEST on em1 to 255.255.255.255 port 67 (xid=0x37c3b531)
> DHCPREQUEST on em1 to 255.255.255.255 port 67 (xid=0x37c3b531)
> DHCPACK from 192.168.1.1 (xid=0x37c3b531)
> bound to 192.168.1.145 -- renewal in 19334 seconds.

Very strange, after restarting the router, network manager started to successfully start the em1 interface. Before restarting the router, only manual dhclient request was working. Can that be related to leasing IP? I couldn't find how to configure that in NetworkManager :(

> After connection though I see the following in the log:
> Jan 10 00:56:36 localhost NetworkManager[7383]: <info>  Activation (em1) successful, device activated.
> Jan 10 00:56:36 localhost dnsmasq[8458]: started, version 2.72 cachesize 400
> Jan 10 00:56:36 localhost dnsmasq[8458]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect
> Jan 10 00:56:36 localhost dnsmasq[8458]: using nameserver 192.168.1.1#53
> Jan 10 00:56:36 localhost dnsmasq[8458]: cleared cache
> Jan 10 00:56:36 localhost dbus[1023]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
> Jan 10 00:56:36 localhost dbus[1023]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.nm-dispatcher.service': Unit dbus-org.freedesktop.nm-dispatcher.service failed to load: No such file or directory.
> Jan 10 00:56:36 localhost NetworkManager[7383]: <info>  NetworkManager state is now CONNECTED_GLOBAL

I'm referring to "dbus-org.freedesktop.nm-dispatcher.service failed to load" message.

hmm, last comment. I've noticed that with the above setup the virbr interface does not go up so I don't see my VMs. I removed the `unmanaged-devices` configuration, restarted the laptop and now everything works like a charm. WTH?!

I think experience could have been better out of the box. Not sure what caused all the trouble.

(In reply to Aleksandar Kostadinov from comment #3)
> Created attachment 978433 [details]
> log from systemctl restart NetworkManager
> 
> I've got rid of virbr0 device by putting into
> /etc/NetworkManager/NetworkManager.conf the following content:
> 
> > [keyfile]
> > unmanaged-devices=interface-name:virbr0
> 
> It also should support unmanaged-devices=mac:...
> 
> But I still can't get em1 to connect. Attaching what I get from
> /var/log/messages on NetworkManager service restart.

Note that the em1 issue you see in this log is https://bugzilla.gnome.org/show_bug.cgi?id=739482 and should be fixed in an F21 update very soon.

I cannot reproduce with latest F21 NetworkManager-0.9.10.1-1.git20150105.fc21.  When NM starts up, it leaves virbr0 alone and shouldn't be starting DHCP on it. virbr0 has its own 192.168.122.1 address which NM leaves in place.

If you experience this issue still, could you grab:

journalctl -b -u NetworkManager

and attach that shows the issue?

Created attachment 980356 [details]
sudo journalctl -b -u NetworkManager

Attaching journal from:
NetworkManager-0.9.10.1-1.2.20150109git.fc21.x86_64

perhaps I need to give some more details. My laptop was first installed with fedora18, then fedup to 20 and now fedup to 21. So the bridge was probably created in the fedora 18 times. I don't have /etc/sysconfig/network-scripts/ifcfg-virbr0, I'm not sure what and how is creating that bridge interface on boot.

Perhaps a simple recreate of the bridge would fix the issue? How can I do that?

libvirt's bridge devices are never listed (and shouldn't ever be listed!) in an ifcfg file. They are created by libvirtd when it is started, and NM shouldn't be touching them, attaching things to them, or even displaying them (and it shouldn't take an "unmanaged-devices" line in /etc/NetworkManager/NetworkManager.conf to make that happen).

If you notice that NM is doing something with your virbrX bridges, don't try to fix it by creating an ifcfg file for that device, that will only make matters worse, as NM will now have tacit permission to manage the device, and even to create a new device with that name the next time the system boots (which will result in libvirt being unable to start its virtual network that uses that bridge).

@Laine, so what do you suggest us to do? What I did at least fixed NM to touch the virt interface. If you know how to fix properly, please explain.

Well, the way to fix it properly is for NM to stop messing with devices that it has no business messing with :-). In the meantime, if adding the "unmanaged-devices" line to NetworkManager.conf is the only workaround to get you going, then definitely use it! (to be more precise - my intent wasn't to say that you should never do that, but rather that you shouldn't *have to* do that. Conversely, adding an ifcfg file for the libvirt network's bridge actually won't help matters at all (will actually make things even more broken), and so should never be done).

Have you tried removing that line with the most recent F21 updates? I have updates and updates-testing enabled on my F21 machine; while "nmcli list" does still show virbr0 and virbr0-nic as "connected" (implying that it is still attempting to "manage" them at some level), it at least hasn't messed with the IP address of virbr0, or disconnected virbr0-nic from virbr0 or tried to set it ~IFF_UP.

if NM destroyed your virbr0 connection, you can try to create virbr0 again by yourself typing:

$ virsh

virsh # net-destroy default
virsh # net-start default


After executing these commands, there should be newly configured virbr0.

I did:
* net-destroy default
* deleted virbr0 from network manager connection editor
* net-start default

result:
network manager applet still shows virbr0 interface

(In reply to Aleksandar Kostadinov from comment #14)
> I did:
> * net-destroy default
> * deleted virbr0 from network manager connection editor
> * net-start default
> 
> result:
> network manager applet still shows virbr0 interface

Yes, NM will show vibr0 interfaces anyway, but the commands, which I posted, should help you in case, when NM disconnects your virbr0 interface and then it keeps connecting it. 

When virbr0 was in "connecting" state, I was not able to reach my virtual machines until reboot. But then, I found net-destroy and net-start commands and after their execution, I don't need to reboot to fix network connection to my VMs.

I have the same problem.  I am running F 21 in a VM under VMware Fusion 7 Pro.

I'm not running xen, or any other virtual hypervisor in the VM.

But I've had the constantly orbiting green ball from vibr0 as long as I can remember.

I just added "unmanaged-devices=vibr0" to /etc/NetworkManager/NetworkManager.conf and restarted NetworkManager via systemctl.  That made it stop.  Thank you Laine!

However, NM shouldn't have been managing vibr0 in the first place.

(In reply to Aleksandar Kostadinov from comment #3)
> Created attachment 978433 [details]
> log from systemctl restart NetworkManager
> 
> I've got rid of virbr0 device by putting into
> /etc/NetworkManager/NetworkManager.conf the following content:
> 
> > [keyfile]
> > unmanaged-devices=interface-name:virbr0
> 
> It also should support unmanaged-devices=mac:...
> 
> But I still can't get em1 to connect. Attaching what I get from
> /var/log/messages on NetworkManager service restart.

This fixed my issue in a clean install of Fedora release 22 (Twenty Two) workstation. /etc/sysconfig/network-scripts/ifcfg-virbr0 does not exist.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Just upgraded to fedora 23 and I still see `virbr0` interface in Network Manager applet.

This time though `unmanaged-devices=interface-name:virbr0` fixes the issue (after restart).

I'm reopening the issue so that Network Manager automatically ignores such interfaces.

The same case here.

unmanaged-devices=interface-name:virbr0 did the trick.

This is especially problematic during the "offline" work when you don't have other connections active: NM is announcing to other applications that there are available network connections and then applications are all making unsuccessful attempts to get network resources polluting the notification system.

This crap came back recently and it's ignoring the unmanaged command in the conf file.

NetworkManager-wwan-1.0.12-2.fc23.x86_64
NetworkManager-vpnc-gnome-1.0.8-1.fc23.x86_64
NetworkManager-openvpn-1.0.8-2.fc23.x86_64
NetworkManager-vpnc-1.0.8-1.fc23.x86_64
NetworkManager-config-connectivity-fedora-1.0.12-2.fc23.x86_64
NetworkManager-1.0.12-2.fc23.x86_64
NetworkManager-pptp-gnome-1.1.0-2.20150428git695d4f2.fc23.x86_64
NetworkManager-openswan-gnome-1.0.8-3.fc23.x86_64
NetworkManager-wifi-1.0.12-2.fc23.x86_64
NetworkManager-pptp-1.1.0-2.20150428git695d4f2.fc23.x86_64
NetworkManager-iodine-gnome-0.0.5-2.fc23.x86_64
NetworkManager-openvpn-gnome-1.0.8-2.fc23.x86_64
NetworkManager-openswan-1.0.8-3.fc23.x86_64
NetworkManager-team-1.0.12-2.fc23.x86_64
NetworkManager-glib-1.0.12-2.fc23.x86_64
NetworkManager-l2tp-0.9.8.7-4.fc23.x86_64
NetworkManager-bluetooth-1.0.12-2.fc23.x86_64
NetworkManager-iodine-0.0.5-2.fc23.x86_64
NetworkManager-adsl-1.0.12-2.fc23.x86_64
NetworkManager-libnm-1.0.12-2.fc23.x86_64
NetworkManager-openconnect-1.0.8-1.fc23.x86_64

Linux tower.meval 4.5.5-201.fc23.x86_64 #1 SMP Sat May 21 15:29:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

This has remained reproducible in every release from Fedora 23 through Fedora 30 -- the problem never went away (and the underlying reason/behavior never changed). Please re-open against Fedora 30.

This occurs in Fedora Workstation when it is installed to disk. The behavior can be very easily seen by booting the Fedora Workstation live image, without any wired or wireless network connections. (However the missing 'libvirt-daemon-config-network' package needs to be installed from local storage after boot, except on Fedora 23 and Fedora 26 where this is already present.)


[liveuser@localhost ~]$ nmcli connection
NAME    UUID                                  TYPE    DEVICE 
virbr0  f20e2384-0dca-4a6b-b4cb-3dd1582c1ce1  bridge  virbr0 
[liveuser@localhost ~]$ nm-online
Connecting...............   30s [online]


This is (falsely) indicating that "the network is connected", as described in nm-online(1). Configuring NetworkManager to not manage virbr0 still works around this:

[liveuser@localhost ~]$ cat > ifcfg-virbr0 << EOF
> DEVICE=virbr0
> NM_CONTROLLED=no
> EOF
[liveuser@localhost ~]$ sudo cp ifcfg-virbr0 /etc/sysconfig/network-scripts/
[liveuser@localhost ~]$ rm ifcfg-virbr0 
[liveuser@localhost ~]$ sudo nmcli connection reload
[liveuser@localhost ~]$ nmcli connection

[liveuser@localhost ~]$ nm-online
Connecting...............    0s [offline]



But this should be an automatic setting, not a manual one. We already know that the explicit configuration of this interface is handled by libvirt (see /etc/libvirt/qemu/networks/default.xml), and NetworkManager should not be managing it and potentially changing its state.

However this more generally applies to any virtual network switch created with libvirt though, whether or not it is named virbr0.

Which package should contain the hooks to do this? Should NetworkManager be able to identify a libvirt network switch, or should libvirt configure NetworkManager (perhaps using a separate settings plugin) to not manage its devices?

(NB: I'm a libvirt developer, so I'm writing this comment from that POV)

1) You should never need to create (actually *shouldn't* create) an ifcfg-* file for a libvirt-created bridge. If proper operation requires this, then there is definitely a bug.

2) NetworkManager should never mess around with bridge devices created by other entities (e.g. libvirt, but really anyone else), and no special plugin should be required to make that happen. If we (libvirt) wanted NM to manage the bridge, we would create it via NM.

People normally don't see the issue you describe, because almost everybody has a permanently online internet connection these days, but I can see why it would be problematic.

Since this behavior has been present in NM for such a long time, I would say that the proper place for the bug to be filed would be upstream rather than in Fedora. I don't know if the NM developers pay more attention to their mailing list, or to their issue tracker on gitlab.freedesktop.org.

When an interface is configured outside of NetworkManager's knowledge (like libvirt's or docker's bridge), then NetworkManager generates an in-memory connection profile and pretends(!) that this is active. This is to show that something is going on with the device, and NetworkManager is supposed to touch that device.

Whether this pretend mode is of any use is a good question. It's obviously confusing. I actually do make use of this behaviour to have a dispatcher script that does something for such external devices, so it's not entirely useless. Also, it causes the output of `nmcli device` to show that *something* is going on there. Although, maybe that's more confusing than helpful. In any case, NetworkManager needs a way to express that something is happening with this devices, and this is the way it does that. It doesn't mean that NetworkManager actually touches the device.


Comment 26 does not say what actual problem NetworkManager causes by this (aside the confusion to the user).

Comment 26 also explains that `nm-online` gives wrong results. I think nm-online has one real use-case: to implement `NetworkManager-wait-online.service` (with the `--startup` option).
Without the `--startup` option, I don't think it's of any use at all. It can only return "online" or "offline". It maps the NetworkManager's states "local", "site", and "global" to "online". I think that is is sensible, if you connect to a libvirt bridge some network is configured and nm-online considers that as "online". However I don't think that describing the online state in two words is meaningful and calling `nm-online` is not useful.


For a while NetworkManager had a udev rule to automatically unmanage docker bridges. But that was dropped, because docker bridges are nothing special. It's a common thing that something aside NetworkManager configures an interface and NetworkManager must not interfere. So, what actual problems are causes by this?


---


> 1) You should never need to create (actually *shouldn't* create) an ifcfg-* file for a libvirt-created bridge. If proper operation requires this, then there is definitely a bug.

Right. NM doesn't.

> 2) NetworkManager should never mess around with bridge devices created by other entities (e.g. libvirt, but really anyone else), and no special plugin should be required to make that happen. If we (libvirt) wanted NM to manage the bridge, we would create it via NM.

Right. NM shouldn't. Does it?

In previous versions of GNOME (such as the version in RHEL 7.6), an active NetworkManager connection profile for virbr0 caused a wired Ethernet connection to appear in the top bar in GNOME Shell. It seems this is actually now suppressed, and GNOME Control Center does not expose the virbr0 device on the network panel (or allow it to be configured there).


For users who are looking to check if a NetworkManager connection profile is active excluding libvirt's, disabling NetworkManager's control of the bridge devices still seems to me like an appropriate method to adjust the behavior of nm-online. I'm not sure I agree with the comment about *never* using ifcfg: the configuration I provided above is equivalent to setting "unmanaged-devices=interface-name:virbr0" in /etc/NetworkManager/NetworkManager.conf. It is read by NetworkManager's ifcfg-rh settings plugin, versus NetworkManager's keyfile settings plugin.

It's also worth mentioning that libvirt allows hook scripts to be run when a virtual network switch is started/stopped, which can substitute for NetworkManager dispatcher scripts for these devices: https://libvirt.org/hooks.html