Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 619863 - [ifcfg-rh] ignore configs with BRIDGE= and VLAN= until we support them
[ifcfg-rh] ignore configs with BRIDGE= and VLAN= until we support them
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: NetworkManager (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Dan Williams
desktop-bugs@redhat.com
: RHELNAK
: 621254 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-30 14:12 EDT by Laine Stump
Modified: 2012-06-26 23:03 EDT (History)
8 users (show)

See Also:
Fixed In Version: NetworkManager-0.8.1-5.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-10 14:32:11 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
the output of nmcli con list (762 bytes, text/plain)
2010-07-30 15:56 EDT, Laine Stump
no flags Details
nmcli con list uuid f702aea3-35ed-4db9-937f-5c33f8abd8d6 (1.31 KB, text/plain)
2010-07-30 16:03 EDT, Laine Stump
no flags Details
log of "ncftool ifup br0" (5.58 KB, application/octet-stream)
2010-07-30 17:07 EDT, Laine Stump
no flags Details
-x output of /sbin/ifup when trying to ifup eth0 after removing br0 (3.01 KB, text/plain)
2010-08-03 22:05 EDT, Laine Stump
no flags Details

  None (edit)
Description Laine Stump 2010-07-30 14:12:15 EDT
(This may be more appropriate assigned to initscripts; if so, feel free. However, if initscripts hardcodes an avoidance of NetworkManager when an interface is a vlan/bridge, then there will be problems when the day arrives the NM *does* handle those types of interfaces...)

If NetworkManager is enabled, /sbin/ifup (and, presumably, /sbin/ifdown) will attempt to use NetworkManager to bring up vlans and bridges, even though NetworkManager doesn't support those types of interfaces.

Either ifup or NetworkManager should be made intelligent enough to allow normal ifup/ifdown processing on interface types that NetworkManager doesn't support.

To see the effect on vlans, create a vlan ifcfg file, eg: ifcfg-eth0.43

DEVICE=eth0.43
VLAN=yes
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.43.149
NETMASK=255.255.255.0

then enable NetworkManager (if it isn't already enabled) and run

  # ifup eth0.43


Notice that eth0 gets the IP address that should be given to eth0.43, and eth0.43 isn't created at all.

If NetworkManager is disabled, ifup will bring up the vlan interface itself, and will do it correctly.

(for vlans, ifdown does the right thing whether NM is enabled or not)

With a bridge interface, create these two files:

root@rhel6 /etc/sysconfig/network-scripts>cat ifcfg-eth0
DEVICE=eth0
HWADDR=00:22:15:59:62:97
ONBOOT=no
BRIDGE=br0
root@rhel6 /etc/sysconfig/network-scripts>cat ifcfg-br0
DEVICE=br0
ONBOOT=no
TYPE=Bridge
BOOTPROTO=dhcp
STP=on
DELAY=0

(this is roughly equivalent to what happens when you define a bridge attached to eth0 using netcf)

then run "ifup br0". There will be a "very long" delay while it is attempting to acquire an IP address, then it will finally fail. If you do "brctl br0" during this time, you'll see that the bridge device was created, but eth0 was not attached.

We're having a lot of problems with people forgetting to disable NetworkManager (or not knowing that NetworkManager and netcf/libvirt don't work well together); without going the full path of supporting bridge configuration in NM, solving intermediate problems like this would go a long way towards a friendlier experience (and less bug reports).
Comment 1 Laine Stump 2010-07-30 14:30:57 EDT
I just noticed that NetworkManager is capable of quickly returning an error to
ifup if it believes it isn't managing an interface:

root@rhel6 /etc/sysconfig/network-scripts>ifup eth0
Error: Connection activation failed: Device not managed by NetworkManager
Interface eth0 bring-up failed!
error: failed to execute external program
error: Running 'ifup eth0' failed with exit code 4

(it's a long story how I got to that state, unimportant at the moment).

It seems it should be possible for ifup to attempt the call to NetworkManager,
which would return "not managed by me" if it was an interface type not
supported; ifup would then do something more intelligent with this failure, ie
do whatever it normally does when NM isn't running.

This way the config files wouldn't need any of the broken workaround
"NM_CONTROLLED=no" settings, and ifup wouldn't have to make any assumptions
about what types of interfaces NetworkManager supports (which will both cause
problems when upgrading to a future version of NM that *does* support these interface types)
Comment 3 RHEL Product and Program Management 2010-07-30 14:47:42 EDT
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 4 Bill Nottingham 2010-07-30 15:34:19 EDT
/etc/sysconfig/network-scripts/ifcfg-eth1.43 =

DEVICE=eth1.43
VLAN=yes
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.43.149
NETMASK=255.255.255.0

dbus-send --system --print-reply --dest=com.redhat.ifcfgrh1 /com/redhat/ifcfgrh1  com.redhat.ifcfgrh1.GetIfcfgDetails string:/etc/sysconfig/network-scripts/ifcfg-eth1.43
yields:
UUID=f702aea3-35ed-4db9-937f-5c33f8abd8d6

That's the problem. Back to NM for now ... should it be returning a UUID for a connection it can't handle right?
Comment 5 Bill Nottingham 2010-07-30 15:50:44 EDT
With respect to your bridging config, if you have ONBOOT=no in ifcfg-eth0, you need to ifup that first before you bring up br0, otherwise it won't be attached.

(We attach bonding slaves automatically to bonding devices, but not bridge ports. I suppose this is inconsistent.)
Comment 6 Laine Stump 2010-07-30 15:56:41 EDT
Created attachment 435634 [details]
the output of nmcli con list

As we discussed in IRC, the vlan interface shows up in NM's list of editable connections (and available connections.

Here is the output of nmcli con list when it's in that state.
Comment 7 Laine Stump 2010-07-30 16:03:39 EDT
Created attachment 435635 [details]
nmcli con list uuid f702aea3-35ed-4db9-937f-5c33f8abd8d6
Comment 8 Laine Stump 2010-07-30 16:35:42 EDT
About Comment 5:

I see now that we've previously worked okay with NM disabled because the standard practice is to put an existing interface onto a bridge without bringing it down. If I start from a bridge ifconfiged down, though, it fails even with NM disabled.

However, if NM is enabled, when I ifconfig up the device, NM tries to use old config, obtaining a DHCP address for the interface, for example. In certain circumstances (which I'm still pinning down), NM actually *overwrites* the minimal ifcfg-eth0 file netcf creates with one of its own, which contains all kinds of extra stuff (including uuid, IP config, etc). (I was actually planning on filing a separate bug for that).
Comment 9 Laine Stump 2010-07-30 17:07:08 EDT
Created attachment 435659 [details]
log of "ncftool ifup br0"

More information about the bridge config.

It turns out that if you do the ifup via ncftool (or virsh iface-start, same thing), it will call ifup for the attached interfaces first. So, "ncftool ifup br0" will do the equivalent of:

   ifup eth0
   ifup br0

The problem is that, even though ifcfg-eth0 contains:

DEVICE=eth0
HWADDR=00:22:15:59:62:97
ONBOOT=yes
BRIDGE=br0

ifup eth0 will acquire an IP address. After that happens, "ifup br0" is called, and it doesn't attach eth0 to the bridge, so it fails to get an IP address.

This same practice works properly when NM is disabled.
Comment 10 Dan Williams 2010-08-02 13:04:11 EDT
I'll make NM ignore configs that have BRIDGE= or VLAN= in them.

QE reproducer instructions:
---------------------------------
1) Create an ifcfg file named "ifcfg-bridge" that contains:
DEVICE=eth0
HWADDR=00:22:15:59:62:97
ONBOOT=yes
BRIDGE=br0


2) run this command from a terminal:

dbus-send --system --print-reply --dest=com.redhat.ifcfgrh1
/com/redhat/ifcfgrh1  com.redhat.ifcfgrh1.GetIfcfgDetails
string:/etc/sysconfig/network-scripts/ifcfg-bridge

and if the problem is fixed, we expect the dbus-send command to return and error.  If the problem is not fixed, it'll return something like:

UUID f702aea3-35ed-4db9-937f-5c33f8abd8d6

3) Create another ifcfg file named "ifcfg-vlan" that contains:
DEVICE=eth0
HWADDR=00:22:15:59:62:97
ONBOOT=yes
VLAN=yes

4) run this command from a terminal:

dbus-send --system --print-reply --dest=com.redhat.ifcfgrh1
/com/redhat/ifcfgrh1  com.redhat.ifcfgrh1.GetIfcfgDetails
string:/etc/sysconfig/network-scripts/ifcfg-vlan

and we expect same results for this command as in step #2


Basically, if the ifcfg file has BRIDGE= or VLAN= in it, NM should ignore that config.
Comment 11 Laine Stump 2010-08-02 16:14:50 EDT
I think you'll also need to identify those with "TYPE=Bridge" as well, won't you?

"VLAN=" catches the vlan interfaces, and "BRIDGE=" gets the physical interfaces that are connected to the bridge, but the bridge itself doesn't have "BRIDGE=" in its config.
Comment 12 Denise Dumas 2010-08-03 14:57:26 EDT
OK, Dan, can you please get fix to Laine for testing and add this to Snapshot
11?
Comment 13 Dan Williams 2010-08-03 16:58:06 EDT
We actually already ignore TYPE=Bridge connections.  Just not BRIDGE= and VLAN=
Comment 14 Dan Williams 2010-08-03 17:17:35 EDT
(In reply to comment #12)
> OK, Dan, can you please get fix to Laine for testing and add this to Snapshot
> 11?    

Can I get the PM acks at the next meeting so I can build packages?
Comment 16 Dan Williams 2010-08-03 18:53:45 EDT
For various date & timezone related reasons I can't do a scratch build right now (yay for my timezone being wrong when generating the tarball), so here's the SRPM if you don't mind doing a scratch build yourselves:

http://people.redhat.com/dcbw/NetworkManager-0.8.1-4.1.el6.src.rpm

otherwise I may be able to do one later tonight.
Comment 17 Laine Stump 2010-08-03 22:00:36 EDT
Yay!!

I built from the srpm, restarted NM, then created both a vlan and two bridges using netcf's ncftool, then ifup'ed them. Both came up with no problems, and ifconfig and brctl showed everything was in order (eth0 wasn't mistakenly given the IP of the bridge or vlan, the bridge and vlan both had their IPs, and the physical interfaces were connected to the bridges).

There is one problem I notice (I'll attach a log in a moment): If I remove the bridge and return ifcfg-eth0 back to its original state, ifup no longer works on it because NM still believes that it's a part of a bridge; is it caching the ifcfg contents?

I don't know if that part is easily fixable, but I don't think the fix you've made here should wait - this will eliminate probably 80% of the bug reports from people trying to use netcf and NetworkManager together - *removing* a bridge is a much less common operation.
Comment 18 Laine Stump 2010-08-03 22:05:53 EDT
Created attachment 436416 [details]
-x output of /sbin/ifup when trying to ifup eth0 after removing br0

This log shows what happens if I have an eth0 that is connected to br0, and subsequently bring down the bridge, and modify the eth0 config to be standalone (I've cat'ed ifcfg-eth0 inline to show what it contains).

After changing ifcfg-eth0 to remove the BRIDGE=br0 line (and add back in dhcp config) NM still believes that the interface is part of a bridge, and reports back that it isn't managed by NM. ifup also doesn't want to handle it, I guess because it believes that NM should?

If I restart NM, everything goes back to normal.
Comment 19 Dan Williams 2010-08-04 01:05:51 EDT
Can you attach either or both of:

1) all your ifcfg files *after* you've removed BRIDGE=br0 when you've made the change mentioned in comment 18

2) syslog from before up until after you removed BRIDGE=br0 as described in comment 18

It may be that you have some other ifcfg file that has HWADDR and NM_CONTROLLED=no; if *any* ifcfg has HWADDR=<ethX MAC address> and NM_CONTROLLED=no then NM will ignore that interface and any connections/ifcfgs that have the same HWADDR in them, because you've told NM to ignore it.
Comment 20 Dan Williams 2010-08-04 01:06:19 EDT
Brew scratch build for the package I refer to in comment 16:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=2650675
Comment 21 Rik van Riel 2010-08-04 12:03:49 EDT
*** Bug 621254 has been marked as a duplicate of this bug. ***
Comment 22 Dan Williams 2010-08-09 15:42:35 EDT
Further fixes for this issue have been made upstream and should hit the next snapshot.

QE: *additional* to the testcases from Comment 10, we now expect NetworkManager to unmanage interfaces listed in ifcfg files that have BRIDGE= + HWADDR= or VLAN= + HWADDR=.  So with this ifcfg file from comment 10:

DEVICE=eth0
HWADDR=00:22:15:59:62:97
ONBOOT=yes
BRIDGE=br0

we would expect NM to report something liek the following in /var/log/messages:

Aug  9 13:08:45 dcbw NetworkManager[29767]:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth0 ... 
Aug  9 13:08:45 dcbw NetworkManager[29767]:    ifcfg-rh:     read connection 'System eth0'
Aug  9 13:08:45 dcbw NetworkManager[29767]:    ifcfg-rh: Ignoring connection 'System eth0' and its device due to NM_CONTROLLED/BRIDGE/VLAN.
Aug  9 13:08:45 dcbw NetworkManager[29767]: <info> (eth0): now unmanaged
Aug  9 13:08:45 dcbw NetworkManager[29767]: <info> (eth0): device state change: 2 -> 1 (reason 3)
Comment 23 Laine Stump 2010-08-10 14:17:32 EDT
I've been testing with NetworkManager-0.8.1-4.4, provided by Dan, and have found that this problem appears to be solved, ie bridges (and their associated physical interfaces, and vlan interfaces are ignored by NetworkManager, so /sbin/ifup is allowed to do its work on those interfaces as if NetworkManager didn't exist. I think the changes Dan has made in this version should be posted to RHEL, and we can close this bug (pending QE testing, of course).

There are a few issues that I've noticed, but those will be filed separately:

1) if an ethernet device is added to or removed from a bridge (by modifying the ifcfg file), and then ifup is called immediately to bring up the device, the inotify message that tells NM the file has been modified seems to always reach NM *after* the ifup request comes in on dbus. The result is that NM believes the device is still managed, but then fails to bring it up.

If a delay of 1 second is put between modifying the file (with ncftool define, for example) and calling ifup, the inotify message is processed by NM before it receives the ifup, so it has up-to-date information on whether or not it should manage the interface.

This creates a problem for virt-manager when the "activate now" checkbox is selected during bridge creation; since there is no delay between defining the bridge and ifup'ing the device, activation fails. The workaround is to uncheck "activate now", create the bridge, and then click on the "start" button separately.

2) In at least one case, after a couple of quick iterations of removing an interface from a bridge, then re-adding it to the bridge, there can be a very long pause while bringing up the interface. I've found that this is caused by the DHCP server attempting to respond to the discover request with a *unicast* packet. I believe this is an issue with the DHCP server (running on a small wireless router), probably not with our dhclient. This same problem occurs if NM is disabled, so it definitely is not an NM problem.

3) When an interface becomes "unmanaged" due to it being placed on a bridge, any nameserver and route data that may be associated with the device are removed from /etc/resolv.conf and the route table respectively. Solving this problem is way beyond the scope of this bug - the solution will probably be to make NM able to manage bridges.

4) As in (3) if the only managed interface is placed on a bridge, this leaves NM with no interfaces to manage, so it sends out notification that there are no working interfaces on the system. This will cause things like openvpn to fail (and probably others). Again, fixing this problem is beyond the scope of this bug.
Comment 26 releng-rhel@redhat.com 2010-11-10 14:32:11 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.