Bug 1355656

Created attachment 1178783 [details]
All logs

I can not reproduce this with cockpit 0.116 and NetworkManager 1.2.2 on Fedora 24 with a otherwise unused network interface instead of em1.

After deleting the bridge, the network interface can be brought up with ifup, nmcli, and Cockpit.

I'll try your kickstart file, but I guess this bug has been fixed already somewhere.

> 1. Install redhat-virtualization-host-4.0-20160708.0.x86_64 with kickstart file in attachment    

Can you give exact instructions for how to do this?  I am not sufficiently familiar with kickstart files, unfortunately.

(In reply to Marius Vollmer from comment #4)
> > 1. Install redhat-virtualization-host-4.0-20160708.0.x86_64 with kickstart file in attachment    
> 
> Can you give exact instructions for how to do this?  I am not sufficiently
> familiar with kickstart files, unfortunately.

Put the liveimg.squashfs and kickstart.ks on http server, then install  liveimg.squashfs via PXE server which will call kickstart.ks to access install process.

But please do not worry this, I think this issue is no matter with the kickstart file, also can reproduce with iso install.

I reproduced this bug with cockpit-0.114-2.el7.x86_64 and NetworkManager-1.0.6-30.el7_2.x86_64.

I think the crucial issue maybe the NIC ifcfg file: ifcfg-em1.

1. After remove bridge0, the ifcfg-em1 looks like below, can not up em1 via "ifup em1".

# cat ifcfg-em1
NAME="em1"
DEVICE="em1"
ONBOOT=yes
NETBOOT=yes
UUID="22a273e8-3c1e-4d1c-abdf-7da8df63178b"
IPV6INIT=yes
BOOTPROTO=dhcp
TYPE=Ethernet
BRIDGE=aa680956-01e5-4c90-9193-dac38cf8ab03
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no

2. Fix ifcfg-em1, delete "BRIDGE=aa680956-01e5-4c90-9193-dac38cf8ab03", then can up em1 successful.

Created attachment 1187332 [details]
After remove bridge0, all files in /etc/sysconfig/network-scripts

(In reply to Marius Vollmer from comment #3)
> I can not reproduce this with cockpit 0.116 and NetworkManager 1.2.2 on
> Fedora 24 with a otherwise unused network interface instead of em1.
> 
> After deleting the bridge, the network interface can be brought up with
> ifup, nmcli, and Cockpit.
> 
> I'll try your kickstart file, but I guess this bug has been fixed already
> somewhere.

Maybe it has been fixed already somewhere. But for RHVH 4.0 build, the latest package is cockpit-0.114-2.el7.x86_64 and NetworkManager-1.0.6-30.el7_2.x86_64, so can still encounter this issue.

(In reply to Marius Vollmer from comment #3)
> I can not reproduce this with cockpit 0.116 and NetworkManager 1.2.2 on
> Fedora 24 with a otherwise unused network interface instead of em1.

I can reproduce it now.

NetworkManager gets itself into a inconsistent state (see below) and I managed to stay on the good side of that inconsistency somehow.  Doing a "nmcli c reload" at the right time will expose the inconsistency.

(In reply to Huijuan Zhao from comment #5)
> 2. Fix ifcfg-em1, delete "BRIDGE=aa680956-01e5-4c90-9193-dac38cf8ab03", then
> can up em1 successful.

Yes, that's the inconsistency, thanks for making me see it.  I'll write up reproduction steps without Cockpit or RHEV and then assign to NetworkManager.

NetworkManager leaves a bogus BRIDGE= line in ifcfg when a bridge slave is removed from a bridge.  A "nmcli c reload" will spontanously turn that interface into a slave again.

Steps:

- Take a otherwise unused network interface, "ens14" in my case.

- Make sure there are regular non-slave connection settings for that interface.

# nmcli dev con ens14
Device 'ens14' successfully activated with 'e0ff484e-c60c-426f-802e-96ba83832728'.
# nmcli dev dis ens14
Device 'ens14' successfully disconnected.
# nmcli c s ens14 | grep 'master:\|slave-type:'
connection.master:                      --
connection.slave-type:                  --
  
- Create a bridge.

# nmcli con add type bridge
Connection 'bridge' (3eb8e7cd-357f-431d-9941-d45e5d093db5) successfully added.

- Make ens14 a slave of the bridge.

# nmcli con mod ens14 connection.master 3eb8e7cd-357f-431d-9941-d45e5d093db5 connection.slave-type bridge
# nmcli c s ens14 | grep 'master:\|slave-type:'
connection.master:                      3eb8e7cd-357f-431d-9941-d45e5d093db5
connection.slave-type:                  bridge
# grep BRIDGE /etc/sysconfig/network-scripts/ifcfg-ens14
BRIDGE=3eb8e7cd-357f-431d-9941-d45e5d093db5

- Liberate ens14 again:

# nmcli con mod ens14 connection.master "" connection.slave-type ""
# nmcli c s ens14 | grep 'master:\|slave-type:'
connection.master:                      --
connection.slave-type:                  --
# grep BRIDGE /etc/sysconfig/network-scripts/ifcfg-ens14
BRIDGE=3eb8e7cd-357f-431d-9941-d45e5d093db5

[ BRIDGE is still there. ]

- Delete bridge

# nmcli c del bridge
Connection 'bridge' (3eb8e7cd-357f-431d-9941-d45e5d093db5) successfully deleted.

- Bring ens14 up

# nmcli dev con ens14
Device 'ens14' successfully activated with 'e0ff484e-c60c-426f-802e-96ba83832728'.
# nmcli dev dis ens14
Device 'ens14' successfully disconnected.

[ This is where I stopped earlier and thought I can't reproduce the bug. ]

- Reload configuration

# nmcli con reload
# nmcli c s ens14 | grep 'master:\|slave-type:'
connection.master:                      3eb8e7cd-357f-431d-9941-d45e5d093db5
connection.slave-type:                  bridge

[ ens14 is a slave of a non-existing bridge. ]

- Try to bring ens14 up

# nmcli dev con ens14
Error: Device activation failed: Can not find a master for ens14: Master connection not found or invalid

- Clean up

# sed -i /etc/sysconfig/network-scripts/ifcfg-ens14 -e '/^BRIDGE=/d'
# nmcli con reload
# nmcli c s ens14 | grep 'master:\|slave-type:'
connection.master:                      --
connection.slave-type:                  --
# nmcli dev con ens14
Device 'ens14' successfully activated with 'e0ff484e-c60c-426f-802e-96ba83832728'.

Opps, the version.  I used

    NetworkManager-1.2.2-2.fc24.x86_64

on Fedora 24.

Created attachment 1187505 [details]
[PATCH] ifcfg-rh: ensure master is cleared when updating a connection

Great, that was fast!  Thanks!

This issue is fixed in redhat-virtualization-host-4.0-20160812.0.

Test version:
redhat-virtualization-host-4.0-20160812.0
imgbased-0.8.4-1.el7ev.noarch
cockpit-ws-0.114-2.el7.x86_64
cockpit-ovirt-dashboard-0.10.6-1.3.6.el7ev.noarch
redhat-virtualization-host-image-update-placeholder-4.0-1.el7.noarch

Steps to Reproduce:
1. Install redhat-virtualization-host-4.0-20160812.0.x86_64 with kickstart file in attachment    
2. Login cockpit website hostIP:9090 with root account
3. Select Networking page, create bridge0 on em1.
4. Delete bridge0 on em1 via cockpit.

Test results:
1. After step4, em1 can be up automatically

(In reply to Huijuan Zhao from comment #13)
> This issue is fixed in redhat-virtualization-host-4.0-20160812.0.
> 
> Test version:
> redhat-virtualization-host-4.0-20160812.0
> imgbased-0.8.4-1.el7ev.noarch
> cockpit-ws-0.114-2.el7.x86_64
> cockpit-ovirt-dashboard-0.10.6-1.3.6.el7ev.noarch
> redhat-virtualization-host-image-update-placeholder-4.0-1.el7.noarch
> 
> Steps to Reproduce:
> 1. Install redhat-virtualization-host-4.0-20160812.0.x86_64 with kickstart
> file in attachment    
> 2. Login cockpit website hostIP:9090 with root account
> 3. Select Networking page, create bridge0 on em1.
> 4. Delete bridge0 on em1 via cockpit.
> 
> Test results:
> 1. After step4, em1 can be up automatically

Update:
After reboot RHVH, the slave NIC em1 can not be up again.
So this issue is not fixed completely

(In reply to Beniamino Galvani from comment #11)
> Created attachment 1187505 [details]
> [PATCH] ifcfg-rh: ensure master is cleared when updating a connection

+ svSetValue (ifcfg, "DEVICETYPE", NULL, FALSE);

this first clears the DEVICETYPE setting, but latter write_connection_setting() may set it again.

Not sure, but doesn't ifcfg-writer aim to preserve comments and positions?
So that means, a file like:

   # set the DEVICETYPE
   DEVICETYPE=TeamPort

   #some other property
   UUID=uuid1


changes to:

   # set the DEVICETYPE
 
   #some other property
   UUID=uuid1
   DEVICETYPE=TeamPort



maybe, we can not clear "DEVICETYPE" in write_connection(), but only in write_connection_setting() if (and only if) it is to be actually cleared?




in write_connection_setting(), say we hit the condition:
    if (nm_setting_connection_is_slave_type (s_con, NM_SETTING_BOND_SET...
        svSetValue (ifcfg, "MASTER", master, FALSE);
        svSetValue (ifcfg, "SLAVE", "yes", FALSE);
I think, we should then also unset "BRIDGE", "TEAM_MASTER" (and "DEVICETYPE").
-- same for other branches.


(otherwise, patch looks right)

Created attachment 1192070 [details]
[PATCH v2] ifcfg-rh: ensure master is cleared when updating a connection

(In reply to Thomas Haller from comment #15)
> Not sure, but doesn't ifcfg-writer aim to preserve comments and positions?

How about v2?

> maybe, we can not clear "DEVICETYPE" in write_connection(), but only in
> write_connection_setting() if (and only if) it is to be actually cleared?

Fixed.

(In reply to Beniamino Galvani from comment #16)
> Created attachment 1192070 [details]
> [PATCH v2] ifcfg-rh: ensure master is cleared when updating a connection

v2 lgtm

lgtm

Applied:

https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=2750714802eb5f45fe381711991c6066c0270ce5

https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?h=nm-1-4&id=0944a86182e5c6be8d6b235917c5cdd79e9564e8

Due to Bug 1367669, can not verify this bug completely now.
So I will verify this bug once RHVH support network in cockpit.

I can convert ethernet to bridge slave and vice versa w/o residues in ifcfg file.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2581.html

*** Bug 1362358 has been marked as a duplicate of this bug. ***

*** Bug 1401416 has been marked as a duplicate of this bug. ***