1448233 – os-net-config crashes when the primary interface of an active/passive bond has no carrier

Bug 1448233 - os-net-config crashes when the primary interface of an active/passive bond has no carrier

Summary: os-net-config crashes when the primary interface of an active/passive bond ha...

Keywords:
Status:	CLOSED DUPLICATE of bug 1437320
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	os-net-config
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	Shai Revivo
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-05-04 22:32 UTC by Andreas Karis
Modified:	2023-09-14 03:57 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-08-17 17:30:46 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	453284	0	None	MERGED	os_net_config should map nics that are down if nic is in mapping file	2020-10-29 00:12:19 UTC

Description Andreas Karis 2017-05-04 22:32:20 UTC

Description of problem:
os-net-config crashes when the primary interface of an active/passive bond has no carrier

Version-Release number of selected component (if applicable):
OSP 10

Additional info:
"Checking the interfaces it seems the networking is not assigned to that node."

The NIC mapping is sensitive to interfaces in the up/down state. Please verify that all NICs are correctly plugged into the node and have carrier. Specifically, eno1 seems to be down in the case of this server.

ovs-vsctl show
~~~
95702f74-5766-4a19-a804-ad66c8db4694
    ovs_version: "2.5.0"
~~~

~~~
[akaris@collab-shell overcloud-compute-3.localdomain]$ cat ./sos_commands/networking/ip_address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 3000 qdisc mq state DOWN qlen 1000
    link/ether 11:11:11:11:11:10 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 3000 qdisc mq state UP qlen 1000
    link/ether 11:11:11:11:11:14 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1260:4bff:fea4:bd94/64 scope link 
       valid_lft forever preferred_lft forever
4: ens2f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:00:00:aa:aa:88 brd ff:ff:ff:ff:ff:ff
5: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:00:00:aa:aa:8c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::0000:00ff:feaa:aa8c/64 scope link 
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ab:ab:ab:ab:ab:ab brd ff:ff:ff:ff:ff:ff
~~~

All of the following from journalctl:

os-net-config configuration:
~~~
May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: + NET_CONFIG='{"network_config": [{"routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "192.0.2.1"}], "use_dhcp": false, "type":
 "interface", "addresses": [{"ip_netmask": "192.0.2.27/24"}], "name": "ens2f0"}, {"dns_servers": ["8.8.8.8", "8.8.4.4"], "type": "ovs_bridge", "name": "br-ex", "members": [{"members": [{"mtu": 3000, "type": "i
nterface", "primary": true, "name": "eno1"}, {"type": "interface", "name": "eno2", "mtu": 3000}], "type": "ovs_bond", "name": "bond1", "ovs_options": "bond_mode=active-backup", "mtu": 3000}, {"device": "bond1"
, "routes": [{"default": true, "next_hop": "192.168.0.2"}], "type": "vlan", "addresses": [{"ip_netmask": "192.168.0.46/24"}], "vlan_id": 3000}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "1
72.16.0.25/24"}], "vlan_id": 3010}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.18.0.13/24"}], "vlan_id": 3011}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.
17.0.22/24"}], "vlan_id": 3009, "mtu": 3000}]}]}'
~~~

~~~
[akaris@collab-shell 01830004]$ echo '{"network_config": [{"routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "192.0.2.1"}], "use_dhcp": false, "type": "interface", "addresses": [{"ip_netmask": "192.0.2.27/24"}], "name": "ens2f0"}, {"dns_servers": ["8.8.8.8", "8.8.4.4"], "type": "ovs_bridge", "name": "br-ex", "members": [{"members": [{"mtu": 3000, "type": "interface", "primary": true, "name": "eno1"}, {"type": "interface", "name": "eno2", "mtu": 3000}], "type": "ovs_bond", "name": "bond1", "ovs_options": "bond_mode=active-backup", "mtu": 3000}, {"device": "bond1", "routes": [{"default": true, "next_hop": "192.168.0.2"}], "type": "vlan", "addresses": [{"ip_netmask": "192.168.0.46/24"}], "vlan_id": 3000}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.16.0.25/24"}], "vlan_id": 3010}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.18.0.13/24"}], "vlan_id": 3011}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.17.0.22/24"}], "vlan_id": 3009, "mtu": 3000}]}]}' | python -m json.tool
{
    "network_config": [
        {
            "addresses": [
                {
                    "ip_netmask": "192.0.2.27/24"
                }
            ],
            "name": "ens2f0",
            "routes": [
                {
                    "ip_netmask": "169.254.169.254/32",
                    "next_hop": "192.0.2.1"
                }
            ],
            "type": "interface",
            "use_dhcp": false
        },
        {
            "dns_servers": [
                "8.8.8.8",
                "8.8.4.4"
            ],
            "members": [
                {
                    "members": [
                        {
                            "mtu": 3000,
                            "name": "eno1",
                            "primary": true,
                            "type": "interface"
                        },
                        {
                            "mtu": 3000,
                            "name": "eno2",
                            "type": "interface"
                        }
                    ],
                    "mtu": 3000,
                    "name": "bond1",
                    "ovs_options": "bond_mode=active-backup",
                    "type": "ovs_bond"
                },
                {
                    "addresses": [
                        {
                            "ip_netmask": "192.168.0.46/24"
                        }
                    ],
                    "device": "bond1",
                    "routes": [
                        {
                            "default": true,
                            "next_hop": "192.168.0.2"
                        }
                    ],
                    "type": "vlan",
                    "vlan_id": 3000
                },
                {
                    "addresses": [
                        {
                            "ip_netmask": "172.16.0.25/24"
                        }
                    ],
                    "device": "bond1",
                    "type": "vlan",
                    "vlan_id": 3010
                },
                {
                    "addresses": [
                        {
                            "ip_netmask": "172.18.0.13/24"
                        }
                    ],
                    "device": "bond1",
                    "type": "vlan",
                    "vlan_id": 3011
                },
                {
                    "addresses": [
                        {
                            "ip_netmask": "172.17.0.22/24"
                        }
                    ],
                    "device": "bond1",
                    "mtu": 3000,
                    "type": "vlan",
                    "vlan_id": 3009
                }
            ],
            "name": "br-ex",
            "type": "ovs_bridge"
        }
    ]
}
~~~

On initialization of os-net-config, it creates an (unused) mapping:
~~~
May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] Ifcfg net config provider created.
May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] nic1 mapped to: eno2
May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] nic2 mapped to: ens2f0
May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] nic3 mapped to: ens2f1
~~~
eno1 is not part of this mapping, meaning that it is down (has no carrier).


os-net-config seems to initially configure the interfaces correctly.
~~~
(...)
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3000
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3009
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3010
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3011
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno1
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno1
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3011
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3010
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-bond1
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3009
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-bond1
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-bond1
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3000
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno1
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3009
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3000
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3010
May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3011
(...)
~~~

eno1 does not seem to come up (not that NM notices that the interface is disconnected) - because of NM_CONTROLLER=no, the interface transitions to unmanaged. But we can clearly see that the interface is down).
~~~
(...)
May 04 10:45:48 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909148.0120] device (eno1): state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
(...)
May 04 10:45:48 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909148.0349] device (eno1): state change: unavailable -> disconnected (reason 'none') [20 30 0]
(...)
May 04 10:48:14 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909294.4968] ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-eno1 (abf4c85b-57cc-4484-4fa9-b4a71689c359,"System eno1")
May 04 10:48:14 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909294.4968] ifcfg-rh: Ignoring connection /etc/sysconfig/network-scripts/ifcfg-eno1 (abf4c85b-57cc-4484-4fa9-b4a71689c359,"System eno1") due to NM_CONTROLLED=no. Unmanaged: interface-name:eno1.
May 04 10:48:14 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909294.4969] device (eno1): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3]
May 04 10:48:15 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.0 eno1: using MSI-X  IRQs: sp 61  fp[0] 63 ... fp[7] 70
May 04 10:48:15 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
~~~

As opposed to eno2
~~~
May 04 10:48:16 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:16 AM] [INFO] running ifup on interface: eno2
May 04 10:48:16 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: using MSI-X  IRQs: sp 82  fp[0] 84 ... fp[7] 91
May 04 10:48:17 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
May 04 10:48:17 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909297.5947] device (eno2): link connected
May 04 10:48:17 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
May 04 10:48:17 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
~~~

Your templates force eno1 to be the  `"primary": true`, hence os-net-config tries to set eno1 as the active-slave, fails and crashes: 
~~~
May 04 10:48:43 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909323.0402] manager: (vlan3009): new Generic device (/org/freedesktop/NetworkManager/Devices/10)
May 04 10:48:43 overcloud-compute-3.localdomain systemd[1]: Cannot add dependency job for unit microcode.service, ignoring: Invalid argument
May 04 10:48:43 overcloud-compute-3.localdomain kernel: device vlan3009 entered promiscuous mode
May 04 10:48:43 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909323.1576] device (vlan3009): link connected
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:47 AM] [INFO] Running ovs-appctl bond/set-active-slave ('bond1', 'eno1')
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Traceback (most recent call last):
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/bin/os-net-config", line 10, in <module>
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: sys.exit(main())
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 187, in main
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: activate=not opts.no_activate)
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 938, in apply
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: self.bond_primary_ifaces[bond])
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 262, in ovs_appctl
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: self.execute(msg, '/bin/ovs-appctl', action, *parameters)
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 224, in execute
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: processutils.execute(cmd, *args, **kwargs)
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: cmd=sanitized_cmd)
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Command: /bin/ovs-appctl bond/set-active-slave bond1 eno1
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Exit code: 2
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Stdout: u''
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Stderr: u'cannot make disabled slave active\novs-appctl: ovs-vswitchd: server returned an error\n'
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + RETVAL=1
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + [[ 1 == 2 ]]
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + [[ 1 != 0 ]]
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + echo 'ERROR: os-net-config configuration failed.'
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ERROR: os-net-config configuration failed.
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + exit 1
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + configure_safe_defaults
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + [[ 1 == 0 ]]
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + cat
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ++ ls /sys/class/net
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ++ grep -v '^lo$'
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + for iface in '$(ls /sys/class/net | grep -v ^lo$)'
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ++ cat /sys/class/net/br-ex/addr_assign_type
May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + local mac_addr_type=3
~~~

The above error leads to a new run of os-net-config, which generates bad network files (this may be a fallback mechanism, I'm not 100% sure):
~~~
May 04 10:49:02 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:02 AM] [INFO] cleaning up interface: vlan3011
May 04 10:49:02 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:02 AM] [INFO] running ifdown on interface: vlan3011
May 04 10:49:03 overcloud-compute-3.localdomain ovs-vsctl[6027]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex vlan3011
May 04 10:49:03 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:03 AM] [INFO] Removing config /etc/sysconfig/network-scripts/ifcfg-vlan3011
May 04 10:49:03 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:03 AM] [INFO] running ifdown on interface: eno2
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] running ifdown on interface: ens2f0
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0
May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] running ifup on interface: eno2
May 04 10:49:05 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: using MSI-X  IRQs: sp 82  fp[0] 84 ... fp[7] 91
May 04 10:49:05 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
May 04 10:49:06 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493909346.1797] device (eno2): link connected
May 04 10:49:06 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
May 04 10:49:06 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
~~~

With all network files now being configured as DHCP:
~~~
[akaris@collab-shell overcloud-compute-3.localdomain]$ cat etc/sysconfig/network-scripts/ifcfg-eno2
# This file is autogenerated by os-net-config
DEVICE=eno2
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
BOOTPROTO=dhcp
[akaris@collab-shell overcloud-compute-3.localdomain]$ cat etc/sysconfig/network-scripts/ifcfg-ens2f0 
# This file is autogenerated by os-net-config
DEVICE=ens2f0
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
BOOTPROTO=dhcp
[akaris@collab-shell overcloud-compute-3.localdomain]$ cat etc/sysconfig/network-scripts/ifcfg-ens2f1
# This file is autogenerated by os-net-config
DEVICE=ens2f1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
BOOTPROTO=dhcp
~~~

Some time later, os-net-config is run again:
~~~
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] running ifdown on bridge: br-ex
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3000
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3009
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3010
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3011
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno1
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno1
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3011
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3010
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-bond1
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3009
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-bond1
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-bond1
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3000
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno1
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3009
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3000
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3010
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3011
May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] running ifup on bridge: br-ex
May 04 11:06:17 overcloud-compute-3.localdomain ovs-vsctl[7187]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex -- set bridge br-ex fail_mode=st
~~~

eno1 though is still disabled by NetworkManager, and hence the result is the same:
~~~
May 04 11:06:44 overcloud-compute-3.localdomain NetworkManager[1355]: <info>  [1493910404.5982] device (vlan3009): link connected
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:49 AM] [INFO] Running ovs-appctl bond/set-active-slave ('bond1', 'eno1')
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Traceback (most recent call last):
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/bin/os-net-config", line 10, in <module>
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: sys.exit(main())
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 187, in main
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: activate=not opts.no_activate)
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 938, in apply
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: self.bond_primary_ifaces[bond])
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 262, in ovs_appctl
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: self.execute(msg, '/bin/ovs-appctl', action, *parameters)
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 224, in execute
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: processutils.execute(cmd, *args, **kwargs)
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: cmd=sanitized_cmd)
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Command: /bin/ovs-appctl bond/set-active-slave bond1 eno1
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Exit code: 2
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Stdout: u''
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Stderr: u'cannot make disabled slave active\novs-appctl: ovs-vswitchd: server returned an error\n'
May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: + RETVAL=1
~~~

And we get back to fallback mode:
~~~
May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2
May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2
May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2
May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0
May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0
May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0
~~~

Comment 1 Bob Fournier 2017-08-04 20:17:23 UTC

Hi Andreas - sorry for missing this earlier.  The fix for handling nics in the mapping file that have no carrier was made here - https://review.openstack.org/#/c/453284/ and has been backported to Ocata and Newton.  Will that help with this?

Comment 2 Bob Fournier 2017-08-17 15:01:46 UTC

Andreas - I'm assuming that this is a duplicate and should be solved by fix for https://bugzilla.redhat.com/show_bug.cgi?id=1465595, let me know otherwise.

Comment 3 Bob Fournier 2017-08-17 17:30:46 UTC


*** This bug has been marked as a duplicate of bug 1465595 ***

Comment 4 Bob Fournier 2017-08-17 17:34:08 UTC


*** This bug has been marked as a duplicate of bug 1437320 ***

Comment 5 Red Hat Bugzilla 2023-09-14 03:57:20 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days

Note You need to log in before you can comment on or make changes to this bug.