Bug 1448233
| Summary: | os-net-config crashes when the primary interface of an active/passive bond has no carrier | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Andreas Karis <akaris> |
| Component: | os-net-config | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | Shai Revivo <srevivo> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 10.0 (Newton) | CC: | akaris, bfournie, hbrock, jslagle, mburns, mlammon, rhel-osp-director-maint |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-17 17:30:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi Andreas - sorry for missing this earlier. The fix for handling nics in the mapping file that have no carrier was made here - https://review.openstack.org/#/c/453284/ and has been backported to Ocata and Newton. Will that help with this? Andreas - I'm assuming that this is a duplicate and should be solved by fix for https://bugzilla.redhat.com/show_bug.cgi?id=1465595, let me know otherwise. *** This bug has been marked as a duplicate of bug 1465595 *** *** This bug has been marked as a duplicate of bug 1437320 *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: os-net-config crashes when the primary interface of an active/passive bond has no carrier Version-Release number of selected component (if applicable): OSP 10 Additional info: "Checking the interfaces it seems the networking is not assigned to that node." The NIC mapping is sensitive to interfaces in the up/down state. Please verify that all NICs are correctly plugged into the node and have carrier. Specifically, eno1 seems to be down in the case of this server. ovs-vsctl show ~~~ 95702f74-5766-4a19-a804-ad66c8db4694 ovs_version: "2.5.0" ~~~ ~~~ [akaris@collab-shell overcloud-compute-3.localdomain]$ cat ./sos_commands/networking/ip_address 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST> mtu 3000 qdisc mq state DOWN qlen 1000 link/ether 11:11:11:11:11:10 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 3000 qdisc mq state UP qlen 1000 link/ether 11:11:11:11:11:14 brd ff:ff:ff:ff:ff:ff inet6 fe80::1260:4bff:fea4:bd94/64 scope link valid_lft forever preferred_lft forever 4: ens2f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000 link/ether 00:00:00:aa:aa:88 brd ff:ff:ff:ff:ff:ff 5: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:00:00:aa:aa:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::0000:00ff:feaa:aa8c/64 scope link valid_lft forever preferred_lft forever 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether ab:ab:ab:ab:ab:ab brd ff:ff:ff:ff:ff:ff ~~~ All of the following from journalctl: os-net-config configuration: ~~~ May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: + NET_CONFIG='{"network_config": [{"routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "192.0.2.1"}], "use_dhcp": false, "type": "interface", "addresses": [{"ip_netmask": "192.0.2.27/24"}], "name": "ens2f0"}, {"dns_servers": ["8.8.8.8", "8.8.4.4"], "type": "ovs_bridge", "name": "br-ex", "members": [{"members": [{"mtu": 3000, "type": "i nterface", "primary": true, "name": "eno1"}, {"type": "interface", "name": "eno2", "mtu": 3000}], "type": "ovs_bond", "name": "bond1", "ovs_options": "bond_mode=active-backup", "mtu": 3000}, {"device": "bond1" , "routes": [{"default": true, "next_hop": "192.168.0.2"}], "type": "vlan", "addresses": [{"ip_netmask": "192.168.0.46/24"}], "vlan_id": 3000}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "1 72.16.0.25/24"}], "vlan_id": 3010}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.18.0.13/24"}], "vlan_id": 3011}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172. 17.0.22/24"}], "vlan_id": 3009, "mtu": 3000}]}]}' ~~~ ~~~ [akaris@collab-shell 01830004]$ echo '{"network_config": [{"routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "192.0.2.1"}], "use_dhcp": false, "type": "interface", "addresses": [{"ip_netmask": "192.0.2.27/24"}], "name": "ens2f0"}, {"dns_servers": ["8.8.8.8", "8.8.4.4"], "type": "ovs_bridge", "name": "br-ex", "members": [{"members": [{"mtu": 3000, "type": "interface", "primary": true, "name": "eno1"}, {"type": "interface", "name": "eno2", "mtu": 3000}], "type": "ovs_bond", "name": "bond1", "ovs_options": "bond_mode=active-backup", "mtu": 3000}, {"device": "bond1", "routes": [{"default": true, "next_hop": "192.168.0.2"}], "type": "vlan", "addresses": [{"ip_netmask": "192.168.0.46/24"}], "vlan_id": 3000}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.16.0.25/24"}], "vlan_id": 3010}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.18.0.13/24"}], "vlan_id": 3011}, {"device": "bond1", "type": "vlan", "addresses": [{"ip_netmask": "172.17.0.22/24"}], "vlan_id": 3009, "mtu": 3000}]}]}' | python -m json.tool { "network_config": [ { "addresses": [ { "ip_netmask": "192.0.2.27/24" } ], "name": "ens2f0", "routes": [ { "ip_netmask": "169.254.169.254/32", "next_hop": "192.0.2.1" } ], "type": "interface", "use_dhcp": false }, { "dns_servers": [ "8.8.8.8", "8.8.4.4" ], "members": [ { "members": [ { "mtu": 3000, "name": "eno1", "primary": true, "type": "interface" }, { "mtu": 3000, "name": "eno2", "type": "interface" } ], "mtu": 3000, "name": "bond1", "ovs_options": "bond_mode=active-backup", "type": "ovs_bond" }, { "addresses": [ { "ip_netmask": "192.168.0.46/24" } ], "device": "bond1", "routes": [ { "default": true, "next_hop": "192.168.0.2" } ], "type": "vlan", "vlan_id": 3000 }, { "addresses": [ { "ip_netmask": "172.16.0.25/24" } ], "device": "bond1", "type": "vlan", "vlan_id": 3010 }, { "addresses": [ { "ip_netmask": "172.18.0.13/24" } ], "device": "bond1", "type": "vlan", "vlan_id": 3011 }, { "addresses": [ { "ip_netmask": "172.17.0.22/24" } ], "device": "bond1", "mtu": 3000, "type": "vlan", "vlan_id": 3009 } ], "name": "br-ex", "type": "ovs_bridge" } ] } ~~~ On initialization of os-net-config, it creates an (unused) mapping: ~~~ May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] Ifcfg net config provider created. May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] nic1 mapped to: eno2 May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] nic2 mapped to: ens2f0 May 04 10:48:11 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:11 AM] [INFO] nic3 mapped to: ens2f1 ~~~ eno1 is not part of this mapping, meaning that it is down (has no carrier). os-net-config seems to initially configure the interfaces correctly. ~~~ (...) May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3000 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3009 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3010 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3011 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno1 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno1 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3011 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3010 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-bond1 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3009 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-bond1 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-bond1 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3000 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno1 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3009 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3000 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3010 May 04 10:48:13 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:13 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3011 (...) ~~~ eno1 does not seem to come up (not that NM notices that the interface is disconnected) - because of NM_CONTROLLER=no, the interface transitions to unmanaged. But we can clearly see that the interface is down). ~~~ (...) May 04 10:45:48 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909148.0120] device (eno1): state change: unmanaged -> unavailable (reason 'managed') [10 20 2] (...) May 04 10:45:48 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909148.0349] device (eno1): state change: unavailable -> disconnected (reason 'none') [20 30 0] (...) May 04 10:48:14 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909294.4968] ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-eno1 (abf4c85b-57cc-4484-4fa9-b4a71689c359,"System eno1") May 04 10:48:14 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909294.4968] ifcfg-rh: Ignoring connection /etc/sysconfig/network-scripts/ifcfg-eno1 (abf4c85b-57cc-4484-4fa9-b4a71689c359,"System eno1") due to NM_CONTROLLED=no. Unmanaged: interface-name:eno1. May 04 10:48:14 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909294.4969] device (eno1): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] May 04 10:48:15 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.0 eno1: using MSI-X IRQs: sp 61 fp[0] 63 ... fp[7] 70 May 04 10:48:15 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready ~~~ As opposed to eno2 ~~~ May 04 10:48:16 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:16 AM] [INFO] running ifup on interface: eno2 May 04 10:48:16 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: using MSI-X IRQs: sp 82 fp[0] 84 ... fp[7] 91 May 04 10:48:17 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready May 04 10:48:17 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909297.5947] device (eno2): link connected May 04 10:48:17 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit May 04 10:48:17 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready ~~~ Your templates force eno1 to be the `"primary": true`, hence os-net-config tries to set eno1 as the active-slave, fails and crashes: ~~~ May 04 10:48:43 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909323.0402] manager: (vlan3009): new Generic device (/org/freedesktop/NetworkManager/Devices/10) May 04 10:48:43 overcloud-compute-3.localdomain systemd[1]: Cannot add dependency job for unit microcode.service, ignoring: Invalid argument May 04 10:48:43 overcloud-compute-3.localdomain kernel: device vlan3009 entered promiscuous mode May 04 10:48:43 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909323.1576] device (vlan3009): link connected May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:48:47 AM] [INFO] Running ovs-appctl bond/set-active-slave ('bond1', 'eno1') May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Traceback (most recent call last): May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/bin/os-net-config", line 10, in <module> May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: sys.exit(main()) May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 187, in main May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: activate=not opts.no_activate) May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 938, in apply May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: self.bond_primary_ifaces[bond]) May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 262, in ovs_appctl May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: self.execute(msg, '/bin/ovs-appctl', action, *parameters) May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 224, in execute May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: processutils.execute(cmd, *args, **kwargs) May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: cmd=sanitized_cmd) May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Command: /bin/ovs-appctl bond/set-active-slave bond1 eno1 May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Exit code: 2 May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Stdout: u'' May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: Stderr: u'cannot make disabled slave active\novs-appctl: ovs-vswitchd: server returned an error\n' May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + RETVAL=1 May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + [[ 1 == 2 ]] May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + [[ 1 != 0 ]] May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + echo 'ERROR: os-net-config configuration failed.' May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ERROR: os-net-config configuration failed. May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + exit 1 May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + configure_safe_defaults May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + [[ 1 == 0 ]] May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + cat May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ++ ls /sys/class/net May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ++ grep -v '^lo$' May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + for iface in '$(ls /sys/class/net | grep -v ^lo$)' May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: ++ cat /sys/class/net/br-ex/addr_assign_type May 04 10:48:47 overcloud-compute-3.localdomain os-collect-config[3204]: + local mac_addr_type=3 ~~~ The above error leads to a new run of os-net-config, which generates bad network files (this may be a fallback mechanism, I'm not 100% sure): ~~~ May 04 10:49:02 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:02 AM] [INFO] cleaning up interface: vlan3011 May 04 10:49:02 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:02 AM] [INFO] running ifdown on interface: vlan3011 May 04 10:49:03 overcloud-compute-3.localdomain ovs-vsctl[6027]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex vlan3011 May 04 10:49:03 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:03 AM] [INFO] Removing config /etc/sysconfig/network-scripts/ifcfg-vlan3011 May 04 10:49:03 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:03 AM] [INFO] running ifdown on interface: eno2 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] running ifdown on interface: ens2f0 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0 May 04 10:49:04 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 10:49:04 AM] [INFO] running ifup on interface: eno2 May 04 10:49:05 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: using MSI-X IRQs: sp 82 fp[0] 84 ... fp[7] 91 May 04 10:49:05 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready May 04 10:49:06 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493909346.1797] device (eno2): link connected May 04 10:49:06 overcloud-compute-3.localdomain kernel: bnx2x 0000:04:00.1 eno2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit May 04 10:49:06 overcloud-compute-3.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready ~~~ With all network files now being configured as DHCP: ~~~ [akaris@collab-shell overcloud-compute-3.localdomain]$ cat etc/sysconfig/network-scripts/ifcfg-eno2 # This file is autogenerated by os-net-config DEVICE=eno2 ONBOOT=yes HOTPLUG=no NM_CONTROLLED=no BOOTPROTO=dhcp [akaris@collab-shell overcloud-compute-3.localdomain]$ cat etc/sysconfig/network-scripts/ifcfg-ens2f0 # This file is autogenerated by os-net-config DEVICE=ens2f0 ONBOOT=yes HOTPLUG=no NM_CONTROLLED=no BOOTPROTO=dhcp [akaris@collab-shell overcloud-compute-3.localdomain]$ cat etc/sysconfig/network-scripts/ifcfg-ens2f1 # This file is autogenerated by os-net-config DEVICE=ens2f1 ONBOOT=yes HOTPLUG=no NM_CONTROLLED=no BOOTPROTO=dhcp ~~~ Some time later, os-net-config is run again: ~~~ May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] running ifdown on bridge: br-ex May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3000 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3009 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3010 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3011 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno1 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno1 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3011 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3010 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-bond1 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3009 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-bond1 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-bond1 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan3000 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno1 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3009 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan3000 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3010 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan3011 May 04 11:06:17 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:17 AM] [INFO] running ifup on bridge: br-ex May 04 11:06:17 overcloud-compute-3.localdomain ovs-vsctl[7187]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex -- set bridge br-ex fail_mode=st ~~~ eno1 though is still disabled by NetworkManager, and hence the result is the same: ~~~ May 04 11:06:44 overcloud-compute-3.localdomain NetworkManager[1355]: <info> [1493910404.5982] device (vlan3009): link connected May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:06:49 AM] [INFO] Running ovs-appctl bond/set-active-slave ('bond1', 'eno1') May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Traceback (most recent call last): May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/bin/os-net-config", line 10, in <module> May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: sys.exit(main()) May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 187, in main May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: activate=not opts.no_activate) May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 938, in apply May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: self.bond_primary_ifaces[bond]) May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 262, in ovs_appctl May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: self.execute(msg, '/bin/ovs-appctl', action, *parameters) May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 224, in execute May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: processutils.execute(cmd, *args, **kwargs) May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: cmd=sanitized_cmd) May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Command: /bin/ovs-appctl bond/set-active-slave bond1 eno1 May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Exit code: 2 May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Stdout: u'' May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: Stderr: u'cannot make disabled slave active\novs-appctl: ovs-vswitchd: server returned an error\n' May 04 11:06:49 overcloud-compute-3.localdomain os-collect-config[3204]: + RETVAL=1 ~~~ And we get back to fallback mode: ~~~ May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eno2 May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-eno2 May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-eno2 May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-ens2f0 May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-ens2f0 May 04 11:07:06 overcloud-compute-3.localdomain os-collect-config[3204]: [2017/05/04 11:07:06 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0 ~~~