rhel-osp-director: upgrade from 7.3 ->8.0 -> 9.0 failed, br-ex on the controllers was missing ip address and pacemaker was unable to bring up the vip for that network. Environment golden singed puddle ( undercloud after upgrade attempts): ---------------------------------------- openstack-heat-common-5.0.1-6.el7ost.noarch openstack-heat-api-cloudwatch-5.0.1-6.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch heat-cfntools-1.2.8-2.el7.noarch openstack-heat-api-cfn-5.0.1-6.el7ost.noarch openstack-heat-templates-0-0.8.20150605git.el7ost.noarch openstack-heat-api-5.0.1-6.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch python-heatclient-1.0.0-1.el7ost.noarch openstack-heat-engine-5.0.1-6.el7ost.noarch instack-0.0.8-2.el7ost.noarch instack-undercloud-2.2.7-7.el7ost.noarch Steps: ------- Attempt to upgrade using ospd from 7.3 to 8.0 to 9.0 Results: -------- after successful upgrade from 7.3 to 8.0 the next overcloud upgrade from 8.0 to 9.0 UPGRADE_FAILED. from the investigation it seems that, the br-ex on the overlcoud controllers was down and missing ip address, that caused pacemaker to be unable to bring the vip for that network. Note: it was possible to manually start the br-ex on the controllers. pcs status: Cluster name: tripleo_cluster Last updated: Fri Aug 5 20:05:18 2016 Last change: Fri Aug 5 06:27:14 2016 by root via cibadmin on overcloud-controller-2 Stack: corosync Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum 3 nodes and 112 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: ip-10.19.184.210 (ocf::heartbeat:IPaddr2): Stopped (unmanaged) ip-192.168.200.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 (unmanaged) Clone Set: haproxy-clone [haproxy] (unmanaged) haproxy (systemd:haproxy): Started overcloud-controller-2 (unmanaged) haproxy (systemd:haproxy): Started overcloud-controller-0 (unmanaged) haproxy (systemd:haproxy): Started overcloud-controller-1 (unmanaged) ip-192.168.0.6 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (unmanaged) ip-10.19.105.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (unmanaged) ip-10.19.104.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 (unmanaged) Master/Slave Set: galera-master [galera] (unmanaged) galera (ocf::heartbeat:galera): Master overcloud-controller-2 (unmanaged) galera (ocf::heartbeat:galera): Master overcloud-controller-0 (unmanaged) galera (ocf::heartbeat:galera): Master overcloud-controller-1 (unmanaged) ip-10.19.104.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (unmanaged) Master/Slave Set: redis-master [redis] (unmanaged) redis (ocf::heartbeat:redis): Master overcloud-controller-2 (unmanaged) redis (ocf::heartbeat:redis): Started overcloud-controller-0 (unmanaged) redis (ocf::heartbeat:redis): Started overcloud-controller-1 (unmanaged) Clone Set: mongod-clone [mongod] (unmanaged) mongod (systemd:mongod): Started overcloud-controller-2 (unmanaged) mongod (systemd:mongod): Started overcloud-controller-0 (unmanaged) mongod (systemd:mongod): Started overcloud-controller-1 (unmanaged) Clone Set: rabbitmq-clone [rabbitmq] (unmanaged) rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2 (unmanaged) rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0 (unmanaged) rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1 (unmanaged) Clone Set: memcached-clone [memcached] (unmanaged) memcached (systemd:memcached): Started overcloud-controller-2 (unmanaged) memcached (systemd:memcached): Started overcloud-controller-0 (unmanaged) memcached (systemd:memcached): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] (unmanaged) openstack-nova-scheduler (systemd:openstack-nova-scheduler): Started overcloud-controller-2 (unmanaged) openstack-nova-scheduler (systemd:openstack-nova-scheduler): Started overcloud-controller-0 (unmanaged) openstack-nova-scheduler (systemd:openstack-nova-scheduler): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-l3-agent-clone [neutron-l3-agent] (unmanaged) neutron-l3-agent (systemd:neutron-l3-agent): Started overcloud-controller-2 (unmanaged) neutron-l3-agent (systemd:neutron-l3-agent): Started overcloud-controller-0 (unmanaged) neutron-l3-agent (systemd:neutron-l3-agent): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier] (unmanaged) openstack-ceilometer-alarm-notifier (systemd:openstack-ceilometer-alarm-notifier): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-alarm-notifier (systemd:openstack-ceilometer-alarm-notifier): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-alarm-notifier (systemd:openstack-ceilometer-alarm-notifier): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-heat-engine-clone [openstack-heat-engine] (unmanaged) openstack-heat-engine (systemd:openstack-heat-engine): Started overcloud-controller-2 (unmanaged) openstack-heat-engine (systemd:openstack-heat-engine): Started overcloud-controller-0 (unmanaged) openstack-heat-engine (systemd:openstack-heat-engine): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api] (unmanaged) openstack-ceilometer-api (systemd:openstack-ceilometer-api): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-api (systemd:openstack-ceilometer-api): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-api (systemd:openstack-ceilometer-api): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] (unmanaged) neutron-metadata-agent (systemd:neutron-metadata-agent): Started overcloud-controller-2 (unmanaged) neutron-metadata-agent (systemd:neutron-metadata-agent): Started overcloud-controller-0 (unmanaged) neutron-metadata-agent (systemd:neutron-metadata-agent): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] (unmanaged) neutron-ovs-cleanup (ocf::neutron:OVSCleanup): Started overcloud-controller-2 (unmanaged) neutron-ovs-cleanup (ocf::neutron:OVSCleanup): Started overcloud-controller-0 (unmanaged) neutron-ovs-cleanup (ocf::neutron:OVSCleanup): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] (unmanaged) neutron-netns-cleanup (ocf::neutron:NetnsCleanup): Started overcloud-controller-2 (unmanaged) neutron-netns-cleanup (ocf::neutron:NetnsCleanup): Started overcloud-controller-0 (unmanaged) neutron-netns-cleanup (ocf::neutron:NetnsCleanup): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-heat-api-clone [openstack-heat-api] (unmanaged) openstack-heat-api (systemd:openstack-heat-api): Started overcloud-controller-2 (unmanaged) openstack-heat-api (systemd:openstack-heat-api): Started overcloud-controller-0 (unmanaged) openstack-heat-api (systemd:openstack-heat-api): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] (unmanaged) openstack-cinder-scheduler (systemd:openstack-cinder-scheduler): Started overcloud-controller-2 (unmanaged) openstack-cinder-scheduler (systemd:openstack-cinder-scheduler): Started overcloud-controller-0 (unmanaged) openstack-cinder-scheduler (systemd:openstack-cinder-scheduler): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-api-clone [openstack-nova-api] (unmanaged) openstack-nova-api (systemd:openstack-nova-api): Started overcloud-controller-2 (unmanaged) openstack-nova-api (systemd:openstack-nova-api): Started overcloud-controller-0 (unmanaged) openstack-nova-api (systemd:openstack-nova-api): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] (unmanaged) openstack-heat-api-cloudwatch (systemd:openstack-heat-api-cloudwatch): Started overcloud-controller-2 (unmanaged) openstack-heat-api-cloudwatch (systemd:openstack-heat-api-cloudwatch): Started overcloud-controller-0 (unmanaged) openstack-heat-api-cloudwatch (systemd:openstack-heat-api-cloudwatch): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] (unmanaged) openstack-ceilometer-collector (systemd:openstack-ceilometer-collector): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-collector (systemd:openstack-ceilometer-collector): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-collector (systemd:openstack-ceilometer-collector): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-keystone-clone [openstack-keystone] (unmanaged) openstack-keystone (systemd:openstack-keystone): Started overcloud-controller-2 (unmanaged) openstack-keystone (systemd:openstack-keystone): Started overcloud-controller-0 (unmanaged) openstack-keystone (systemd:openstack-keystone): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] (unmanaged) openstack-nova-consoleauth (systemd:openstack-nova-consoleauth): Started overcloud-controller-2 (unmanaged) openstack-nova-consoleauth (systemd:openstack-nova-consoleauth): Started overcloud-controller-0 (unmanaged) openstack-nova-consoleauth (systemd:openstack-nova-consoleauth): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-glance-registry-clone [openstack-glance-registry] (unmanaged) openstack-glance-registry (systemd:openstack-glance-registry): Started overcloud-controller-2 (unmanaged) openstack-glance-registry (systemd:openstack-glance-registry): Started overcloud-controller-0 (unmanaged) openstack-glance-registry (systemd:openstack-glance-registry): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] (unmanaged) openstack-ceilometer-notification (systemd:openstack-ceilometer-notification): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-notification (systemd:openstack-ceilometer-notification): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-notification (systemd:openstack-ceilometer-notification): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-cinder-api-clone [openstack-cinder-api] (unmanaged) openstack-cinder-api (systemd:openstack-cinder-api): Started overcloud-controller-2 (unmanaged) openstack-cinder-api (systemd:openstack-cinder-api): Started overcloud-controller-0 (unmanaged) openstack-cinder-api (systemd:openstack-cinder-api): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] (unmanaged) neutron-dhcp-agent (systemd:neutron-dhcp-agent): Started overcloud-controller-2 (unmanaged) neutron-dhcp-agent (systemd:neutron-dhcp-agent): Started overcloud-controller-0 (unmanaged) neutron-dhcp-agent (systemd:neutron-dhcp-agent): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-glance-api-clone [openstack-glance-api] (unmanaged) openstack-glance-api (systemd:openstack-glance-api): Started overcloud-controller-2 (unmanaged) openstack-glance-api (systemd:openstack-glance-api): Started overcloud-controller-0 (unmanaged) openstack-glance-api (systemd:openstack-glance-api): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] (unmanaged) neutron-openvswitch-agent (systemd:neutron-openvswitch-agent): Started overcloud-controller-2 (unmanaged) neutron-openvswitch-agent (systemd:neutron-openvswitch-agent): Started overcloud-controller-0 (unmanaged) neutron-openvswitch-agent (systemd:neutron-openvswitch-agent): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] (unmanaged) openstack-nova-novncproxy (systemd:openstack-nova-novncproxy): Started overcloud-controller-2 (unmanaged) openstack-nova-novncproxy (systemd:openstack-nova-novncproxy): Started overcloud-controller-0 (unmanaged) openstack-nova-novncproxy (systemd:openstack-nova-novncproxy): Started overcloud-controller-1 (unmanaged) Clone Set: delay-clone [delay] (unmanaged) delay (ocf::heartbeat:Delay): Started overcloud-controller-2 (unmanaged) delay (ocf::heartbeat:Delay): Started overcloud-controller-0 (unmanaged) delay (ocf::heartbeat:Delay): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-server-clone [neutron-server] (unmanaged) neutron-server (systemd:neutron-server): Started overcloud-controller-2 (unmanaged) neutron-server (systemd:neutron-server): Started overcloud-controller-0 (unmanaged) neutron-server (systemd:neutron-server): Started overcloud-controller-1 (unmanaged) Clone Set: httpd-clone [httpd] (unmanaged) httpd (systemd:httpd): Started overcloud-controller-2 (unmanaged) httpd (systemd:httpd): Started overcloud-controller-0 (unmanaged) httpd (systemd:httpd): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] (unmanaged) openstack-ceilometer-central (systemd:openstack-ceilometer-central): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-central (systemd:openstack-ceilometer-central): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-central (systemd:openstack-ceilometer-central): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator] (unmanaged) openstack-ceilometer-alarm-evaluator (systemd:openstack-ceilometer-alarm-evaluator): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-alarm-evaluator (systemd:openstack-ceilometer-alarm-evaluator): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-alarm-evaluator (systemd:openstack-ceilometer-alarm-evaluator): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] (unmanaged) openstack-heat-api-cfn (systemd:openstack-heat-api-cfn): Started overcloud-controller-2 (unmanaged) openstack-heat-api-cfn (systemd:openstack-heat-api-cfn): Started overcloud-controller-0 (unmanaged) openstack-heat-api-cfn (systemd:openstack-heat-api-cfn): Started overcloud-controller-1 (unmanaged) openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 (unmanaged) Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] (unmanaged) openstack-nova-conductor (systemd:openstack-nova-conductor): Started overcloud-controller-2 (unmanaged) openstack-nova-conductor (systemd:openstack-nova-conductor): Started overcloud-controller-0 (unmanaged) openstack-nova-conductor (systemd:openstack-nova-conductor): Started overcloud-controller-1 (unmanaged) Failed Actions: * ip-10.19.184.210_start_0 on overcloud-controller-2 'unknown error' (1): call=213, status=complete, exitreason='Unable to find nic or netmask.', last-rc-change='Fri Aug 5 05:41:29 2016', queued=0ms, exec=69ms * ip-10.19.184.210_start_0 on overcloud-controller-0 'unknown error' (1): call=205, status=complete, exitreason='Unable to find nic or netmask.', last-rc-change='Fri Aug 5 05:41:26 2016', queued=0ms, exec=71ms * ip-10.19.184.210_start_0 on overcloud-controller-1 'unknown error' (1): call=212, status=complete, exitreason='Unable to find nic or netmask.', last-rc-change='Fri Aug 5 05:41:27 2016', queued=0ms, exec=33ms PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled logs: ------ Aug 5 00:00:42 localhost os-collect-config: dib-run-parts Fri Aug 5 00:00:42 EDT 2016 20-os-apply-config completed Aug 5 00:00:42 localhost os-collect-config: dib-run-parts Fri Aug 5 00:00:42 EDT 2016 Running /usr/libexec/os-refresh-config/configure.d/20-os-net-config Aug 5 00:00:42 localhost os-collect-config: ++ os-apply-config --key os_net_config --type raw --key-default '' Aug 5 00:00:42 localhost os-collect-config: + NET_CONFIG='{"network_config": [{"routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "192.168.0.1"}, {"default": true, "next_hop": "192.168.0.1"}], "use_dhcp": false, "typ e": "interface", "name": "nic3", "addresses": [{"ip_netmask": "192.168.0.11/24"}]}, {"dns_servers": ["10.16.36.29"], "name": "br-ex", "members": [{"type": "interface", "name": "nic1", "primary": true}], "routes": [{"ip_netmask ": "0.0.0.0/0", "next_hop": "10.19.184.254"}], "use_dhcp": false, "type": "ovs_bridge", "addresses": [{"ip_netmask": "10.19.184.213/24"}]}, {"use_dhcp": false, "type": "ovs_bridge", "name": "br-nic2", "members": [{"type": "int erface", "name": "nic2", "primary": true}, {"type": "vlan", "addresses": [{"ip_netmask": "10.19.105.15/24"}], "vlan_id": 185}], "addresses": [{"ip_netmask": "10.19.104.15/24"}]}, {"use_dhcp": false, "type": "ovs_bridge", "name ": "br-nic4", "members": [{"type": "interface", "name": "nic4", "primary": true}, {"type": "vlan", "addresses": [{"ip_netmask": "192.168.200.15/24"}], "vlan_id": 203}], "addresses": [{"ip_netmask": "192.168.150.13/24"}]}]}' Aug 5 00:00:42 localhost os-collect-config: + '[' -n '{"network_config": [{"routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "192.168.0.1"}, {"default": true, "next_hop": "192.168.0.1"}], "use_dhcp": false, "type": "interface", "name": "nic3", "addresses": [{"ip_netmask": "192.168.0.11/24"}]}, {"dns_servers": ["10.16.36.29"], "name": "br-ex", "members": [{"type": "interface", "name": "nic1", "primary": true}], "routes": [{"ip_netmask": " 0.0.0.0/0", "next_hop": "10.19.184.254"}], "use_dhcp": false, "type": "ovs_bridge", "addresses": [{"ip_netmask": "10.19.184.213/24"}]}, {"use_dhcp": false, "type": "ovs_bridge", "name": "br-nic2", "members": [{"type": "interfa ce", "name": "nic2", "primary": true}, {"type": "vlan", "addresses": [{"ip_netmask": "10.19.105.15/24"}], "vlan_id": 185}], "addresses": [{"ip_netmask": "10.19.104.15/24"}]}, {"use_dhcp": false, "type": "ovs_bridge", "name": " br-nic4", "members": [{"type": "interface", "name": "nic4", "primary": true}, {"type": "vlan", "addresses": [{"ip_netmask": "192.168.200.15/24"}], "vlan_id": 203}], "addresses": [{"ip_netmask": "192.168.150.13/24"}]}]}' ']' Aug 5 00:00:42 localhost os-collect-config: + os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] Using config file at: /etc/os-net-config/config.json Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] Ifcfg net config provider created. Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] nic1 mapped to: em1 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] nic2 mapped to: em2 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] nic3 mapped to: p2p1 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] nic4 mapped to: p2p2 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding interface: p2p1 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding custom route for interface: p2p1 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding bridge: br-ex Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding custom route for interface: br-ex Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding interface: em1 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding bridge: br-nic2 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding interface: em2 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding vlan: vlan185 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding bridge: br-nic4 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding interface: p2p2 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] adding vlan: vlan203 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] applying network configs... Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] running ifdown on interface: p2p1 Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p1): device state change: activated -> deactivating (reason 'user-requested') [100 110 39] Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p1): device state change: deactivating -> disconnected (reason 'user-requested') [110 30 39] Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p1): canceled DHCP transaction, DHCP client pid 1168 Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p1): DHCPv4 state changed bound -> done Aug 5 00:00:43 localhost dbus-daemon: dbus[746]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' Aug 5 00:00:43 localhost dbus[746]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] running ifdown on interface: p2p1 Aug 5 00:00:43 localhost systemd: Starting Network Manager Script Dispatcher Service... Aug 5 00:00:43 localhost dbus[746]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Aug 5 00:00:43 localhost systemd: Started Network Manager Script Dispatcher Service. Aug 5 00:00:43 localhost nm-dispatcher: Dispatching action 'down' for p2p1 Aug 5 00:00:43 localhost dbus-daemon: dbus[746]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] running ifdown on interface: vlan203 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] running ifdown on interface: vlan203 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] running ifdown on interface: p2p2 Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p2): device state change: ip-config -> deactivating (reason 'user-requested') [70 110 39] Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p2): device state change: deactivating -> disconnected (reason 'user-requested') [110 30 39] Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p2): canceled DHCP transaction, DHCP client pid 1609 Aug 5 00:00:43 localhost NetworkManager[758]: <info> (p2p2): DHCPv4 state changed unknown -> done Aug 5 00:00:43 localhost nm-dispatcher: Dispatching action 'down' for p2p2 Aug 5 00:00:43 localhost os-collect-config: [2016/08/05 12:00:43 AM] [INFO] running ifdown on interface: p2p2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: vlan185 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: vlan185 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: em2 Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em2): device state change: ip-config -> deactivating (reason 'user-requested') [70 110 39] Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em2): device state change: deactivating -> disconnected (reason 'user-requested') [110 30 39] Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em2): canceled DHCP transaction, DHCP client pid 1605 Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em2): DHCPv4 state changed unknown -> done Aug 5 00:00:44 localhost NetworkManager[758]: <info> Policy set 'System em1' (em1) as default for IPv6 routing and DNS. Aug 5 00:00:44 localhost nm-dispatcher: Dispatching action 'down' for em2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: em2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: em1 Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em1): device state change: activated -> deactivating (reason 'user-requested') [100 110 39] Aug 5 00:00:44 localhost NetworkManager[758]: <info> NetworkManager state is now DISCONNECTING Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em1): device state change: deactivating -> disconnected (reason 'user-requested') [110 30 39] Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em1): canceled DHCP transaction, DHCP client pid 1574 Aug 5 00:00:44 localhost NetworkManager[758]: <info> (em1): DHCPv4 state changed bound -> done Aug 5 00:00:44 localhost NetworkManager[758]: <info> NetworkManager state is now DISCONNECTED Aug 5 00:00:44 localhost nm-dispatcher: Dispatching action 'down' for em1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: em1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: vlan203 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: p2p2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: vlan185 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: em2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on interface: em1 Aug 5 00:00:44 localhost dracut: *** Creating image file done *** Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on bridge: br-nic4 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on bridge: br-nic2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifdown on bridge: br-ex Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-nic2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-nic4 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan185 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-em2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan185 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-em1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-p2p2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-p2p1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-nic4 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-nic2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-em1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-em2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-p2p2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-p2p1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan185 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-p2p1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-p2p2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-em1 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-em2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-nic2 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-nic4 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-vlan203 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan203 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan203 Aug 5 00:00:44 localhost os-collect-config: [2016/08/05 12:00:44 AM] [INFO] running ifup on bridge: br-nic4 Aug 5 00:00:44 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-nic4 -- set bridge br-nic4 other-config:hwaddr=00:0a:f7:79:93:d3 Aug 5 00:00:44 localhost kernel: device ovs-system entered promiscuous mode Aug 5 00:00:44 localhost NetworkManager[758]: <info> (ovs-system): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 6) Aug 5 00:00:45 localhost systemd: Configuration file /usr/lib/systemd/system/dhcp-interface@.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 5 00:00:45 localhost systemd: Starting DHCP interface ovs/system... Aug 5 00:00:45 localhost kernel: device br-nic4 entered promiscuous mode Aug 5 00:00:45 localhost NetworkManager[758]: <info> (br-nic4): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 7) Aug 5 00:00:45 localhost systemd: Configuration file /usr/lib/systemd/system/dhcp-interface@.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 5 00:00:45 localhost dhcp-all-interfaces.sh: cat: /sys/class/net/ovs/system/addr_assign_type: No such file or directory Aug 5 00:00:45 localhost systemd: Starting DHCP interface br/nic4... Aug 5 00:00:45 localhost dhcp-all-interfaces.sh: Inspecting interface: ovs/system...Device has generated MAC, skipping. Aug 5 00:00:45 localhost dhcp-all-interfaces.sh: cat: /sys/class/net/br/nic4/addr_assign_type: No such file or directory Aug 5 00:00:45 localhost dhcp-all-interfaces.sh: Inspecting interface: br/nic4...Device has generated MAC, skipping. Aug 5 00:00:45 localhost ifup: /sbin/ifup: configuration for ovs/system not found. Aug 5 00:00:45 localhost ifup: Usage: ifup <configuration> Aug 5 00:00:45 localhost systemd: dhcp-interface: main process exited, code=exited, status=1/FAILURE Aug 5 00:00:45 localhost systemd: Failed to start DHCP interface ovs/system. Aug 5 00:00:45 localhost systemd: Unit dhcp-interface entered failed state. Aug 5 00:00:45 localhost ifup: /sbin/ifup: configuration for br/nic4 not found. Aug 5 00:00:45 localhost ifup: Usage: ifup <configuration> Aug 5 00:00:45 localhost systemd: dhcp-interface failed. Aug 5 00:00:45 localhost systemd: dhcp-interface: main process exited, code=exited, status=1/FAILURE Aug 5 00:00:45 localhost systemd: Failed to start DHCP interface br/nic4. Aug 5 00:00:45 localhost systemd: Unit dhcp-interface entered failed state. Aug 5 00:00:45 localhost systemd: dhcp-interface failed. Aug 5 00:00:45 localhost NetworkManager[758]: <info> (br-nic4): link connected Aug 5 00:00:47 localhost NetworkManager[758]: <info> keyfile: add connection in-memory (902f1621-7049-4b41-a8a1-3954222b9bb0,"br-nic4") Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: unavailable -> disconnected (reason 'connection-assumed') [20 30 41] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): Activation: starting connection 'br-nic4' (902f1621-7049-4b41-a8a1-3954222b9bb0) Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: prepare -> config (reason 'none') [40 50 0] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: ip-config -> ip-check (reason 'none') [70 80 0] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: ip-check -> secondaries (reason 'none') [80 90 0] Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 5 00:00:47 localhost NetworkManager[758]: <info> NetworkManager state is now CONNECTED_LOCAL Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic4): Activation: successful, device activated. Aug 5 00:00:47 localhost nm-dispatcher: Dispatching action 'up' for br-nic4 Aug 5 00:00:47 localhost systemd: Unit iscsi.service cannot be reloaded because it is inactive. Aug 5 00:00:47 localhost os-collect-config: [2016/08/05 12:00:47 AM] [INFO] running ifup on bridge: br-nic2 Aug 5 00:00:47 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-nic2 -- set bridge br-nic2 other-config:hwaddr=c8:1f:66:c7:dc:e0 Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic2): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 8) Aug 5 00:00:47 localhost kernel: device br-nic2 entered promiscuous mode Aug 5 00:00:47 localhost systemd: Configuration file /usr/lib/systemd/system/dhcp-interface@.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 5 00:00:47 localhost systemd: Starting DHCP interface br/nic2... Aug 5 00:00:47 localhost dhcp-all-interfaces.sh: cat: /sys/class/net/br/nic2/addr_assign_type: No such file or directory Aug 5 00:00:47 localhost dhcp-all-interfaces.sh: Inspecting interface: br/nic2...Device has generated MAC, skipping. Aug 5 00:00:47 localhost ifup: /sbin/ifup: configuration for br/nic2 not found. Aug 5 00:00:47 localhost ifup: Usage: ifup <configuration> Aug 5 00:00:47 localhost systemd: dhcp-interface: main process exited, code=exited, status=1/FAILURE Aug 5 00:00:47 localhost systemd: Failed to start DHCP interface br/nic2. Aug 5 00:00:47 localhost systemd: Unit dhcp-interface entered failed state. Aug 5 00:00:47 localhost systemd: dhcp-interface failed. Aug 5 00:00:47 localhost NetworkManager[758]: <info> (br-nic2): link connected Aug 5 00:00:49 localhost kdumpctl: kexec: loaded kdump kernel Aug 5 00:00:49 localhost kdumpctl: Starting kdump: [OK] Aug 5 00:00:49 localhost systemd: Started Crash recovery kernel arming. Aug 5 00:00:49 localhost systemd: Startup finished in 2.001s (kernel) + 1.587s (initrd) + 50.608s (userspace) = 54.197s. Aug 5 00:00:49 localhost NetworkManager[758]: <info> keyfile: add connection in-memory (98b619dd-22b6-40a9-8d08-1e0df0666da8,"br-nic2") Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: unavailable -> disconnected (reason 'connection-assumed') [20 30 41] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): Activation: starting connection 'br-nic2' (98b619dd-22b6-40a9-8d08-1e0df0666da8) Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: prepare -> config (reason 'none') [40 50 0] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: ip-config -> ip-check (reason 'none') [70 80 0] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: ip-check -> secondaries (reason 'none') [80 90 0] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 5 00:00:49 localhost NetworkManager[758]: <info> (br-nic2): Activation: successful, device activated. Aug 5 00:00:49 localhost nm-dispatcher: Dispatching action 'up' for br-nic2 Aug 5 00:00:49 localhost systemd: Unit iscsi.service cannot be reloaded because it is inactive. Aug 5 00:00:50 localhost os-collect-config: [2016/08/05 12:00:50 AM] [INFO] running ifup on bridge: br-ex Aug 5 00:00:50 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex -- set bridge br-ex other-config:hwaddr=c8:1f:66:c7:dc:df Aug 5 00:00:50 localhost kernel: device br-ex entered promiscuous mode Aug 5 00:00:50 localhost NetworkManager[758]: <info> (br-ex): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 9) Aug 5 00:00:50 localhost systemd: Configuration file /usr/lib/systemd/system/dhcp-interface@.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 5 00:00:50 localhost systemd: Starting DHCP interface br/ex... Aug 5 00:00:50 localhost dhcp-all-interfaces.sh: cat: /sys/class/net/br/ex/addr_assign_type: No such file or directory Aug 5 00:00:50 localhost dhcp-all-interfaces.sh: Inspecting interface: br/ex...Device has generated MAC, skipping. Aug 5 00:00:50 localhost ifup: /sbin/ifup: configuration for br/ex not found. Aug 5 00:00:50 localhost ifup: Usage: ifup <configuration> Aug 5 00:00:50 localhost systemd: dhcp-interface: main process exited, code=exited, status=1/FAILURE Aug 5 00:00:50 localhost systemd: Failed to start DHCP interface br/ex. Aug 5 00:00:50 localhost systemd: Unit dhcp-interface entered failed state. Aug 5 00:00:50 localhost systemd: dhcp-interface failed. Aug 5 00:00:50 localhost NetworkManager[758]: <info> (br-ex): link connected Aug 5 00:00:52 localhost NetworkManager[758]: <info> keyfile: add connection in-memory (8b55236e-b58e-4bfd-b5ba-5482144ea258,"br-ex") Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: unavailable -> disconnected (reason 'connection-assumed') [20 30 41] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): Activation: starting connection 'br-ex' (8b55236e-b58e-4bfd-b5ba-5482144ea258) Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: prepare -> config (reason 'none') [40 50 0] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: ip-config -> ip-check (reason 'none') [70 80 0] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: ip-check -> secondaries (reason 'none') [80 90 0] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 5 00:00:52 localhost NetworkManager[758]: <info> (br-ex): Activation: successful, device activated. Aug 5 00:00:52 localhost nm-dispatcher: Dispatching action 'up' for br-ex Aug 5 00:00:52 localhost systemd: Unit iscsi.service cannot be reloaded because it is inactive. Aug 5 00:00:53 localhost NetworkManager[758]: <info> NetworkManager state is now CONNECTED_GLOBAL Aug 5 00:00:53 localhost NetworkManager[758]: <info> Policy set 'br-ex' (br-ex) as default for IPv4 routing and DNS. Aug 5 00:00:53 localhost NET[11168]: /etc/sysconfig/network-scripts/ifup-post : updated /etc/resolv.conf Aug 5 00:00:53 localhost os-collect-config: [2016/08/05 12:00:53 AM] [INFO] running ifup on interface: p2p1 Aug 5 00:00:54 localhost systemd: Starting Stop Read-Ahead Data Collection... Aug 5 00:00:54 localhost systemd: Started Stop Read-Ahead Data Collection. Aug 5 00:00:55 localhost NetworkManager[758]: <info> ifcfg-rh: add connection in-memory (474556b2-1217-49db-bd17-f6eb58a01d69,"p2p1") Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): Activation: starting connection 'p2p1' (474556b2-1217-49db-bd17-f6eb58a01d69) Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): device state change: prepare -> config (reason 'none') [40 50 0] Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): device state change: ip-config -> ip-check (reason 'none') [70 80 0] Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): device state change: ip-check -> secondaries (reason 'none') [80 90 0] Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 5 00:00:55 localhost NetworkManager[758]: <info> (p2p1): Activation: successful, device activated. Aug 5 00:00:55 localhost nm-dispatcher: Dispatching action 'up' for p2p1 Aug 5 00:00:55 localhost systemd: Unit iscsi.service cannot be reloaded because it is inactive. Aug 5 00:00:56 localhost os-collect-config: [2016/08/05 12:00:56 AM] [INFO] running ifup on interface: p2p1 Aug 5 00:00:57 localhost os-collect-config: [2016/08/05 12:00:57 AM] [INFO] running ifup on interface: vlan203 Aug 5 00:00:57 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-nic4 vlan203 tag=203 -- set Interface vlan203 type=internal Aug 5 00:00:57 localhost kernel: device vlan203 entered promiscuous mode Aug 5 00:00:57 localhost NetworkManager[758]: <info> (vlan203): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 10) Aug 5 00:00:57 localhost systemd: Configuration file /usr/lib/systemd/system/dhcp-interface@.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 5 00:00:57 localhost systemd: Started DHCP interface vlan203. Aug 5 00:00:57 localhost NetworkManager[758]: <info> (vlan203): link connected Aug 5 00:00:59 localhost NetworkManager[758]: <info> keyfile: add connection in-memory (98b2a926-2d03-4e6b-9b5b-3da9efdec380,"vlan203") Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: unavailable -> disconnected (reason 'connection-assumed') [20 30 41] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): Activation: starting connection 'vlan203' (98b2a926-2d03-4e6b-9b5b-3da9efdec380) Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: prepare -> config (reason 'none') [40 50 0] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: ip-config -> ip-check (reason 'none') [70 80 0] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: ip-check -> secondaries (reason 'none') [80 90 0] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 5 00:00:59 localhost NetworkManager[758]: <info> (vlan203): Activation: successful, device activated. Aug 5 00:00:59 localhost nm-dispatcher: Dispatching action 'up' for vlan203 Aug 5 00:00:59 localhost systemd: Unit iscsi.service cannot be reloaded because it is inactive. Aug 5 00:01:00 localhost os-collect-config: [2016/08/05 12:01:00 AM] [INFO] running ifup on interface: vlan203 Aug 5 00:01:00 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-nic4 vlan203 tag=203 -- set Interface vlan203 type=internal Aug 5 00:01:01 localhost os-collect-config: [2016/08/05 12:01:01 AM] [INFO] running ifup on interface: p2p2 Aug 5 00:01:01 localhost systemd: Created slice user-0.slice. Aug 5 00:01:01 localhost systemd: Starting user-0.slice. Aug 5 00:01:01 localhost systemd: Started Session 1 of user root. Aug 5 00:01:01 localhost systemd: Starting Session 1 of user root. Aug 5 00:01:02 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-nic4 p2p2 -- add-port br-nic4 p2p2 Aug 5 00:01:02 localhost kernel: device p2p2 entered promiscuous mode Aug 5 00:01:02 localhost NetworkManager[758]: <info> (p2p2): enslaved to non-master-type device ovs-system; ignoring Aug 5 00:01:02 localhost os-collect-config: [2016/08/05 12:01:02 AM] [INFO] running ifup on interface: p2p2 Aug 5 00:01:03 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-nic4 p2p2 -- add-port br-nic4 p2p2 Aug 5 00:01:03 localhost os-collect-config: [2016/08/05 12:01:03 AM] [INFO] running ifup on interface: vlan185 Aug 5 00:01:03 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-nic2 vlan185 tag=185 -- set Interface vlan185 type=internal Aug 5 00:01:03 localhost kernel: device vlan185 entered promiscuous mode Aug 5 00:01:03 localhost NetworkManager[758]: <info> (vlan185): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 11) Aug 5 00:01:03 localhost systemd: Configuration file /usr/lib/systemd/system/dhcp-interface@.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 5 00:01:03 localhost systemd: Started DHCP interface vlan185. Aug 5 00:01:03 localhost NetworkManager[758]: <info> (vlan185): link connected Aug 5 00:01:05 localhost NetworkManager[758]: <info> keyfile: add connection in-memory (cc5296a2-e640-40c3-9c22-43a50598799f,"vlan185") Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: unavailable -> disconnected (reason 'connection-assumed') [20 30 41] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): Activation: starting connection 'vlan185' (cc5296a2-e640-40c3-9c22-43a50598799f) Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: prepare -> config (reason 'none') [40 50 0] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: ip-config -> ip-check (reason 'none') [70 80 0] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: ip-check -> secondaries (reason 'none') [80 90 0] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 5 00:01:05 localhost NetworkManager[758]: <info> (vlan185): Activation: successful, device activated. Aug 5 00:01:05 localhost nm-dispatcher: Dispatching action 'up' for vlan185 Aug 5 00:01:05 localhost systemd: Unit iscsi.service cannot be reloaded because it is inactive. Aug 5 00:01:06 localhost os-collect-config: [2016/08/05 12:01:06 AM] [INFO] running ifup on interface: vlan185 Aug 5 00:01:06 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-nic2 vlan185 tag=185 -- set Interface vlan185 type=internal Aug 5 00:01:07 localhost os-collect-config: [2016/08/05 12:01:07 AM] [INFO] running ifup on interface: em2 Aug 5 00:01:08 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-nic2 em2 -- add-port br-nic2 em2 Aug 5 00:01:08 localhost kernel: device em2 entered promiscuous mode Aug 5 00:01:08 localhost NetworkManager[758]: <info> (em2): enslaved to non-master-type device ovs-system; ignoring Aug 5 00:01:08 localhost os-collect-config: [2016/08/05 12:01:08 AM] [INFO] running ifup on interface: em2 Aug 5 00:01:09 localhost NetworkManager[758]: <info> Policy set 'vlan185' (vlan185) as default for IPv6 routing and DNS. Aug 5 00:01:09 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-nic2 em2 -- add-port br-nic2 em2 Aug 5 00:01:09 localhost os-collect-config: [2016/08/05 12:01:09 AM] [INFO] running ifup on interface: em1 Aug 5 00:01:10 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex em1 -- add-port br-ex em1 Aug 5 00:01:10 localhost kernel: device em1 entered promiscuous mode Aug 5 00:01:10 localhost NetworkManager[758]: <info> (em1): enslaved to non-master-type device ovs-system; ignoring Aug 5 00:01:11 localhost os-collect-config: [2016/08/05 12:01:11 AM] [INFO] running ifup on interface: em1 Aug 5 00:01:12 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex em1 -- add-port br-ex em1 Aug 5 00:01:12 localhost os-collect-config: [2016/08/05 12:01:12 AM] [INFO] running ifup on interface: vlan203 Aug 5 00:01:12 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-nic4 vlan203 tag=203 -- set Interface vlan203 type=internal Aug 5 00:01:13 localhost os-collect-config: [2016/08/05 12:01:13 AM] [INFO] running ifup on interface: p2p2 Aug 5 00:01:14 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-nic4 p2p2 -- add-port br-nic4 p2p2 Aug 5 00:01:14 localhost os-collect-config: [2016/08/05 12:01:14 AM] [INFO] running ifup on interface: vlan185 Aug 5 00:01:14 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-nic2 vlan185 tag=185 -- set Interface vlan185 type=internal Aug 5 00:01:15 localhost os-collect-config: [2016/08/05 12:01:15 AM] [INFO] running ifup on interface: em2 Aug 5 00:01:16 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-nic2 em2 -- add-port br-nic2 em2 Aug 5 00:01:16 localhost os-collect-config: [2016/08/05 12:01:16 AM] [INFO] running ifup on interface: em1 Aug 5 00:01:17 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex em1 -- add-port br-ex em1 Aug 5 00:01:17 localhost os-collect-config: + RETVAL=2 Aug 5 00:01:17 localhost os-collect-config: + [[ 2 == 2 ]]
I've changed the title : rhel-osp-director: upgrade from 7.3 ->8.0 , br-ex on the controllers was missing ip address and pacemaker was unable to bring up the vip for that network. The actual failure occurred on 7.3 -> 8.0 overcloud upgrade. We need to check if it's not something new that got introduced by using this puddle : http://download.lab.bos.redhat.com/rel-eng/OpenStack/8.0-RHEL-7-director/2016-08-05.3/
Basil, Can you review if this is indeed a RC blocker, or can it wait for a fix for GA (0day) or a later maintenance release. Thanks, Scott
workaround (worked for me): Before you begin with the overcloud upgrade: 1. On all overcloud nodes run: systemctl stop NetworkManager systemctl mask NetworkManager 2. On all overcloud nodes run: service network restart 3. On one controller run: pcs cluster stop --all wait until the cluster stops. 4. On one controller run: pcs cluster start --all wait until all the resources are running on all controllers.
Note: running "systemctl NetworkManager stop" even on interface(s) with NM_CONTROLLED=no may bring it down due to the issue[1] with NM. 1.https://bugzilla.redhat.com/show_bug.cgi?id=1366348
Note: after applying the workaround from Comment #12 -> the following upgrade from 8.0 to 9.0 failed over (disabled NetworkManager) : https://bugzilla.redhat.com/show_bug.cgi?id=1366392
+1, agree with handling under OSP8
As there is a workaround for the upgrade to 8, this no longer is an issue with other upgrades, and we do not have any fixes planned (coming up on 2 years now) I would like to close this bug. Please respond with any objections.