Description of problem: (overcloud) [stack@undercloud-0 ~]$ source overcloudrc (overcloud) [stack@undercloud-0 ~]$ nova list No handlers could be found for logger "keystoneauth.identity.generic.base" ERROR (ConnectFailure): Unable to establish connection to http://10.0.0.104:5000/v2.0/tokens: HTTPConnectionPool(host='10.0.0.104', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x301ab90>: Failed to establish a new connection: [Errno 113] No route to host',)) [heat-admin@overcloud-controller-0 ~]$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 080848b55fd5 192.168.24.1:8787/rhosp12/openstack-ceilometer-central-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago ceilometer_gnocchi_upgrade 8ab5d31dc9f1 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago nova_api_discover_hosts b2b239eefa4b 192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes gnocchi_api 2b4b8083c8b9 192.168.24.1:8787/rhosp12/openstack-panko-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes panko_api f1ef9f63c065 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) nova_metadata 48dadc7c7716 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) nova_api 258d99a7fc5e 192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) glance_api 6dbb46bc2592 192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_account_server d37cb914e99a 192.168.24.1:8787/rhosp12/openstack-aodh-listener-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) aodh_listener 6bd9813d8903 192.168.24.1:8787/rhosp12/openstack-swift-container-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_container_auditor 0aa069d63444 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) heat_api_cron 41dee94b92c0 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_object_expirer ae475746bc65 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_object_updater 466c6c3eaa15 192.168.24.1:8787/rhosp12/openstack-swift-container-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_container_replicator ae6031e3404a 192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_account_auditor df4c4cf11326 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes logrotate_crond 1ff310206f90 192.168.24.1:8787/rhosp12/openstack-heat-api-cfn-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) heat_api_cfn db4c6d7a69fa 192.168.24.1:8787/rhosp12/openstack-nova-scheduler-docker:2017-09-22.5 "kolla_start" About an hour ago Up About a minute (unhealthy) nova_scheduler d4c4888bb2ef 192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes gnocchi_metricd c10285b34c4e 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_object_replicator bb8d35b69ae3 192.168.24.1:8787/rhosp12/openstack-swift-container-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_container_server 43456bb93b40 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) heat_engine 4090c76370ca 192.168.24.1:8787/rhosp12/openstack-aodh-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes aodh_api 57554d295f55 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_rsync b22f73b718fc 192.168.24.1:8787/rhosp12/openstack-nova-novncproxy-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) nova_vnc_proxy 3e22c56eadfc 192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago gnocchi_db_sync 59efb8fa78b3 192.168.24.1:8787/rhosp12/openstack-ceilometer-notification-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) ceilometer_agent_notification 68c3c0aa57e2 192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_account_reaper 5490dcfabe50 192.168.24.1:8787/rhosp12/openstack-nova-consoleauth-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) nova_consoleauth 4b3e18eb0688 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) nova_api_cron 710277f0394f 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-09-22.5 "kolla_start" About an hour ago Restarting (1) 2 minutes ago gnocchi_statsd 1d3017200b3f 192.168.24.1:8787/rhosp12/openstack-aodh-notifier-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) aodh_notifier 2edc8209261c 192.168.24.1:8787/rhosp12/openstack-ceilometer-central-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) ceilometer_agent_central f17735b9d634 192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_account_replicator a8a694023d19 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_object_auditor 0d5d4b9be9d1 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) heat_api 3ae69141d9c6 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_proxy ce39853ce2fe 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_object_server 73c24388bada 192.168.24.1:8787/rhosp12/openstack-nova-conductor-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) nova_conductor 85ebb454ef3d 192.168.24.1:8787/rhosp12/openstack-swift-container-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) swift_container_updater 13cb1c1f97be 192.168.24.1:8787/rhosp12/openstack-aodh-evaluator-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (unhealthy) aodh_evaluator 5877f77ac544 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-09-22.5 "/bin/bash -c '/usr/l" About an hour ago Up 58 minutes (healthy) keystone_cron 3f49bd88c7d7 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago nova_db_sync 9a79a704b44c 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes (healthy) keystone 8c2e4e51f999 192.168.24.1:8787/rhosp12/openstack-iscsid-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes iscsid fcc739f95282 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago nova_api_create_default_cell e4f90ad976d7 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago nova_api_map_cell0 e11059bf4cd9 192.168.24.1:8787/rhosp12/openstack-nova-placement-api-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes nova_placement dce7fca7edd9 192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-09-22.5 "chown -R swift: /srv" About an hour ago Exited (0) About an hour ago swift_setup_srv f6108d2adccf 192.168.24.1:8787/rhosp12/openstack-horizon-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes horizon 6404b747f7a4 192.168.24.1:8787/rhosp12/openstack-ceilometer-notification-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago ceilometer_init_log 82d6cfd5eac6 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago keystone_db_sync 43206056fbdb 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago nova_api_db_sync 9e1f3650a6dd 192.168.24.1:8787/rhosp12/openstack-panko-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago panko_db_sync 343fb64ca4b5 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-09-22.5 "/bin/bash -c 'cp -v " About an hour ago Exited (0) About an hour ago swift_copy_rings 10642b1cf056 192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago glance_api_db_sync d0fb252faaa3 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago heat_engine_db_sync 1b362c5756a9 192.168.24.1:8787/rhosp12/openstack-aodh-api-docker:2017-09-22.5 "/usr/bin/bootstrap_h" About an hour ago Exited (0) About an hour ago aodh_db_sync de9cdce6d1ad 192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 58 minutes ago haproxy-bundle-docker-0 0717d8505b81 192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 58 minutes ago redis-bundle-docker-0 5615629b5140 192.168.24.1:8787/rhosp12/openstack-haproxy-docker:2017-09-22.5 "/bin/bash -c 'cp -a " About an hour ago Exited (0) About an hour ago haproxy_init_bundle 1f42b88dbf80 192.168.24.1:8787/rhosp12/openstack-redis-docker:2017-09-22.5 "/bin/bash -c 'cp -a " About an hour ago Exited (0) About an hour ago redis_init_bundle e34ee525991a 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 58 minutes ago galera-bundle-docker-0 2d93bc713f10 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 58 minutes ago rabbitmq-bundle-docker-0 072ec43b0626 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-22.5 "/bin/bash -c 'cp -a " About an hour ago Exited (0) About an hour ago mysql_init_bundle a22652aad369 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-22.5 "kolla_start" About an hour ago Up 58 minutes clustercheck df5cc7b4d331 192.168.24.1:8787/rhosp12/openstack-aodh-api-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago aodh_init_log 3c2630455871 192.168.24.1:8787/rhosp12/openstack-panko-api-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago panko_init_log bb6cb6d106bc 192.168.24.1:8787/rhosp12/openstack-horizon-docker:2017-09-22.5 "/bin/bash -c 'touch " About an hour ago Exited (0) About an hour ago horizon_fix_perms 885596feff02 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago keystone_init_log 4f76c8971d11 192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago glance_init_logs f28820211151 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago heat_init_log 626c3d370af2 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago nova_init_logs 51b6793f32ca 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:2017-09-22.5 "/bin/bash -c 'cp -a " About an hour ago Exited (0) About an hour ago rabbitmq_init_bundle bb25c9fc1ee5 192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-09-22.5 "/bin/bash -c 'chown " About an hour ago Exited (0) About an hour ago gnocchi_init_log dc4a51d6dd7c 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-22.5 "/bin/bash -c '/usr/b" About an hour ago Exited (0) About an hour ago mysql_image_tag 9c621a774a0d 192.168.24.1:8787/rhosp12/openstack-memcached-docker:2017-09-22.5 "/bin/bash -c 'source" About an hour ago Up 58 minutes memcached bb1f159c61e3 192.168.24.1:8787/rhosp12/openstack-haproxy-docker:2017-09-22.5 "/bin/bash -c '/usr/b" About an hour ago Exited (0) About an hour ago haproxy_image_tag bce6fa3ea4a4 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-22.5 "bash -ecx 'if [ -e /" About an hour ago Exited (0) About an hour ago mysql_bootstrap 7a8ebfe8d77b 192.168.24.1:8787/rhosp12/openstack-redis-docker:2017-09-22.5 "/bin/bash -c '/usr/b" About an hour ago Exited (0) About an hour ago redis_image_tag 6951da07adf3 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:2017-09-22.5 "/bin/bash -c '/usr/b" About an hour ago Exited (0) About an hour ago rabbitmq_image_tag ccd7dec8616f 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:2017-09-22.5 "kolla_start" About an hour ago Exited (0) About an hour ago rabbitmq_bootstrap 53d833309232 192.168.24.1:8787/rhosp12/openstack-memcached-docker:2017-09-22.5 "/bin/bash -c 'source" About an hour ago Exited (0) About an hour ago memcached_init_logs 458111503cb6 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-22.5 "chown -R mysql: /var" About an hour ago Exited (0) About an hour ago mysql_data_ownership [heat-admin@overcloud-controller-0 ~]$ sudo docker ps -a |grep "Exited (1)" de9cdce6d1ad 192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 53 minutes ago haproxy-bundle-docker-0 0717d8505b81 192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 53 minutes ago redis-bundle-docker-0 e34ee525991a 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 53 minutes ago 2d93bc713f10 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 53 minutes ago rabbitmq-bundle-docker-0 710277f0394f 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-09-22.5 "kolla_start" About an hour ago Restarting (1) 24 minutes ago gnocchi_statsd [heat-admin@overcloud-controller-1 ~]$ sudo docker ps -a |grep "(1)" a0d589092751 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-09-22.5 "kolla_start" About an hour ago Restarting (1) 3 minutes ago gnocchi_statsd baba1e156202 192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 59 minutes ago redis-bundle-docker-1 3e6224af6d46 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) 59 minutes ago rabbitmq-bundle-docker-1 [heat-admin@overcloud-controller-2 ~]$ sudo docker ps -a |grep "(1)" 3e0fc8a87084 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-09-22.5 "kolla_start" About an hour ago Restarting (1) 4 minutes ago gnocchi_statsd ed2ed1320627 192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) About an hour ago haproxy-bundle-docker-2 59abedb60dee 192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) About an hour ago redis-bundle-docker-2 4668dfc0a4d7 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) About an hour ago galera-bundle-docker-2 6c8031d4c026 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest "/bin/bash /usr/local" About an hour ago Exited (1) About an hour ago rabbitmq-bundle-docker-2 [heat-admin@overcloud-controller-0 ~]$ sudo pcs status Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.2-94ff4df) - partition WITHOUT quorum Last updated: Mon Sep 25 14:39:26 2017 Last change: Mon Sep 25 13:24:51 2017 by root via cibadmin on overcloud-controller-0 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 ] OFFLINE: [ overcloud-controller-1 overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped galera-bundle-1 (ocf::heartbeat:galera): Stopped galera-bundle-2 (ocf::heartbeat:galera): Stopped Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Stopped redis-bundle-1 (ocf::heartbeat:redis): Stopped redis-bundle-2 (ocf::heartbeat:redis): Stopped ip-192.168.24.12 (ocf::heartbeat:IPaddr2): Stopped ip-10.0.0.104 (ocf::heartbeat:IPaddr2): Stopped ip-172.17.1.13 (ocf::heartbeat:IPaddr2): Stopped ip-172.17.1.18 (ocf::heartbeat:IPaddr2): Stopped ip-172.17.3.12 (ocf::heartbeat:IPaddr2): Stopped ip-172.17.4.10 (ocf::heartbeat:IPaddr2): Stopped Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Stopped haproxy-bundle-docker-1 (ocf::heartbeat:docker): Stopped haproxy-bundle-docker-2 (ocf::heartbeat:docker): Stopped openstack-cinder-volume (systemd:openstack-cinder-volume): Stopped Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Logs: gnocchi_statsd http://pastebin.test.redhat.com/518984 haproxy-bundle-docker-0 http://pastebin.test.redhat.com/518985 galera: http://pastebin.test.redhat.com/518988 rabbitmq http://pastebin.test.redhat.com/518990 redis-logs:http://pastebin.test.redhat.com/518995 Version-Release number of selected component (if applicable): OSP12 docker images - 2017-09-22.5 How reproducible: openstack-swift-object-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-neutron-ml2-11.0.1-0.20170919192755.82db373.el7ost.noarch openstack-heat-api-cfn-9.0.1-0.20170913170444.0c64134.el7ost.noarch openstack-tripleo-image-elements-7.0.0-0.20170914203419.526772d.el7ost.noarch openstack-zaqar-5.0.1-0.20170905222047.el7ost.noarch openstack-nova-common-16.0.1-0.20170919190458.9ad426a.el7ost.noarch openstack-neutron-11.0.1-0.20170919192755.82db373.el7ost.noarch openstack-ironic-conductor-9.1.1-0.20170908114346.feb64c2.el7ost.noarch openstack-tempest-17.0.0-0.20170901201711.ad75393.el7ost.noarch openstack-tripleo-heat-templates-7.0.1-0.20170919183703.el7ost.noarch openstack-mistral-executor-5.1.1-0.20170913191927.9e3b36b.el7ost.noarch python-openstackclient-lang-3.12.0-0.20170821150739.f67ebce.el7ost.noarch openstack-swift-account-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-neutron-common-11.0.1-0.20170919192755.82db373.el7ost.noarch openstack-ironic-inspector-6.0.1-0.20170824132804.0e72dcb.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch openstack-tripleo-common-7.6.1-0.20170913110336.el7ost.noarch openstack-mistral-common-5.1.1-0.20170913191927.9e3b36b.el7ost.noarch openstack-nova-api-16.0.1-0.20170919190458.9ad426a.el7ost.noarch openstack-nova-conductor-16.0.1-0.20170919190458.9ad426a.el7ost.noarch openstack-glance-15.0.0-0.20170830130905.9820166.el7ost.noarch openstack-keystone-12.0.1-0.20170907172639.6a67918.el7ost.noarch openstack-nova-compute-16.0.1-0.20170919190458.9ad426a.el7ost.noarch openstack-heat-api-9.0.1-0.20170913170444.0c64134.el7ost.noarch openstack-ironic-common-9.1.1-0.20170908114346.feb64c2.el7ost.noarch openstack-mistral-api-5.1.1-0.20170913191927.9e3b36b.el7ost.noarch openstack-nova-scheduler-16.0.1-0.20170919190458.9ad426a.el7ost.noarch openstack-swift-proxy-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-ironic-api-9.1.1-0.20170908114346.feb64c2.el7ost.noarch puppet-openstack_extras-11.3.1-0.20170906070209.b99c3a4.el7ost.noarch openstack-mistral-engine-5.1.1-0.20170913191927.9e3b36b.el7ost.noarch openstack-tripleo-common-containers-7.6.1-0.20170913110336.el7ost.noarch openstack-swift-container-2.15.2-0.20170913222602.34601cc.el7ost.noarch openstack-neutron-openvswitch-11.0.1-0.20170919192755.82db373.el7ost.noarch openstack-heat-engine-9.0.1-0.20170913170444.0c64134.el7ost.noarch openstack-tripleo-validations-7.4.0-0.20170918153806.efe8a72.el7ost.noarch openstack-selinux-0.8.10-0.20170914195211.e16a8f8.2.el7ost.noarch openstack-nova-placement-api-16.0.1-0.20170919190458.9ad426a.el7ost.noarch openstack-tripleo-puppet-elements-7.0.0-0.20170914203705.2094778.el7ost.noarch python-openstacksdk-0.9.17-0.20170821143340.7946243.el7ost.noarch python-openstackclient-3.12.0-0.20170821150739.f67ebce.el7ost.noarch openstack-heat-common-9.0.1-0.20170913170444.0c64134.el7ost.noarch puppet-openstacklib-11.3.1-0.20170825142820.18ee919.el7ost.noarch openstack-tripleo-ui-7.4.1-0.20170915125211.1031ea6.el7ost.noarch Steps to Reproduce: 1.Deploy osp12 2.restart oc nodes 3.wait 5-10 minutes Actual results: overcloud ha services is down Expected results: overcloud in operable state Additional info:
So the HA containers are down because the network is busted. You can tell that from the following initial lines in 'pcs status': Online: [ overcloud-controller-0 ] OFFLINE: [ overcloud-controller-1 overcloud-controller-2 ] So node 0 cannot talk to node 1 and 2. [root@overcloud-controller-0 ~]# ping overcloud-controller-1 PING overcloud-controller-1.localdomain (172.17.1.25) 56(84) bytes of data. ^C --- overcloud-controller-1.localdomain ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1000ms [root@overcloud-controller-0 ~]# ping overcloud-controller-2 PING overcloud-controller-2.localdomain (172.17.1.20) 56(84) bytes of data. ^C --- overcloud-controller-2.localdomain ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 999ms Initially I thought this was a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1473763 But I see that you have: [root@overcloud-controller-0 ~]# rpm -q os-net-config os-net-config-7.3.0-0.20170910153345.77fe592.el7ost.noarch And the above BZ (1473763) has fixed-in: os-net-config-7.2.1-0.20170825174722.77fe592.el7ost And the patch from the BZ is in the package (I just checked). So not sure about this one. Maybe it is related to the '[rhos-dev] Overcloud Nodes Network Layout' thread on rhos-dev ?
I don't exactly know where to start looking on the test setup, just some things I noticed: 1. overcloud-controller-0 had a full set of iptables rules, overcloud-controller-1 has none 2. overcloud-controller-1 is flooding the vlan20 network with ARP frames, don't know why that is yet I'll keep looking tomorrow, but if I can get more information on how things are configured here it might help.
Artem, this is likely a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1490281
The configuration does look just like the other bug looking at ovs-vsctl output, as well as the ARP storm on vlan20.
Closing as duplicate as the reproducer steps and symptoms are the same. *** This bug has been marked as a duplicate of bug 1490281 ***