[OSP12] when using undercloud nova to stop controller nodes, pacemaker resource stop operation times out ('overcloudrc' won't function after clean reboot) Environment (OSP12 RC Puddle) : ------------------------------- pacemaker-1.1.16-12.el7_4.4.x86_64 pacemaker-cli-1.1.16-12.el7_4.4.x86_64 ansible-pacemaker-1.0.3-2.el7ost.noarch puppet-pacemaker-0.6.1-0.20171103172230.9a46ecd.el7ost.noarch pacemaker-remote-1.1.16-12.el7_4.4.x86_64 pacemaker-libs-1.1.16-12.el7_4.4.x86_64 pacemaker-cluster-libs-1.1.16-12.el7_4.4.x86_64 openstack-tripleo-heat-templates-7.0.3-8.el7ost.noarch puppet-pacemaker-0.6.1-0.20171103172230.9a46ecd.el7ost.noarch openstack-nova-common-16.0.2-2.el7ost.noarch openstack-mistral-common-5.1.1-0.20171104083011.5372770.el7ost.noarch python-pycadf-common-2.6.0-1.el7ost.noarch openstack-neutron-common-11.0.1-4.el7ost.noarch openstack-ironic-common-9.1.3-0.20171106091155.fceb40f.el7ost.noarch libX11-common-1.6.5-1.el7.noarch docker-common-1.12.6-61.git85d7426.el7.x86_64 rhevm-guest-agent-common-1.0.11-3.el7ev.noarch dhcp-common-4.2.5-58.el7.x86_64 openstack-tripleo-common-containers-7.6.3-3.el7ost.noarch glibc-common-2.17-196.el7.x86_64 python-ryu-common-4.15-1.el7ost.noarch openstack-heat-common-9.0.1-3.el7ost.noarch grub2-common-2.02-0.65.el7_4.2.noarch openstack-tripleo-common-7.6.3-3.el7ost.noarch qemu-kvm-common-rhev-2.9.0-16.el7_4.11.x86_64 Steps to reproduce : --------------------- (1) Run 'Clean reboot' from the undercloud --> [root@undercloud74 tripleo]# vi reboot_oc_nodes.sh #!/bin/bash source /home/stack/stackrc echo "turning all nodes off" for i in `nova list|awk -F"|" '/Running/ {print $2}'`; do echo $i; nova stop $i; done sleep 60 echo "starting all but ceph nodes" for i in `nova list|awk -F"|" '!/ceph/ && /SHUTOFF/ {print $2}'`; do echo $i; nova start $i; done sleep 60 echo "starting ceph nodes" for i in `nova list|awk -F"|" '/ceph/ && /SHUTOFF/ {print $2}'`; do echo $i; nova start $i; done Results : --------- (A) pacemaker resource stop operation times out (B) Overcloudrc won't function: (overcloud) [stack@undercloud74 ~]$ openstack server list Failed to discover available identity versions when contacting https://10.19.184.160:13000/v2.0. Attempting to parse version from URL. Unable to establish connection to https://10.19.184.160:13000/v2.0/tokens: HTTPSConnectionPool(host='10.19.184.160', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x34f7f50>: Failed to establish a new connection: [Errno 113] No route to host',)) Logs: ------ [root@undercloud74 ~]# journalctl | grep -e 'Stopping Docker App' Nov 16 21:40:10 undercloud74.local systemd[1]: Stopping Docker Application Container Engine... [root@undercloud74 ~]# # when did you restart ? [root@undercloud74 ~]# journalctl -- Logs begin at Tue 2017-10-10 12:45:58 EDT, end at Fri 2017-11-17 09:20:53 EST. -- Oct 10 12:45:58 localhost systemd-journal[103]: Runtime journal is using 8.0M (max allowed 995.8M, trying to leave 1.4G free of 9.7G available → current limit 995.8M). Oct 10 12:45:58 localhost kernel: Initializing cgroup subsys cpuset Oct 10 12:45:58 localhost kernel: Initializing cgroup subsys cpu Oct 10 12:45:58 localhost kernel: Initializing cgroup subsys cpuacct Oct 10 12:45:58 localhost kernel: Linux version 3.10.0-693.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SM Oct 10 12:45:58 localhost kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115 Oct 10 12:45:58 localhost kernel: e820: BIOS-provided physical RAM map: Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bfff6fff] usable Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000bfff7000-0x00000000bfffffff] reserved Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x0000000100000000-0x000000053fffffff] usable Oct 10 12:45:58 localhost kernel: NX (Execute Disable) protection: active Oct 10 12:45:58 localhost kernel: SMBIOS 2.4 present. Oct 10 12:45:58 localhost kernel: DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011 Oct 10 12:45:58 localhost kernel: Hypervisor detected: KVM Oct 10 12:45:58 localhost kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Oct 10 12:45:58 localhost kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Oct 10 12:45:58 localhost kernel: e820: last_pfn = 0x540000 max_arch_pfn = 0x400000000 Oct 10 12:45:58 localhost kernel: MTRR default type: write-back heat-admin@overcloud-controller-0 ~]$ sudo su [root@overcloud-controller-0 heat-admin]# pcs status Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum Last updated: Fri Nov 17 13:57:10 2017 Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Stopped overcloud-controller-0 galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-controller-0 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Stopped overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Stopped overcloud-controller-0 redis-bundle-2 (ocf::heartbeat:redis): Stopped overcloud-controller-0 ip-192.168.0.12 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.184.160 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.104.14 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.104.19 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.105.11 (ocf::heartbeat:IPaddr2): Stopped ip-192.168.200.15 (ocf::heartbeat:IPaddr2): Stopped Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ] haproxy-bundle-docker-1 (ocf::heartbeat:docker): FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ] haproxy-bundle-docker-2 (ocf::heartbeat:docker): FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Failed Actions: * rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=93, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20023ms * rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=94, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20037ms * galera-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=96, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20035ms * rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=95, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20036ms * galera-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=97, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20034ms * galera-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=98, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20039ms * redis-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=99, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20032ms * redis-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=100, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20033ms * redis-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=101, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20030ms * haproxy-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=102, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20027ms * haproxy-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=103, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms * haproxy-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=104, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms * rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=91, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms * rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=92, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms * rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=93, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20006ms * galera-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=94, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * galera-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=95, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms * galera-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=96, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms * redis-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=97, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * redis-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=98, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * redis-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=99, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * haproxy-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=100, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * haproxy-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=101, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * haproxy-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=102, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=91, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms * rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=92, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms * rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=93, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * galera-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=94, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20021ms * galera-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=95, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms * galera-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=96, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms * redis-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=97, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20017ms * redis-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=98, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20016ms * redis-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=99, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms * haproxy-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=100, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms * haproxy-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=101, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms * haproxy-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=102, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@overcloud-controller-0 heat-admin]# pcs status | less Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum Last updated: Fri Nov 17 13:57:22 2017 Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Stopped overcloud-controller-0 galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-controller-0 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Stopped overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Stopped overcloud-controller-0 redis-bundle-2 (ocf::heartbeat:redis): Stopped overcloud-controller-0 ip-192.168.0.12 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.184.160 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.104.14 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.104.19 (ocf::heartbeat:IPaddr2): Stopped ip-10.19.105.11 (ocf::heartbeat:IPaddr2): Stopped ip-192.168.200.15 (ocf::heartbeat:IPaddr2): Stopped Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ] haproxy-bundle-docker-1 (ocf::heartbeat:docker): FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ] haproxy-bundle-docker-2 (ocf::heartbeat:docker): FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Failed Actions: * rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=93, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20023ms * rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=94, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20037ms * galera-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=96, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20035ms * rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=95, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20036ms * galera-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=97, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20034ms * galera-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=98, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20039ms * redis-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=99, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20032ms * redis-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=100, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20033ms * redis-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=101, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20030ms * haproxy-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=102, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20027ms * haproxy-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=103, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms * haproxy-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=104, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms * rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=91, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms * rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=92, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms * rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=93, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20006ms * galera-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=94, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * galera-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=95, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms * galera-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=96, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms * redis-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=97, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * redis-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=98, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * redis-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=99, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * haproxy-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=100, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * haproxy-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=101, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms * haproxy-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=102, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=91, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms * rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=92, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms * rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=93, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms * galera-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=94, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20021ms * galera-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=95, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms * galera-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=96, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms * redis-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=97, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20017ms * redis-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=98, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20016ms * redis-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=99, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms * haproxy-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=100, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms * haproxy-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=101, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms * haproxy-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=102, status=Timed Out, exitreason='none', last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@overcloud-controller-0 heat-admin]# pcs resource cleanup Error: Cleaning up all resources on all nodes will execute more than 100 operations in the cluster, which may negatively impact the responsiveness of the cluster. Consider specifying resource and/or node, use --force to override [root@overcloud-controller-0 heat-admin]# pcs resource cleanup --force Waiting for 1 replies from the CRMd. OK [root@overcloud-controller-0 heat-admin]# pcs status Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:36 UTC): haproxy[16629]: proxy mysql has no server available! Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:36 UTC): haproxy[16629]: proxy horizon has no server available! Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum Last updated: Fri Nov 17 13:58:37 2017 Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-2 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Stopped overcloud-controller-1 galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-controller-2 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Stopped overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Stopped overcloud-controller-1 redis-bundle-2 (ocf::heartbeat:redis): Stopped overcloud-controller-2 ip-192.168.0.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-10.19.184.160 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-10.19.104.14 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-10.19.104.19 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-10.19.105.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.200.15 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started overcloud-controller-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started overcloud-controller-2 openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Daemon Status: corosync: active/enabled Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:37 UTC): haproxy[16629]: proxy neutron has no server available! pacemaker: active/enabled pcsd: active/enabled [root@overcloud-controller-0 heat-admin]# Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:37 UTC): haproxy[16629]: proxy redis has no server available! Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:39 UTC): haproxy[16629]: proxy gnocchi has no server available! [root@overcloud-controller-0 heat-admin]# Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:46 UTC): haproxy[16629]: proxy cinder has no server available! pcs status Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:47 UTC): haproxy[16629]: proxy nova_metadata has no server available! Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:47 UTC): haproxy[16629]: proxy nova_osapi has no server available! Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum Last updated: Fri Nov 17 13:58:47 2017 Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-2 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Slave overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Stopped overcloud-controller-1 galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-controller-2 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Slave overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Stopped overcloud-controller-1 redis-bundle-2 (ocf::heartbeat:redis): Stopped overcloud-controller-2 ip-192.168.0.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-10.19.184.160 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-10.19.104.14 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-10.19.104.19 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-10.19.105.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.200.15 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started overcloud-controller-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started overcloud-controller-2 openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@overcloud-controller-0 heat-admin]# exit exit [heat-admin@overcloud-controller-0 ~]$ exit logout Connection to 192.168.0.17 closed. (undercloud) [stack@undercloud74 ~]$ ping 10.19.184.160 PING 10.19.184.160 (10.19.184.160) 56(84) bytes of data. 64 bytes from 10.19.184.160: icmp_seq=1 ttl=64 time=0.516 ms 64 bytes from 10.19.184.160: icmp_seq=2 ttl=64 time=0.387 ms ^C --- 10.19.184.160 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.387/0.451/0.516/0.067 ms (undercloud) [stack@undercloud74 ~]$ source overcloud -bash: overcloud: No such file or directory (undercloud) [stack@undercloud74 ~]$ source overcloudrc (overcloud) [stack@undercloud74 ~]$ openstack server list ^CTraceback (most recent call last): File "/bin/openstack", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 213, in main return OpenStackShell().run(argv) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 393, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 200, in prepare_to_run_command return super(OpenStackShell, self).prepare_to_run_command(cmd) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 437, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 99, in auth_ref return super(ClientManager, self).auth_ref File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 239, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref authenticated=False, log=False) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 853, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request resp = super(TimingSession, self).request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner return wrapped(*args, **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 703, in request resp = send(**kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 761, in _send_request resp = self.session.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 423, in send timeout=timeout File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen chunked=chunked) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 386, in _make_request httplib_response = conn.getresponse(buffering=True) File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse response.begin() File "/usr/lib64/python2.7/httplib.py", line 444, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib64/python2.7/ssl.py", line 759, in recv return self.read(buflen) File "/usr/lib64/python2.7/ssl.py", line 653, in read v = self._sslobj.read(len or 1024) KeyboardInterrupt (overcloud) [stack@undercloud74 ~]$ ^C (overcloud) [stack@undercloud74 ~]$ ^C (overcloud) [stack@undercloud74 ~]$ ssh heat-admin.0.17 Last login: Fri Nov 17 13:57:03 2017 from 192.168.0.1 [heat-admin@overcloud-controller-0 ~]$ journalctl -t haproxy -- Logs begin at Fri 2017-11-17 03:29:13 UTC, end at Fri 2017-11-17 13:59:59 UTC. -- Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy aodh started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy cinder started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy glance_api started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy gnocchi started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy haproxy.stats started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy heat_api started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy heat_cfn started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy horizon started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy keystone_admin started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy keystone_public started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy mysql started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy neutron started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_metadata started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_novncproxy started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_osapi started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_placement started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy panko started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy redis started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy swift_proxy_server started. Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: Server aodh/overcloud-controller-0.internalapi.localdomain is DOWN, reason: Layer4 connection problem, info: "Connection Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: Server aodh/overcloud-controller-1.internalapi.localdomain is DOWN, reason: Layer4 connection problem, info: "Connection Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: Server aodh/overcloud-controller-2.internalapi.localdomain is DOWN, reason: Layer4 connection problem, info: "Connection Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: proxy aodh has no server available! ...skipping... Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:49909 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:51667 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:40795 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:41864 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:42039 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:40362 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:60199 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:40534 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Server nova_metadata/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:32825 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:52839 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:53001 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:41682 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:40958 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:34070 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:54571 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:59420 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Server neutron/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:38309 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:56823 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:39281 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:51705 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:38496 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:42106 to 10.19.104.19:3306 (mysql/TCP) [heat-admin@overcloud-controller-0 ~]$ journalctl -t haproxy -f -- Logs begin at Fri 2017-11-17 03:29:13 UTC. -- Nov 17 14:00:02 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:53112 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:02 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:50447 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:03 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:36493 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:03 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:33056 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:04 overcloud-controller-0 haproxy[16629]: Server cinder/overcloud-controller-2.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 18ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Nov 17 14:00:04 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:37571 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:04 overcloud-controller-0 haproxy[16629]: Server cinder/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Nov 17 14:00:05 overcloud-controller-0 haproxy[16629]: Server cinder/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 7ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Nov 17 14:00:06 overcloud-controller-0 haproxy[16629]: Server nova_osapi/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 20ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:37950 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Server neutron/overcloud-controller-2.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 7ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:54674 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:35393 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:09 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:44323 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:09 overcloud-controller-0 haproxy[16629]: Server neutron/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Nov 17 14:00:10 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:45506 to 10.19.104.19:3306 (mysql/TCP) Nov 17 14:00:11 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:43403 to 10.19.104.19:3306 (mysql/TCP) ^C [heat-admin@overcloud-controller-0 ~]$ ^C [heat-admin@overcloud-controller-0 ~]$ systemct^C [heat-admin@overcloud-controller-0 ~]$ sudo su [root@overcloud-controller-0 heat-admin]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 26s ago Docs: man:httpd(8) man:apachectl(8) Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS) Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" Memory: 792.6M CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ├─ 5824 /usr/sbin/httpd -DFOREGROUND ├─ 5825 /usr/sbin/httpd -DFOREGROUND ├─ 5827 /usr/sbin/httpd -DFOREGROUND ├─ 5828 /usr/sbin/httpd -DFOREGROUND ├─18338 /usr/sbin/httpd -DFOREGROUND ├─18441 /usr/sbin/httpd -DFOREGROUND ├─20505 /usr/sbin/httpd -DFOREGROUND ├─21603 /usr/sbin/httpd -DFOREGROUND ├─21813 /usr/sbin/httpd -DFOREGROUND ├─21814 /usr/sbin/httpd -DFOREGROUND ├─22761 /usr/sbin/httpd -DFOREGROUND ├─22870 /usr/sbin/httpd -DFOREGROUND ├─24253 /usr/sbin/httpd -DFOREGROUND ├─24325 /usr/sbin/httpd -DFOREGROUND ├─24326 /usr/sbin/httpd -DFOREGROUND ├─25972 /usr/sbin/httpd -DFOREGROUND └─26081 /usr/sbin/httpd -DFOREGROUND Nov 17 13:54:41 overcloud-controller-0 systemd[1]: Starting The Apache HTTP Server... Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressing... done Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressed 6 block(s) from 4 template(s) for 1 context(s). Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressing... done Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressed 6 block(s) from 4 template(s) for 1 context(s). Nov 17 13:55:51 overcloud-controller-0 systemd[1]: Started The Apache HTTP Server. [root@overcloud-controller-0 heat-admin]# systemctl status httpd | less ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 30s ago Docs: man:httpd(8) man:apachectl(8) Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS) Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" Memory: 792.6M CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 30s ago Docs: man:httpd(8) man:apachectl(8) Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS) Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" Memory: 792.6M CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ├─ 5824 /usr/sbin/httpd -DFOREGROUND [root@overcloud-controller-0 heat-admin]# syste Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 14:00:35 UTC): haproxy[16629]: proxy mysql has no server available! ^C [root@overcloud-controller-0 heat-admin]# systemctl start httpd [root@overcloud-controller-0 heat-admin]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 59s ago Docs: man:httpd(8) man:apachectl(8) Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS) Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" Memory: 792.8M CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ├─ 5824 /usr/sbin/httpd -DFOREGROUND ├─ 5825 /usr/sbin/httpd -DFOREGROUND ├─ 5827 /usr/sbin/httpd -DFOREGROUND ├─ 5828 /usr/sbin/httpd -DFOREGROUND ├─18338 /usr/sbin/httpd -DFOREGROUND ├─18441 /usr/sbin/httpd -DFOREGROUND ├─20505 /usr/sbin/httpd -DFOREGROUND ├─21603 /usr/sbin/httpd -DFOREGROUND ├─21813 /usr/sbin/httpd -DFOREGROUND ├─21814 /usr/sbin/httpd -DFOREGROUND ├─22761 /usr/sbin/httpd -DFOREGROUND ├─22870 /usr/sbin/httpd -DFOREGROUND ├─24253 /usr/sbin/httpd -DFOREGROUND ├─24325 /usr/sbin/httpd -DFOREGROUND ├─24326 /usr/sbin/httpd -DFOREGROUND ├─25972 /usr/sbin/httpd -DFOREGROUND └─26081 /usr/sbin/httpd -DFOREGROUND Nov 17 13:54:41 overcloud-controller-0 systemd[1]: Starting The Apache HTTP Server... Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressing... done Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressed 6 block(s) from 4 template(s) for 1 context(s). Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressing... done Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressed 6 block(s) from 4 template(s) for 1 context(s). Nov 17 13:55:51 overcloud-controller-0 systemd[1]: Started The Apache HTTP Server. [root@overcloud-controller-0 heat-admin]# systemctl status httpd | less ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 5min ago Docs: man:httpd(8) man:apachectl(8) Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS) Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" Memory: 792.8M CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ├─ 5824 /usr/sbin/httpd -DFOREGROUND [root@overcloud-controller-0 heat-admin]# systemctl enable httpd [root@overcloud-controller-0 heat-admin]# systemctl start httpd [root@overcloud-controller-0 heat-admin]# systemctl status httpd | less ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 5min ago Docs: man:httpd(8) man:apachectl(8) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 5min ago Docs: man:httpd(8) man:apachectl(8) Main PID: 5628 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" CGroup: /system.slice/httpd.service ├─ 5628 /usr/sbin/httpd -DFOREGROUND ├─ 5806 cinder_wsgi -DFOREGROUND ├─ 5808 cinder_wsgi -DFOREGROUND ├─ 5810 cinder_wsgi -DFOREGROUND ├─ 5813 cinder_wsgi -DFOREGROUND ├─ 5814 cinder_wsgi -DFOREGROUND ├─ 5815 cinder_wsgi -DFOREGROUND ├─ 5819 /usr/sbin/httpd -DFOREGROUND ├─ 5822 /usr/sbin/httpd -DFOREGROUND ├─ 5823 /usr/sbin/httpd -DFOREGROUND ├─ 5824 /usr/sbin/httpd -DFOREGROUND ├─ 5825 /usr/sbin/httpd -DFOREGROUND ├─ 5827 /usr/sbin/httpd -DFOREGROUND ├─ 5828 /usr/sbin/httpd -DFOREGROUND [root@overcloud-controller-0 heat-admin]# systemctl status httpd | less^C [root@overcloud-controller-0 heat-admin]# exit exit [heat-admin@overcloud-controller-0 ~]$ exit logout Connection to 192.168.0.17 closed. (reverse-i-search)`': ^C (overcloud) [stack@undercloud74 ~]$ openstack server list (overcloud) [stack@undercloud74 ~]$ openstack endpoint list +----------------------------------+-----------+--------------+----------------+ | ID | Region | Service Name | Service Type | +----------------------------------+-----------+--------------+----------------+ | c314035a0b144662a210576757a74286 | regionOne | aodh | alarming | | 312741ccc0734674ad95714568aff6c2 | regionOne | cinderv3 | volumev3 | | 3f4af20d4439446d81f88e395daa0f0b | regionOne | keystone | identity | | 5c8947ec92854abfbc972b5612f5ade5 | regionOne | placement | placement | | b1c92d23f4bc425697f8d80cca5b969b | regionOne | neutron | network | | ee8fb8c4b25f41adafc6c171b27ea5ac | regionOne | swift | object-store | | e6aab5cebe664588b450205007e8ec50 | regionOne | cinder | volume | | f536679edb9d43b78aafb14c67d3cd32 | regionOne | panko | event | | b8556f00a64c4bd8bdead8e48d5b5f30 | regionOne | gnocchi | metric | | 561851f651de4a31bc1db2807db455f8 | regionOne | heat | orchestration | | c749af9547c1402b905b5ab0c23b7e76 | regionOne | cinderv2 | volumev2 | | a45108301bcb4db7973e7bcaf62a3bf3 | regionOne | heat-cfn | cloudformation | | 48d4a63f2326465aa0a8b119dcc8158b | regionOne | glance | image | | 07ac040083a24dd0adc52a7e37140984 | regionOne | nova | compute | +----------------------------------+-----------+--------------+----------------+ (overcloud) [stack@undercloud74 ~]$ openstack endpoint list +----------------------------------+-----------+--------------+----------------+ | ID | Region | Service Name | Service Type | +----------------------------------+-----------+--------------+----------------+ | c314035a0b144662a210576757a74286 | regionOne | aodh | alarming | | 312741ccc0734674ad95714568aff6c2 | regionOne | cinderv3 | volumev3 | | 3f4af20d4439446d81f88e395daa0f0b | regionOne | keystone | identity | | 5c8947ec92854abfbc972b5612f5ade5 | regionOne | placement | placement | | b1c92d23f4bc425697f8d80cca5b969b | regionOne | neutron | network | | ee8fb8c4b25f41adafc6c171b27ea5ac | regionOne | swift | object-store | | e6aab5cebe664588b450205007e8ec50 | regionOne | cinder | volume | | f536679edb9d43b78aafb14c67d3cd32 | regionOne | panko | event | | b8556f00a64c4bd8bdead8e48d5b5f30 | regionOne | gnocchi | metric | | 561851f651de4a31bc1db2807db455f8 | regionOne | heat | orchestration | | c749af9547c1402b905b5ab0c23b7e76 | regionOne | cinderv2 | volumev2 | | a45108301bcb4db7973e7bcaf62a3bf3 | regionOne | heat-cfn | cloudformation | | 48d4a63f2326465aa0a8b119dcc8158b | regionOne | glance | image | | 07ac040083a24dd0adc52a7e37140984 | regionOne | nova | compute | +----------------------------------+-----------+--------------+----------------+ (overcloud) [stack@undercloud74 ~]$ openstack endpoint list --long +----------------------------------+-----------+--------------+----------------+---------------------------------------------------+-------------------------------------------+------------------------------------------------+ | ID | Region | Service Name | Service Type | PublicURL | AdminURL | InternalURL | +----------------------------------+-----------+--------------+----------------+---------------------------------------------------+-------------------------------------------+------------------------------------------------+ | c314035a0b144662a210576757a74286 | regionOne | aodh | alarming | https://10.19.184.160:13042 | http://10.19.104.19:8042 | http://10.19.104.19:8042 | | 312741ccc0734674ad95714568aff6c2 | regionOne | cinderv3 | volumev3 | https://10.19.184.160:13776/v3/%(tenant_id)s | http://10.19.104.19:8776/v3/%(tenant_id)s | http://10.19.104.19:8776/v3/%(tenant_id)s | | 3f4af20d4439446d81f88e395daa0f0b | regionOne | keystone | identity | https://10.19.184.160:13000 | http://192.168.0.12:35357 | http://10.19.104.19:5000 | | 5c8947ec92854abfbc972b5612f5ade5 | regionOne | placement | placement | https://10.19.184.160:13778/placement | http://10.19.104.19:8778/placement | http://10.19.104.19:8778/placement | | b1c92d23f4bc425697f8d80cca5b969b | regionOne | neutron | network | https://10.19.184.160:13696 | http://10.19.104.19:9696 | http://10.19.104.19:9696 | | ee8fb8c4b25f41adafc6c171b27ea5ac | regionOne | swift | object-store | https://10.19.184.160:13808/v1/AUTH_%(tenant_id)s | http://10.19.105.11:8080 | http://10.19.105.11:8080/v1/AUTH_%(tenant_id)s | | e6aab5cebe664588b450205007e8ec50 | regionOne | cinder | volume | https://10.19.184.160:13776/v1/%(tenant_id)s | http://10.19.104.19:8776/v1/%(tenant_id)s | http://10.19.104.19:8776/v1/%(tenant_id)s | | f536679edb9d43b78aafb14c67d3cd32 | regionOne | panko | event | https://10.19.184.160:13977 | http://10.19.104.19:8977 | http://10.19.104.19:8977 | | b8556f00a64c4bd8bdead8e48d5b5f30 | regionOne | gnocchi | metric | https://10.19.184.160:13041 | http://10.19.104.19:8041 | http://10.19.104.19:8041 | | 561851f651de4a31bc1db2807db455f8 | regionOne | heat | orchestration | https://10.19.184.160:13004/v1/%(tenant_id)s | http://10.19.104.19:8004/v1/%(tenant_id)s | http://10.19.104.19:8004/v1/%(tenant_id)s | | c749af9547c1402b905b5ab0c23b7e76 | regionOne | cinderv2 | volumev2 | https://10.19.184.160:13776/v2/%(tenant_id)s | http://10.19.104.19:8776/v2/%(tenant_id)s | http://10.19.104.19:8776/v2/%(tenant_id)s | | a45108301bcb4db7973e7bcaf62a3bf3 | regionOne | heat-cfn | cloudformation | https://10.19.184.160:13005/v1 | http://10.19.104.19:8000/v1 | http://10.19.104.19:8000/v1 | | 48d4a63f2326465aa0a8b119dcc8158b | regionOne | glance | image | https://10.19.184.160:13292 | http://10.19.104.19:9292 | http://10.19.104.19:9292 | | 07ac040083a24dd0adc52a7e37140984 | regionOne | nova | compute | https://10.19.184.160:13774/v2.1 | http://10.19.104.19:8774/v2.1 | http://10.19.104.19:8774/v2.1 | +----------------------------------+-----------+--------------+----------------+---------------------------------------------------+-------------------------------------------+------------------------------------------------+ (overcloud) [stack@undercloud74 ~]$ logout
workaround (from the Bz boddy ): --------------------------------- (1) connect to controller and do pcs cleanup (2) start manually httpd (if needed)
I'm truly not sure, so I'm setting blocker flag to -> '?' so we'll have PM decision.
Damien applied the patch https://review.openstack.org/#/c/521852/1 . (+1) on my environment , I rebooted the overcloud nodes using the script (graceful reboot, few minutes after the script started the servers pacemaker started the cluster successful and source overcloudrc returned correct results.
Verified on: puppet-tripleo-7.4.3-9.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ openstack server stop controller-0 (undercloud) [stack@undercloud-0 ~]$ openstack server stop controller-1 (undercloud) [stack@undercloud-0 ~]$ openstack server stop controller-2 (undercloud) [stack@undercloud-0 ~]$ openstack server stop compute-0 Waited till they got in to SHUTOFF mode started them and pcs status was clean + was able to boot an instance
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462