Bug 1514495 – [OSP12] when using undercloud nova to stop controller nodes, pacemaker resource stop operation times out ('overcloudrc' won't function after clean reboot)

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1514495 - [OSP12] when using undercloud nova to stop controller nodes, pacemaker resource stop operation times out ('overcloudrc' won't function after clean reboot)

Summary:	[OSP12] when using undercloud nova to stop controller nodes, pacemaker resour...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	puppet-tripleo (Show other bugs)
Sub Component:
Version:	12.0 (Pike)
Hardware:	x86_64 Linux

Priority	high Severity high
Target Milestone:	rc
Target Release:	12.0 (Pike)
Assigned To:	Michele Baldessari
QA Contact:	Omri Hochman
Docs Contact:

URL:
Whiteboard:
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2017-11-17 10:03 EST by Omri Hochman
Modified:	2018-02-05 14:18 EST (History)
CC List:	14 users (show)

See Also:
Fixed In Version:	puppet-tripleo-7.4.3-8.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-12-13 17:22:16 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Launchpad	1733348	None	None	None	2017-11-20 10:58 EST
OpenStack gerrit	521852	None	None	None	2017-11-21 09:03 EST
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-15 20:43:25 EST

Groups: None (edit)

Description Omri Hochman 2017-11-17 10:03:38 EST

[OSP12] when using undercloud nova to stop controller nodes, pacemaker resource stop operation times out ('overcloudrc' won't function after clean reboot) 


Environment (OSP12 RC Puddle) : 
-------------------------------
pacemaker-1.1.16-12.el7_4.4.x86_64
pacemaker-cli-1.1.16-12.el7_4.4.x86_64
ansible-pacemaker-1.0.3-2.el7ost.noarch
puppet-pacemaker-0.6.1-0.20171103172230.9a46ecd.el7ost.noarch
pacemaker-remote-1.1.16-12.el7_4.4.x86_64
pacemaker-libs-1.1.16-12.el7_4.4.x86_64
pacemaker-cluster-libs-1.1.16-12.el7_4.4.x86_64
openstack-tripleo-heat-templates-7.0.3-8.el7ost.noarch
puppet-pacemaker-0.6.1-0.20171103172230.9a46ecd.el7ost.noarch
openstack-nova-common-16.0.2-2.el7ost.noarch
openstack-mistral-common-5.1.1-0.20171104083011.5372770.el7ost.noarch
python-pycadf-common-2.6.0-1.el7ost.noarch
openstack-neutron-common-11.0.1-4.el7ost.noarch
openstack-ironic-common-9.1.3-0.20171106091155.fceb40f.el7ost.noarch
libX11-common-1.6.5-1.el7.noarch
docker-common-1.12.6-61.git85d7426.el7.x86_64
rhevm-guest-agent-common-1.0.11-3.el7ev.noarch
dhcp-common-4.2.5-58.el7.x86_64
openstack-tripleo-common-containers-7.6.3-3.el7ost.noarch
glibc-common-2.17-196.el7.x86_64
python-ryu-common-4.15-1.el7ost.noarch
openstack-heat-common-9.0.1-3.el7ost.noarch
grub2-common-2.02-0.65.el7_4.2.noarch
openstack-tripleo-common-7.6.3-3.el7ost.noarch
qemu-kvm-common-rhev-2.9.0-16.el7_4.11.x86_64



Steps to reproduce :
---------------------
(1) Run 'Clean reboot' from the undercloud --> 
[root@undercloud74 tripleo]# vi reboot_oc_nodes.sh
#!/bin/bash
source /home/stack/stackrc
echo "turning all nodes off"
for i in `nova list|awk -F"|" '/Running/ {print $2}'`; do echo $i; nova stop $i; done
sleep 60
echo "starting all but ceph nodes"
for i in `nova list|awk -F"|" '!/ceph/ && /SHUTOFF/ {print $2}'`; do echo $i; nova start $i;  done
sleep 60
echo "starting ceph nodes"
for i in `nova list|awk -F"|" '/ceph/ && /SHUTOFF/ {print $2}'`; do echo $i; nova start $i;  done


Results :
---------

(A)  pacemaker resource stop operation times out 
(B) Overcloudrc won't function: 
(overcloud) [stack@undercloud74 ~]$ openstack server list
Failed to discover available identity versions when contacting https://10.19.184.160:13000/v2.0. Attempting to parse version from URL.
Unable to establish connection to https://10.19.184.160:13000/v2.0/tokens: HTTPSConnectionPool(host='10.19.184.160', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x34f7f50>: Failed to establish a new connection: [Errno 113] No route to host',))



Logs: 
------

[root@undercloud74 ~]# journalctl | grep -e 'Stopping Docker App'
Nov 16 21:40:10 undercloud74.local systemd[1]: Stopping Docker Application Container Engine...
[root@undercloud74 ~]# # when did you restart ?
[root@undercloud74 ~]# journalctl
-- Logs begin at Tue 2017-10-10 12:45:58 EDT, end at Fri 2017-11-17 09:20:53 EST. --
Oct 10 12:45:58 localhost systemd-journal[103]: Runtime journal is using 8.0M (max allowed 995.8M, trying to leave 1.4G free of 9.7G available → current limit 995.8M).
Oct 10 12:45:58 localhost kernel: Initializing cgroup subsys cpuset
Oct 10 12:45:58 localhost kernel: Initializing cgroup subsys cpu
Oct 10 12:45:58 localhost kernel: Initializing cgroup subsys cpuacct
Oct 10 12:45:58 localhost kernel: Linux version 3.10.0-693.el7.x86_64 (mockbuild@x86-038.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SM
Oct 10 12:45:58 localhost kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115
Oct 10 12:45:58 localhost kernel: e820: BIOS-provided physical RAM map:
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bfff6fff] usable
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000bfff7000-0x00000000bfffffff] reserved
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Oct 10 12:45:58 localhost kernel: BIOS-e820: [mem 0x0000000100000000-0x000000053fffffff] usable
Oct 10 12:45:58 localhost kernel: NX (Execute Disable) protection: active
Oct 10 12:45:58 localhost kernel: SMBIOS 2.4 present.
Oct 10 12:45:58 localhost kernel: DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011
Oct 10 12:45:58 localhost kernel: Hypervisor detected: KVM
Oct 10 12:45:58 localhost kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Oct 10 12:45:58 localhost kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Oct 10 12:45:58 localhost kernel: e820: last_pfn = 0x540000 max_arch_pfn = 0x400000000
Oct 10 12:45:58 localhost kernel: MTRR default type: write-back

heat-admin@overcloud-controller-0 ~]$ sudo su
[root@overcloud-controller-0 heat-admin]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Fri Nov 17 13:57:10 2017
Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0

12 nodes configured
37 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
   galera-bundle-2      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Stopped overcloud-controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Stopped overcloud-controller-0
   redis-bundle-2       (ocf::heartbeat:redis): Stopped overcloud-controller-0
 ip-192.168.0.12        (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.184.160       (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.104.14        (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.104.19        (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.105.11        (ocf::heartbeat:IPaddr2):       Stopped
 ip-192.168.200.15      (ocf::heartbeat:IPaddr2):       Stopped
 Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ]
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ]
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0

Failed Actions:
* rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=93, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20023ms
* rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=94, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20037ms
* galera-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=96, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20035ms
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=95, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20036ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=97, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20034ms
* galera-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=98, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20039ms
* redis-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=99, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20032ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=100, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20033ms
* redis-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=101, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20030ms
* haproxy-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=102, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20027ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=103, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms
* haproxy-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=104, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms
* rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=91, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms
* rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=92, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=93, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20006ms
* galera-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=94, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=95, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms
* galera-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=96, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms
* redis-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=97, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=98, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* redis-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=99, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* haproxy-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=100, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=101, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* haproxy-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=102, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=91, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms
* rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=92, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=93, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* galera-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=94, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20021ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=95, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms
* galera-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=96, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms
* redis-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=97, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20017ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=98, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20016ms
* redis-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=99, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms
* haproxy-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=100, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=101, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms
* haproxy-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=102, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 heat-admin]# pcs status | less
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Fri Nov 17 13:57:22 2017
Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0

12 nodes configured
37 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
   galera-bundle-2      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Stopped overcloud-controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Stopped overcloud-controller-0
   redis-bundle-2       (ocf::heartbeat:redis): Stopped overcloud-controller-0
 ip-192.168.0.12        (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.184.160       (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.104.14        (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.104.19        (ocf::heartbeat:IPaddr2):       Stopped
 ip-10.19.105.11        (ocf::heartbeat:IPaddr2):       Stopped
 ip-192.168.200.15      (ocf::heartbeat:IPaddr2):       Stopped
 Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ]
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ]
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        FAILED (blocked)[ overcloud-controller-0 overcloud-controller-2 overcloud-controller-1 ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0

Failed Actions:
* rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=93, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20023ms
* rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=94, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20037ms
* galera-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=96, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20035ms
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=95, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20036ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=97, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20034ms
* galera-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=98, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20039ms
* redis-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=99, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20032ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=100, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20033ms
* redis-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=101, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20030ms
* haproxy-bundle-docker-0_stop_0 on overcloud-controller-0 'unknown error' (1): call=102, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20027ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-0 'unknown error' (1): call=103, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms
* haproxy-bundle-docker-2_stop_0 on overcloud-controller-0 'unknown error' (1): call=104, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms
* rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=91, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms
* rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=92, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=93, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20006ms
* galera-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=94, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=95, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms
* galera-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=96, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms
* redis-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=97, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=98, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* redis-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=99, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* haproxy-bundle-docker-0_stop_0 on overcloud-controller-2 'unknown error' (1): call=100, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-2 'unknown error' (1): call=101, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20001ms
* haproxy-bundle-docker-2_stop_0 on overcloud-controller-2 'unknown error' (1): call=102, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* rabbitmq-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=91, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20004ms
* rabbitmq-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=92, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20022ms
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=93, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20002ms
* galera-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=94, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20021ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=95, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms
* galera-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=96, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20019ms
* redis-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=97, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20017ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=98, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20016ms
* redis-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=99, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms
* haproxy-bundle-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=100, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20013ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=101, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20007ms
* haproxy-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=102, status=Timed Out, exitreason='none',
    last-rc-change='Fri Nov 17 13:55:35 2017', queued=0ms, exec=20005ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 heat-admin]# pcs resource cleanup
Error: Cleaning up all resources on all nodes will execute more than 100 operations in the cluster, which may negatively impact the responsiveness of the cluster. Consider specifying resource and/or node, use --force to override
[root@overcloud-controller-0 heat-admin]# pcs resource cleanup --force
Waiting for 1 replies from the CRMd. OK
[root@overcloud-controller-0 heat-admin]# pcs status

Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:36 UTC):

haproxy[16629]: proxy mysql has no server available!


Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:36 UTC):

haproxy[16629]: proxy horizon has no server available!

Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Fri Nov 17 13:58:37 2017
Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0

12 nodes configured
37 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-1
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-2
 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Stopped overcloud-controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Stopped overcloud-controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Stopped overcloud-controller-2
 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Stopped overcloud-controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Stopped overcloud-controller-1
   redis-bundle-2       (ocf::heartbeat:redis): Stopped overcloud-controller-2
 ip-192.168.0.12        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.19.184.160       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 ip-10.19.104.14        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 ip-10.19.104.19        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.19.105.11        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 ip-192.168.200.15      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started overcloud-controller-0
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started overcloud-controller-1
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started overcloud-controller-2
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0

Daemon Status:
  corosync: active/enabled

Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:37 UTC):

haproxy[16629]: proxy neutron has no server available!

  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 heat-admin]#
Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:37 UTC):

haproxy[16629]: proxy redis has no server available!


Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:39 UTC):

haproxy[16629]: proxy gnocchi has no server available!


[root@overcloud-controller-0 heat-admin]#
Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:46 UTC):

haproxy[16629]: proxy cinder has no server available!

pcs status
Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:47 UTC):

haproxy[16629]: proxy nova_metadata has no server available!



Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 13:58:47 UTC):

haproxy[16629]: proxy nova_osapi has no server available!

Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Fri Nov 17 13:58:47 2017
Last change: Fri Nov 17 04:36:15 2017 by root via cibadmin on overcloud-controller-0

12 nodes configured
37 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.0.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-1
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Stopped overcloud-controller-2
 Docker container set: galera-bundle [192.168.0.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Slave overcloud-controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Stopped overcloud-controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Stopped overcloud-controller-2
 Docker container set: redis-bundle [192.168.0.1:8787/rhosp12/openstack-redis-docker:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Slave overcloud-controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Stopped overcloud-controller-1
   redis-bundle-2       (ocf::heartbeat:redis): Stopped overcloud-controller-2
 ip-192.168.0.12        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.19.184.160       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 ip-10.19.104.14        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 ip-10.19.104.19        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.19.105.11        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 ip-192.168.200.15      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Docker container set: haproxy-bundle [192.168.0.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started overcloud-controller-0
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started overcloud-controller-1
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started overcloud-controller-2
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 heat-admin]# exit
exit
[heat-admin@overcloud-controller-0 ~]$ exit
logout
Connection to 192.168.0.17 closed.
(undercloud) [stack@undercloud74 ~]$ ping 10.19.184.160
PING 10.19.184.160 (10.19.184.160) 56(84) bytes of data.
64 bytes from 10.19.184.160: icmp_seq=1 ttl=64 time=0.516 ms
64 bytes from 10.19.184.160: icmp_seq=2 ttl=64 time=0.387 ms
^C
--- 10.19.184.160 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.387/0.451/0.516/0.067 ms
(undercloud) [stack@undercloud74 ~]$ source overcloud
-bash: overcloud: No such file or directory
(undercloud) [stack@undercloud74 ~]$ source overcloudrc
(overcloud) [stack@undercloud74 ~]$ openstack server list
^CTraceback (most recent call last):
  File "/bin/openstack", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 213, in main
    return OpenStackShell().run(argv)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 393, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 200, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 437, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 99, in auth_ref
    return super(ClientManager, self).auth_ref
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 239, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 853, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 703, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 761, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
    chunked=chunked)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 386, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 759, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 653, in read
    v = self._sslobj.read(len or 1024)
KeyboardInterrupt
(overcloud) [stack@undercloud74 ~]$ ^C
(overcloud) [stack@undercloud74 ~]$ ^C
(overcloud) [stack@undercloud74 ~]$ ssh heat-admin@192.168.0.17
Last login: Fri Nov 17 13:57:03 2017 from 192.168.0.1
[heat-admin@overcloud-controller-0 ~]$ journalctl -t haproxy
-- Logs begin at Fri 2017-11-17 03:29:13 UTC, end at Fri 2017-11-17 13:59:59 UTC. --
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy aodh started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy cinder started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy glance_api started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy gnocchi started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy haproxy.stats started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy heat_api started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy heat_cfn started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy horizon started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy keystone_admin started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy keystone_public started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy mysql started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy neutron started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_metadata started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_novncproxy started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_osapi started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy nova_placement started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy panko started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy redis started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66751]: Proxy swift_proxy_server started.
Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: Server aodh/overcloud-controller-0.internalapi.localdomain is DOWN, reason: Layer4 connection problem, info: "Connection
Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: Server aodh/overcloud-controller-1.internalapi.localdomain is DOWN, reason: Layer4 connection problem, info: "Connection
Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: Server aodh/overcloud-controller-2.internalapi.localdomain is DOWN, reason: Layer4 connection problem, info: "Connection
Nov 17 04:14:37 overcloud-controller-0 haproxy[66758]: proxy aodh has no server available!
...skipping...
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:49909 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:51667 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:40795 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:41864 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:42039 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:40362 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:60199 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:40534 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Server nova_metadata/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info:
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:32825 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:52839 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:53001 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:41682 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:40958 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:34070 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:54571 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:00 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:59420 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Server neutron/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK",
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:38309 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:56823 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:39281 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:51705 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.16:38496 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:01 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:42106 to 10.19.104.19:3306 (mysql/TCP)
[heat-admin@overcloud-controller-0 ~]$ journalctl -t haproxy -f
-- Logs begin at Fri 2017-11-17 03:29:13 UTC. --
Nov 17 14:00:02 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:53112 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:02 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:50447 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:03 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:36493 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:03 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:33056 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:04 overcloud-controller-0 haproxy[16629]: Server cinder/overcloud-controller-2.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 18ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Nov 17 14:00:04 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:37571 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:04 overcloud-controller-0 haproxy[16629]: Server cinder/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Nov 17 14:00:05 overcloud-controller-0 haproxy[16629]: Server cinder/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 7ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Nov 17 14:00:06 overcloud-controller-0 haproxy[16629]: Server nova_osapi/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 20ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:37950 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Server neutron/overcloud-controller-2.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 7ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.17:54674 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:07 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:35393 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:09 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:44323 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:09 overcloud-controller-0 haproxy[16629]: Server neutron/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Nov 17 14:00:10 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:45506 to 10.19.104.19:3306 (mysql/TCP)
Nov 17 14:00:11 overcloud-controller-0 haproxy[16629]: Connect from 10.19.104.23:43403 to 10.19.104.19:3306 (mysql/TCP)
^C
[heat-admin@overcloud-controller-0 ~]$ ^C
[heat-admin@overcloud-controller-0 ~]$ systemct^C
[heat-admin@overcloud-controller-0 ~]$ sudo su
[root@overcloud-controller-0 heat-admin]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 26s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   Memory: 792.6M
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
           ├─ 5824 /usr/sbin/httpd -DFOREGROUND
           ├─ 5825 /usr/sbin/httpd -DFOREGROUND
           ├─ 5827 /usr/sbin/httpd -DFOREGROUND
           ├─ 5828 /usr/sbin/httpd -DFOREGROUND
           ├─18338 /usr/sbin/httpd -DFOREGROUND
           ├─18441 /usr/sbin/httpd -DFOREGROUND
           ├─20505 /usr/sbin/httpd -DFOREGROUND
           ├─21603 /usr/sbin/httpd -DFOREGROUND
           ├─21813 /usr/sbin/httpd -DFOREGROUND
           ├─21814 /usr/sbin/httpd -DFOREGROUND
           ├─22761 /usr/sbin/httpd -DFOREGROUND
           ├─22870 /usr/sbin/httpd -DFOREGROUND
           ├─24253 /usr/sbin/httpd -DFOREGROUND
           ├─24325 /usr/sbin/httpd -DFOREGROUND
           ├─24326 /usr/sbin/httpd -DFOREGROUND
           ├─25972 /usr/sbin/httpd -DFOREGROUND
           └─26081 /usr/sbin/httpd -DFOREGROUND

Nov 17 13:54:41 overcloud-controller-0 systemd[1]: Starting The Apache HTTP Server...
Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressing... done
Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressed 6 block(s) from 4 template(s) for 1 context(s).
Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressing... done
Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressed 6 block(s) from 4 template(s) for 1 context(s).
Nov 17 13:55:51 overcloud-controller-0 systemd[1]: Started The Apache HTTP Server.
[root@overcloud-controller-0 heat-admin]# systemctl status httpd | less
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 30s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   Memory: 792.6M
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 30s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   Memory: 792.6M
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
           ├─ 5824 /usr/sbin/httpd -DFOREGROUND
[root@overcloud-controller-0 heat-admin]# syste
Broadcast message from systemd-journald@overcloud-controller-0 (Fri 2017-11-17 14:00:35 UTC):

haproxy[16629]: proxy mysql has no server available!

^C
[root@overcloud-controller-0 heat-admin]# systemctl start httpd
[root@overcloud-controller-0 heat-admin]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 4min 59s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   Memory: 792.8M
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
           ├─ 5824 /usr/sbin/httpd -DFOREGROUND
           ├─ 5825 /usr/sbin/httpd -DFOREGROUND
           ├─ 5827 /usr/sbin/httpd -DFOREGROUND
           ├─ 5828 /usr/sbin/httpd -DFOREGROUND
           ├─18338 /usr/sbin/httpd -DFOREGROUND
           ├─18441 /usr/sbin/httpd -DFOREGROUND
           ├─20505 /usr/sbin/httpd -DFOREGROUND
           ├─21603 /usr/sbin/httpd -DFOREGROUND
           ├─21813 /usr/sbin/httpd -DFOREGROUND
           ├─21814 /usr/sbin/httpd -DFOREGROUND
           ├─22761 /usr/sbin/httpd -DFOREGROUND
           ├─22870 /usr/sbin/httpd -DFOREGROUND
           ├─24253 /usr/sbin/httpd -DFOREGROUND
           ├─24325 /usr/sbin/httpd -DFOREGROUND
           ├─24326 /usr/sbin/httpd -DFOREGROUND
           ├─25972 /usr/sbin/httpd -DFOREGROUND
           └─26081 /usr/sbin/httpd -DFOREGROUND

Nov 17 13:54:41 overcloud-controller-0 systemd[1]: Starting The Apache HTTP Server...
Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressing... done
Nov 17 13:55:39 overcloud-controller-0 python[3787]: Compressed 6 block(s) from 4 template(s) for 1 context(s).
Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressing... done
Nov 17 13:55:49 overcloud-controller-0 python[5097]: Compressed 6 block(s) from 4 template(s) for 1 context(s).
Nov 17 13:55:51 overcloud-controller-0 systemd[1]: Started The Apache HTTP Server.
[root@overcloud-controller-0 heat-admin]# systemctl status httpd | less
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 5min ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5097 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 3787 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   Memory: 792.8M
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
           ├─ 5824 /usr/sbin/httpd -DFOREGROUND
[root@overcloud-controller-0 heat-admin]# systemctl enable httpd
[root@overcloud-controller-0 heat-admin]# systemctl start httpd
[root@overcloud-controller-0 heat-admin]# systemctl status httpd | less
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 5min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Fri 2017-11-17 13:55:51 UTC; 5min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 5628 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─ 5628 /usr/sbin/httpd -DFOREGROUND
           ├─ 5806 cinder_wsgi     -DFOREGROUND
           ├─ 5808 cinder_wsgi     -DFOREGROUND
           ├─ 5810 cinder_wsgi     -DFOREGROUND
           ├─ 5813 cinder_wsgi     -DFOREGROUND
           ├─ 5814 cinder_wsgi     -DFOREGROUND
           ├─ 5815 cinder_wsgi     -DFOREGROUND
           ├─ 5819 /usr/sbin/httpd -DFOREGROUND
           ├─ 5822 /usr/sbin/httpd -DFOREGROUND
           ├─ 5823 /usr/sbin/httpd -DFOREGROUND
           ├─ 5824 /usr/sbin/httpd -DFOREGROUND
           ├─ 5825 /usr/sbin/httpd -DFOREGROUND
           ├─ 5827 /usr/sbin/httpd -DFOREGROUND
           ├─ 5828 /usr/sbin/httpd -DFOREGROUND
[root@overcloud-controller-0 heat-admin]# systemctl status httpd | less^C
[root@overcloud-controller-0 heat-admin]# exit
exit
[heat-admin@overcloud-controller-0 ~]$ exit
logout
Connection to 192.168.0.17 closed.
(reverse-i-search)`': ^C
(overcloud) [stack@undercloud74 ~]$ openstack server list

(overcloud) [stack@undercloud74 ~]$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+
| ID                               | Region    | Service Name | Service Type   |
+----------------------------------+-----------+--------------+----------------+
| c314035a0b144662a210576757a74286 | regionOne | aodh         | alarming       |
| 312741ccc0734674ad95714568aff6c2 | regionOne | cinderv3     | volumev3       |
| 3f4af20d4439446d81f88e395daa0f0b | regionOne | keystone     | identity       |
| 5c8947ec92854abfbc972b5612f5ade5 | regionOne | placement    | placement      |
| b1c92d23f4bc425697f8d80cca5b969b | regionOne | neutron      | network        |
| ee8fb8c4b25f41adafc6c171b27ea5ac | regionOne | swift        | object-store   |
| e6aab5cebe664588b450205007e8ec50 | regionOne | cinder       | volume         |
| f536679edb9d43b78aafb14c67d3cd32 | regionOne | panko        | event          |
| b8556f00a64c4bd8bdead8e48d5b5f30 | regionOne | gnocchi      | metric         |
| 561851f651de4a31bc1db2807db455f8 | regionOne | heat         | orchestration  |
| c749af9547c1402b905b5ab0c23b7e76 | regionOne | cinderv2     | volumev2       |
| a45108301bcb4db7973e7bcaf62a3bf3 | regionOne | heat-cfn     | cloudformation |
| 48d4a63f2326465aa0a8b119dcc8158b | regionOne | glance       | image          |
| 07ac040083a24dd0adc52a7e37140984 | regionOne | nova         | compute        |
+----------------------------------+-----------+--------------+----------------+
(overcloud) [stack@undercloud74 ~]$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+
| ID                               | Region    | Service Name | Service Type   |
+----------------------------------+-----------+--------------+----------------+
| c314035a0b144662a210576757a74286 | regionOne | aodh         | alarming       |
| 312741ccc0734674ad95714568aff6c2 | regionOne | cinderv3     | volumev3       |
| 3f4af20d4439446d81f88e395daa0f0b | regionOne | keystone     | identity       |
| 5c8947ec92854abfbc972b5612f5ade5 | regionOne | placement    | placement      |
| b1c92d23f4bc425697f8d80cca5b969b | regionOne | neutron      | network        |
| ee8fb8c4b25f41adafc6c171b27ea5ac | regionOne | swift        | object-store   |
| e6aab5cebe664588b450205007e8ec50 | regionOne | cinder       | volume         |
| f536679edb9d43b78aafb14c67d3cd32 | regionOne | panko        | event          |
| b8556f00a64c4bd8bdead8e48d5b5f30 | regionOne | gnocchi      | metric         |
| 561851f651de4a31bc1db2807db455f8 | regionOne | heat         | orchestration  |
| c749af9547c1402b905b5ab0c23b7e76 | regionOne | cinderv2     | volumev2       |
| a45108301bcb4db7973e7bcaf62a3bf3 | regionOne | heat-cfn     | cloudformation |
| 48d4a63f2326465aa0a8b119dcc8158b | regionOne | glance       | image          |
| 07ac040083a24dd0adc52a7e37140984 | regionOne | nova         | compute        |
+----------------------------------+-----------+--------------+----------------+
(overcloud) [stack@undercloud74 ~]$ openstack endpoint list --long
+----------------------------------+-----------+--------------+----------------+---------------------------------------------------+-------------------------------------------+------------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | PublicURL                                         | AdminURL                                  | InternalURL                                    |
+----------------------------------+-----------+--------------+----------------+---------------------------------------------------+-------------------------------------------+------------------------------------------------+
| c314035a0b144662a210576757a74286 | regionOne | aodh         | alarming       | https://10.19.184.160:13042                       | http://10.19.104.19:8042                  | http://10.19.104.19:8042                       |
| 312741ccc0734674ad95714568aff6c2 | regionOne | cinderv3     | volumev3       | https://10.19.184.160:13776/v3/%(tenant_id)s      | http://10.19.104.19:8776/v3/%(tenant_id)s | http://10.19.104.19:8776/v3/%(tenant_id)s      |
| 3f4af20d4439446d81f88e395daa0f0b | regionOne | keystone     | identity       | https://10.19.184.160:13000                       | http://192.168.0.12:35357                 | http://10.19.104.19:5000                       |
| 5c8947ec92854abfbc972b5612f5ade5 | regionOne | placement    | placement      | https://10.19.184.160:13778/placement             | http://10.19.104.19:8778/placement        | http://10.19.104.19:8778/placement             |
| b1c92d23f4bc425697f8d80cca5b969b | regionOne | neutron      | network        | https://10.19.184.160:13696                       | http://10.19.104.19:9696                  | http://10.19.104.19:9696                       |
| ee8fb8c4b25f41adafc6c171b27ea5ac | regionOne | swift        | object-store   | https://10.19.184.160:13808/v1/AUTH_%(tenant_id)s | http://10.19.105.11:8080                  | http://10.19.105.11:8080/v1/AUTH_%(tenant_id)s |
| e6aab5cebe664588b450205007e8ec50 | regionOne | cinder       | volume         | https://10.19.184.160:13776/v1/%(tenant_id)s      | http://10.19.104.19:8776/v1/%(tenant_id)s | http://10.19.104.19:8776/v1/%(tenant_id)s      |
| f536679edb9d43b78aafb14c67d3cd32 | regionOne | panko        | event          | https://10.19.184.160:13977                       | http://10.19.104.19:8977                  | http://10.19.104.19:8977                       |
| b8556f00a64c4bd8bdead8e48d5b5f30 | regionOne | gnocchi      | metric         | https://10.19.184.160:13041                       | http://10.19.104.19:8041                  | http://10.19.104.19:8041                       |
| 561851f651de4a31bc1db2807db455f8 | regionOne | heat         | orchestration  | https://10.19.184.160:13004/v1/%(tenant_id)s      | http://10.19.104.19:8004/v1/%(tenant_id)s | http://10.19.104.19:8004/v1/%(tenant_id)s      |
| c749af9547c1402b905b5ab0c23b7e76 | regionOne | cinderv2     | volumev2       | https://10.19.184.160:13776/v2/%(tenant_id)s      | http://10.19.104.19:8776/v2/%(tenant_id)s | http://10.19.104.19:8776/v2/%(tenant_id)s      |
| a45108301bcb4db7973e7bcaf62a3bf3 | regionOne | heat-cfn     | cloudformation | https://10.19.184.160:13005/v1                    | http://10.19.104.19:8000/v1               | http://10.19.104.19:8000/v1                    |
| 48d4a63f2326465aa0a8b119dcc8158b | regionOne | glance       | image          | https://10.19.184.160:13292                       | http://10.19.104.19:9292                  | http://10.19.104.19:9292                       |
| 07ac040083a24dd0adc52a7e37140984 | regionOne | nova         | compute        | https://10.19.184.160:13774/v2.1                  | http://10.19.104.19:8774/v2.1             | http://10.19.104.19:8774/v2.1                  |
+----------------------------------+-----------+--------------+----------------+---------------------------------------------------+-------------------------------------------+------------------------------------------------+
(overcloud) [stack@undercloud74 ~]$ logout

Comment 1 Omri Hochman 2017-11-17 10:05:55 EST

workaround (from the Bz boddy ):
---------------------------------
(1) connect to controller and do pcs cleanup 
(2) start manually httpd (if needed)

Comment 2 Omri Hochman 2017-11-17 14:42:55 EST

I'm truly not sure, so I'm setting blocker flag to -> '?' so we'll have PM decision.

Comment 3 Omri Hochman 2017-11-21 14:40:39 EST

Damien applied the patch https://review.openstack.org/#/c/521852/1 . (+1) on my environment ,  I rebooted the overcloud nodes using the script (graceful reboot, few minutes after the script started the servers pacemaker started the cluster successful and source overcloudrc returned correct results.

Comment 5 Udi Shkalim 2017-11-28 07:41:54 EST

Verified on: puppet-tripleo-7.4.3-9.el7ost.noarch

(undercloud) [stack@undercloud-0 ~]$ openstack server stop controller-0
(undercloud) [stack@undercloud-0 ~]$ openstack server stop controller-1
(undercloud) [stack@undercloud-0 ~]$ openstack server stop controller-2
(undercloud) [stack@undercloud-0 ~]$ openstack server stop compute-0

Waited till they got in to SHUTOFF mode started them and pcs status was clean + was able to boot an instance

Comment 8 errata-xmlrpc 2017-12-13 17:22:16 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.