Bug 1713337 - [osp15] Instance HA is broken on OSP15
Summary: [osp15] Instance HA is broken on OSP15
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-pacemaker
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 15.0 (Stein)
Assignee: Michele Baldessari
QA Contact: pkomarov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-23 12:32 UTC by Michele Baldessari
Modified: 2019-09-26 10:51 UTC (History)
5 users (show)

Fixed In Version: puppet-pacemaker-0.7.3-0.20190612211132.658f4ee.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:22:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 660110 0 None None None 2019-05-23 12:36:36 UTC
OpenStack gerrit 660883 0 None None None 2019-05-23 12:36:36 UTC
OpenStack gerrit 660884 0 None None None 2019-05-23 12:36:36 UTC
OpenStack gerrit 664501 0 None None None 2019-06-11 14:30:25 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:22:49 UTC

Description Michele Baldessari 2019-05-23 12:32:28 UTC
Description of problem:
It currently does not work due to some changes in pcs/pcmk 2.0 around clones and remotes.

Comment 8 pkomarov 2019-07-27 17:00:47 UTC
verified: 

#check rpms: 
[stack@undercloud-0 ~]$ ansible compute-1  -mshell -a'rpm -qa|grep puppet-pacemaker-0.7.3-0.20190612211132.658f4ee.el8ost'
 [WARNING]: Found both group and host with same name: undercloud

 [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'.  If you need to
use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command
task or set 'command_warnings=False' in ansible.cfg to get rid of this message.

compute-1 | CHANGED | rc=0 >>
puppet-pacemaker-0.7.3-0.20190612211132.658f4ee.el8ost.noarch


#check instance-ha services are deployed: 
[stack@undercloud-0 ~]$ ansible compute -b -mshell -a"pcs status|grep -i 'compute\|evacuate'"

compute-1 | CHANGED | rc=0 >>
RemoteOnline: [ overcloud-novacomputeiha-0 overcloud-novacomputeiha-1 ]
 overcloud-novacomputeiha-0	(ocf::pacemaker:remote):	Started controller-0
 overcloud-novacomputeiha-1	(ocf::pacemaker:remote):	Started controller-1
 stonith-fence_compute-fence-nova	(stonith:fence_compute):	Started controller-2
 Clone Set: compute-unfence-trigger-clone [compute-unfence-trigger]
     Started: [ overcloud-novacomputeiha-0 overcloud-novacomputeiha-1 ]
 nova-evacuate	(ocf::openstack:NovaEvacuate):	Started controller-0

compute-0 | CHANGED | rc=0 >>
RemoteOnline: [ overcloud-novacomputeiha-0 overcloud-novacomputeiha-1 ]
 overcloud-novacomputeiha-0	(ocf::pacemaker:remote):	Started controller-0
 overcloud-novacomputeiha-1	(ocf::pacemaker:remote):	Started controller-1
 stonith-fence_compute-fence-nova	(stonith:fence_compute):	Started controller-2
 Clone Set: compute-unfence-trigger-clone [compute-unfence-trigger]
     Started: [ overcloud-novacomputeiha-0 overcloud-novacomputeiha-1 ]
 nova-evacuate	(ocf::openstack:NovaEvacuate):	Started controller-0

#test evacuation : 
(overcloud) [stack@undercloud-0 ~]$ . overcloudrc && openstack server create --flavor `openstack flavor list |grep -m1 m1|awk '{print $4}'` --image `openstack image list |grep -m1 cirros|awk '{print $4}'` --wait iha_test_vm1

+-------------------------------------+-----------------------------------------------------------------+
| Field                               | Value                                                           |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                          |
| OS-EXT-AZ:availability_zone         | nova                                                            |
| OS-EXT-SRV-ATTR:host                | overcloud-novacomputeiha-0.localdomain                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacomputeiha-0.localdomain                          |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000005                                               |
| OS-EXT-STS:power_state              | Running                                                         |
| OS-EXT-STS:task_state               | None                                                            |
| OS-EXT-STS:vm_state                 | active                                                          |
| OS-SRV-USG:launched_at              | 2019-07-27T15:03:32.000000                                      |
| OS-SRV-USG:terminated_at            | None                                                            |
| accessIPv4                          |                                                                 |
| accessIPv6                          |                                                                 |
| addresses                           | public=10.0.0.242                                               |
| adminPass                           | eEMqhR99LbTD                                                    |
| config_drive                        |                                                                 |
| created                             | 2019-07-27T15:03:12Z                                            |
| flavor                              | m1.nano (42)                                                    |
| hostId                              | 93767be5724d4c5474413fdaabdbf7cbb8ee669122f448d1054df1b4        |
| id                                  | 514fd3b0-800c-42d6-a2bb-3b3ec8c05d96                            |
| image                               | cirros-0.3.4-x86_64-disk (d92bdb7f-a2e1-470b-a3eb-e38b442961bd) |
| key_name                            | None                                                            |
| name                                | iha_test_vm1                                                    |
| progress                            | 0                                                               |
| project_id                          | de7f959a946d46faa03a9d4391da7cdf                                |
| properties                          |                                                                 |
| security_groups                     | name='default'                                                  |
| status                              | ACTIVE                                                          |
| updated                             | 2019-07-27T15:03:32Z                                            |
| user_id                             | 66a0ddd59f554e5f86f8d36de0f46083                                |
| volumes_attached                    |                                                                 |
+-------------------------------------+-----------------------------------------------------------------+

#reboot compute with vm : 
[stack@undercloud-0 ~]$ ansible compute-0 -b -mshell -a"echo b > /proc/sysrq-trigger"
 [WARNING]: Found both group and host with same name: undercloud

^[[Bcompute-0 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts.\r\npacket_write_wait: Connection to 192.168.24.6 port 22: Broken pipe",
    "unreachable": true
}


#now the vm is on the other compute : 
(overcloud) [stack@undercloud-0 ~]$ openstack server show iha_test_vm1
+-------------------------------------+-----------------------------------------------------------------+
| Field                               | Value                                                           |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                          |
| OS-EXT-AZ:availability_zone         | nova                                                            |
| OS-EXT-SRV-ATTR:host                | overcloud-novacomputeiha-1.localdomain                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacomputeiha-1.localdomain                          |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000005                                               |
| OS-EXT-STS:power_state              | Running                                                         |
| OS-EXT-STS:task_state               | None                                                            |
| OS-EXT-STS:vm_state                 | active                                                          |
| OS-SRV-USG:launched_at              | 2019-07-27T16:49:14.000000                                      |
| OS-SRV-USG:terminated_at            | None                                                            |
| accessIPv4                          |                                                                 |
| accessIPv6                          |                                                                 |
| addresses                           | public=10.0.0.242                                               |
| config_drive                        |                                                                 |
| created                             | 2019-07-27T15:03:12Z                                            |
| flavor                              | m1.nano (42)                                                    |
| hostId                              | 1abb1e0c962a4559e0ec1351c0d940ed318391d43aabd77880d0c507        |
| id                                  | 514fd3b0-800c-42d6-a2bb-3b3ec8c05d96                            |
| image                               | cirros-0.3.4-x86_64-disk (d92bdb7f-a2e1-470b-a3eb-e38b442961bd) |
| key_name                            | None                                                            |
| name                                | iha_test_vm1                                                    |
| progress                            | 0                                                               |
| project_id                          | de7f959a946d46faa03a9d4391da7cdf                                |
| properties                          |                                                                 |
| security_groups                     | name='default'                                                  |
| status                              | ACTIVE                                                          |
| updated                             | 2019-07-27T16:49:25Z                                            |
| user_id                             | 66a0ddd59f554e5f86f8d36de0f46083                                |
| volumes_attached                    |                                                                 |
+-------------------------------------+-----------------------------------------------------------------+

Comment 12 errata-xmlrpc 2019-09-21 11:22:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.