Bug 1444621 - Autofencing failed to configure fencing on node with pacemaker_remote
Summary: Autofencing failed to configure fencing on node with pacemaker_remote
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-pacemaker
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z4
: 11.0 (Ocata)
Assignee: Chris Jones
QA Contact: Udi Shkalim
URL:
Whiteboard:
Depends On: 1483920
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-23 11:38 UTC by Asaf Hirshberg
Modified: 2018-02-13 16:31 UTC (History)
15 users (show)

Fixed In Version: puppet-pacemaker-0.6.0-1.el7ost puppet-tripleo-6.5.0-7.el7ost openstack-tripleo-heat-templates-6.2.0-2.el7ost
Doc Type: Bug Fix
Doc Text:
Prior to this update, fencing could not be configured on nodes running Pacemaker Remote. With this update, the fencing configuration is now compatible with Pacemaker Remote nodes. As a result, fencing is now properly created.
Clone Of:
Environment:
Last Closed: 2018-02-13 16:31:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1686115 0 None None None 2017-04-25 15:13:55 UTC
OpenStack gerrit 459743 0 None None None 2017-04-25 15:14:12 UTC
OpenStack gerrit 459744 0 None None None 2017-04-25 15:14:32 UTC
OpenStack gerrit 459787 0 None None None 2017-04-25 16:46:11 UTC
Red Hat Product Errata RHBA-2018:0310 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 director Bug Fix Advisory 2018-02-14 00:13:02 UTC

Description Asaf Hirshberg 2017-04-23 11:38:49 UTC
Description of problem:
autofencing didn't configure fencing on nodes with pacemaker_remote in composable roles environment. 

[stack@puma33 ~]$ openstack overcloud generate fencing -a reboot --ipmi-lanplus --ipmi-level administrator instackenv.json
parameter_defaults:
  EnableFencing: true
  FencingConfig:
    devices:
    - agent: fence_ipmilan
      host_mac: 44:1e:a1:73:39:91
      params:
        action: reboot
        ipaddr: 10.35.160.18
        lanplus: true
        login: admin
        passwd: admin
        pcmk_host_list: puma04
        privlvl: administrator
    - agent: fence_ipmilan
      host_mac: 44:1e:a1:73:3d:43
      params:
        action: reboot
        ipaddr: 10.35.160.20
        lanplus: true
        login: admin
        passwd: admin
        pcmk_host_list: puma05
        privlvl: administrator
    - agent: fence_ipmilan
      host_mac: 44:1e:a1:73:36:6b
      params:
        action: reboot
        ipaddr: 10.35.160.78
        lanplus: true
        login: admin
        passwd: admin
        pcmk_host_list: puma34
        privlvl: administrator
    - agent: fence_ipmilan
      host_mac: 44:1e:a1:73:38:5f
      params:
        action: reboot
        ipaddr: 10.35.160.90
        lanplus: true
        login: admin
        passwd: admin
        pcmk_host_list: puma40
        privlvl: administrator
    - agent: fence_ipmilan
      host_mac: 44:1e:a1:71:a7:45
      params:
        action: reboot
        ipaddr: 10.35.160.38
        lanplus: true
        login: admin
        passwd: admin
        pcmk_host_list: puma14
        privlvl: administrator
    - agent: fence_ipmilan
      host_mac: 44:1e:a1:71:a6:37
      params:
        action: reboot
        ipaddr: 10.35.160.42
        lanplus: true
        login: admin
        passwd: admin
        pcmk_host_list: puma16
        privlvl: administrator

[stack@puma33 ~]$ 
[root@puma04 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: puma04 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
Last updated: Sun Apr 23 11:24:20 2017		Last change: Sun Apr 23 11:21:49 2017 by root via cibadmin on puma34

6 nodes and 37 resources configured

Online: [ puma04 puma05 puma34 ]
RemoteOnline: [ puma14 puma16 puma40 ]

Full list of resources:

 puma14	(ocf::pacemaker:remote):	Started puma04
 puma16	(ocf::pacemaker:remote):	Started puma05
 puma40	(ocf::pacemaker:remote):	Started puma34
 Master/Slave Set: galera-master [galera]
     Masters: [ puma14 puma16 puma40 ]
     Stopped: [ puma04 puma05 puma34 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ puma34 ]
     Slaves: [ puma04 puma05 ]
     Stopped: [ puma14 puma16 puma40 ]
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ puma04 puma05 puma34 ]
     Stopped: [ puma14 puma16 puma40 ]
 ip-192.168.24.11	(ocf::heartbeat:IPaddr2):	Started puma04
 ip-10.35.180.17	(ocf::heartbeat:IPaddr2):	Started puma05
 ip-172.17.0.17	(ocf::heartbeat:IPaddr2):	Started puma34
 ip-172.17.0.16	(ocf::heartbeat:IPaddr2):	Started puma04
 ip-172.18.0.11	(ocf::heartbeat:IPaddr2):	Started puma05
 ip-172.19.0.17	(ocf::heartbeat:IPaddr2):	Started puma34
 Clone Set: haproxy-clone [haproxy]
     Started: [ puma04 puma05 puma34 ]
     Stopped: [ puma14 puma16 puma40 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started puma04
 stonith-fence_ipmilan-441ea173366b	(stonith:fence_ipmilan):	Started puma05
 stonith-fence_ipmilan-441ea1733d43	(stonith:fence_ipmilan):	Started puma34
 stonith-fence_ipmilan-441ea1733991	(stonith:fence_ipmilan):	Started puma05

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@puma04 ~]# pcs stonith show stonith-fence_ipmilan-441ea173366b
 Resource: stonith-fence_ipmilan-441ea173366b (class=stonith type=fence_ipmilan)
  Attributes: pcmk_host_list=puma34 ipaddr=10.35.160.78 passwd=admin lanplus=true login=admin action=reboot privlvl=administrator
  Operations: monitor interval=60s (stonith-fence_ipmilan-441ea173366b-monitor-interval-60s)
[root@puma04 ~]# pcs stonith show stonith-fence_ipmilan-441ea1733d43
 Resource: stonith-fence_ipmilan-441ea1733d43 (class=stonith type=fence_ipmilan)
  Attributes: pcmk_host_list=puma05 ipaddr=10.35.160.20 passwd=admin lanplus=true login=admin action=reboot privlvl=administrator
  Operations: monitor interval=60s (stonith-fence_ipmilan-441ea1733d43-monitor-interval-60s)
[root@puma04 ~]# pcs stonith show stonith-fence_ipmilan-441ea1733991
 Resource: stonith-fence_ipmilan-441ea1733991 (class=stonith type=fence_ipmilan)
  Attributes: pcmk_host_list=puma04 ipaddr=10.35.160.18 passwd=admin lanplus=true login=admin action=reboot privlvl=administrator
  Operations: monitor interval=60s (stonith-fence_ipmilan-441ea1733991-monitor-interval-60s)
[root@puma04 ~]# 

roles_data.yaml 
- name: Galera
  HostnameFormatDefault: '%stackname%-galera-%index%'
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::PacemakerRemote
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient


Steps to Reproduce:
1. deploy composable roles environment and assign the pacemaker_remote to a role
2. use openstack overcloud generate fencing command and verify all hosts with pacemaker/pacemaker_remote are present.
3. $DEPLOY -e generate_fencing_output.yml
4. log into one of the controllers and check theat the installation confgured fencing to each node which have pacemaker/pacemaker_remote it is role.

Actual results:
autofencing configured fencing only to hosts with OS::TripleO::Services::Pacemaker


Expected results:
autofencing should be configured also to nodes with OS::TripleO::Services::PacemakerRemote

Comment 8 Chris Jones 2018-01-22 21:11:46 UTC
Adding doc test.

@Udi - could we get this verified in time for the z-stream release?

Comment 9 Udi Shkalim 2018-01-24 16:42:20 UTC
Verified:

[root@controller-2 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-2 (version 1.1.16-12.el7_4.5-94ff4df) - partition with quorum
Last updated: Wed Jan 24 16:38:23 2018
Last change: Wed Jan 24 15:06:44 2018 by root via cibadmin on controller-0

6 nodes configured
40 resources configured

Online: [ controller-0 controller-1 controller-2 ]
RemoteOnline: [ database-0 database-1 database-2 ]

Full list of resources:

 database-0	(ocf::pacemaker:remote):	Started controller-0
 database-1	(ocf::pacemaker:remote):	Started controller-1
 database-2	(ocf::pacemaker:remote):	Started controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ controller-0 controller-1 controller-2 ]
     Stopped: [ database-0 database-1 database-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ database-0 database-1 database-2 ]
     Stopped: [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ controller-2 ]
     Slaves: [ controller-0 controller-1 ]
     Stopped: [ database-0 database-1 database-2 ]
 ip-192.168.24.13	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-10.0.0.101	(ocf::heartbeat:IPaddr2):	Started controller-1
 ip-172.17.1.12	(ocf::heartbeat:IPaddr2):	Started controller-2
 ip-172.17.1.17	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-172.17.3.15	(ocf::heartbeat:IPaddr2):	Started controller-1
 ip-172.17.4.14	(ocf::heartbeat:IPaddr2):	Started controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ controller-0 controller-1 controller-2 ]
     Stopped: [ database-0 database-1 database-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started controller-0
 stonith-fence_ipmilan-525400f4f9b9	(stonith:fence_ipmilan):	Started controller-1
 stonith-fence_ipmilan-5254007d21be	(stonith:fence_ipmilan):	Started controller-2
 stonith-fence_ipmilan-52540018d15d	(stonith:fence_ipmilan):	Started controller-0
 stonith-fence_ipmilan-52540077902c	(stonith:fence_ipmilan):	Started controller-1
 stonith-fence_ipmilan-52540035e1e8	(stonith:fence_ipmilan):	Started controller-2
 stonith-fence_ipmilan-525400155a79	(stonith:fence_ipmilan):	Started controller-1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Comment 12 errata-xmlrpc 2018-02-13 16:31:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0310


Note You need to log in before you can comment on or make changes to this bug.