Bug 1289517 - HA | After upgrading HA dpeloyment from OSPD 7.1 to OSPD 7.2 stonith property becomes false instead of true,therefore fencing won't work.
HA | After upgrading HA dpeloyment from OSPD 7.1 to OSPD 7.2 stonith property...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity high
: y3
: 7.0 (Kilo)
Assigned To: Dan Macpherson
RHOS Documentation Team
: Documentation
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-08 06:39 EST by Leonid Natapov
Modified: 2016-04-07 11:12 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-07 11:12:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Leonid Natapov 2015-12-08 06:39:30 EST
Description of problem:

OSPD 7.1 HA deployment with fencing configured and enabled was upgraded to OSPD 7.2.

HA | After upgrading HA dpeloyment from OSPD 7.1 to OSPD 7.2 stonith property becomes false instead of true,therefore fencing won't work.

Here is hstory output before the upgrade:
-----------------------------------------
    5  pcs stonith create ipmilan-overcloud-controller-0  fence_ipmilan pcmk_host_list=overcloud-controller-0 ipaddr=10.35.160.78 login=admin passwd=admin lanplus=1 cipher=1 op monitor interval=60s;
    6  pcs constraint location ipmilan-overcloud-controller-0 avoids overcloud-controller-0;
    7  pcs stonith create ipmilan-overcloud-controller-1  fence_ipmilan pcmk_host_list=overcloud-controller-1 ipaddr=10.35.160.38 login=admin passwd=admin lanplus=1 cipher=1 op monitor interval=60s;
    8  pcs constraint location ipmilan-overcloud-controller-1 avoids overcloud-controller-1;
    9  pcs stonith create ipmilan-overcloud-controller-2  fence_ipmilan pcmk_host_list=overcloud-controller-2 ipaddr=10.35.160.18 login=admin passwd=admin lanplus=1 cipher=1 op monitor interval=60s;
   10  pcs constraint location ipmilan-overcloud-controller-2 avoids overcloud-controller-2;
   11  pcs property set stonith-enabled=true
----------------------------------------------

As you can see stonith-enable was set to true.


After upgrade:

[root@overcloud-controller-0 heat-admin]# pcs config | grep stonith
 Resource: ipmilan-overcloud-controller-0 (class=stonith type=fence_ipmilan)
 Resource: ipmilan-overcloud-controller-1 (class=stonith type=fence_ipmilan)
 Resource: ipmilan-overcloud-controller-2 (class=stonith type=fence_ipmilan)
 stonith-enabled: false


Work around:
Manually set to true stonith-enably property.
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Emilien Macchi 2015-12-14 09:25:34 EST
Leonid, I looked at the code and here is how it works:
If your environment contains EnableFencing set to True, the property will be automatically managed by Puppet, before and after the upgrade.

The EnableFencing parameter is set to False by default so if you don't define it in your environment I don't think you'll have it enabled.

My questions for you:

* Do you have EnableFencing set to True?
* Did you enable Fencing manually by running 'pcs' CLI?

If yes, Puppet will override the Fencing configuration by disabling it (since it's disabled by default).
If no, we have a bug and I'm interested to dig further.

Thanks!
Comment 3 Leonid Natapov 2015-12-14 09:33:52 EST
I enable fencing manually by running pcs command via CLI in ospd 7.1
adter updating the system to ospd 7.2 fencing was 'false'
Comment 4 Emilien Macchi 2015-12-14 10:11:28 EST
Leonid, that's a normal behavior: Puppet is managing whether we want or do not want Fencing enabled in Pacemaker by using EnableFencing parameter in TripleO Heat Templates.

So I don't think this is a bug, but rather something you should fix in our environment by setting EnableFencing to True so Puppet will not override it with False when you upgrade.
Comment 5 Ofer Blaut 2015-12-16 04:44:01 EST
Hi

Since we don't have auto fencing configuration in ospd 7 , we configure fencing manually post deployment, changing EnableFencing to True  is  not required by any document 

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/Director_Installation_and_Usage/index.html#sect-Advanced-Fencing_the_Controller_Nodes

We need to solve it for upgrade as part as upgrade process
Comment 7 Mike Burns 2016-01-27 10:30:37 EST
We need to note this is the update docs.
Comment 8 Dan Macpherson 2016-04-06 13:35:17 EDT
There's a note added to the upgrade section:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/Director_Installation_and_Usage/index.html#sect-Updating_the_Overcloud_Packages

==Important==
If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes:

$ sudo pcs property set stonith-enabled=true
===


mburns, is there anything else required for this BZ? Or can I close it?
Comment 9 Mike Burns 2016-04-07 06:25:36 EDT
Looks good to me

Note You need to log in before you can comment on or make changes to this bug.