+++ This bug was initially created as a clone of Bug #1855885 +++

Description of problem:

The SAPHana resource agent uses the `system_replication/mode` attribute from global.ini as a fallback if the `$hdbState` command fails. The expectation is that a takeover event updates the `mode` parameter so that it's usually a valid representation of which node is currently primary.

However, `mode` is a static parameter that does not change with a takeover event. Instead, the takeover updates the `actual mode` parameter.

Our resource agent needs to be updated to query the correct parameter in the event that `$hdbState` fails. This way, we can respond more appropriately to edge-case situations like missing hdb* binaries.


Adapted from an SAP engineer in a support collaboration email:
~~~
# # Before takeover
# # node1 is primary, node2 is secondary
	
global.ini on node1:
  mode  = primary
  actual mode = primary
  operation_mode = logreplay

global.ini on node2:
  mode  = sync
  actual mode = sync
  operation_mode = logreplay

hdbnsutil -sr_state on node1:
  mode: primary
  operation mode: primary

hdbnsutil -sr_state on node2:
  mode: sync
  operation mode: logreplay


# # After takeover/failover
# # node1 is secondary, node2 is primary

global.ini on node1:
  mode  = primary
  actual mode = sync
  operation_mode = logreplay

global.ini on node2:
  mode  = sync
  actual mode = primary
  operation_mode = logreplay

hdbnsutil -sr_state on node1:
  mode: sync
  operation mode: logreplay

hdbnsutil -sr_state on node2:
  mode: primary
  operation mode: primary


Just have a look how we change the parameter values depending on the source – global.ini or hdbnsutil -sr_state. I highlighted major differences.

This behavior doesn’t depend on removing binaries, it’s normal HANA parameter change after the takeover/re-registering secondaries to new primary. Confirmation is provided in SAP Note 1999880 - FAQ: SAP HANA System Replication:

47. Why do I see deviating values in the system replication mode parameter?

The value in parameter global.ini -> [system_replication] -> mode depends on the original role of the site:

·         If site was originally configured as primary site: mode = 'primary'

·         If site was originally configured as secondary / tertiary site: mode = 'sync', 'async', ... (dependent on the system replication mode)

As a consequence the mode value can be different in two identically configured systems if a takeover happened in one system, but not in the other.
~~~

-----

Version-Release number of selected component (if applicable):

resource-agents-sap-hana-4.1.1-53.el7

-----

How reproducible:

Always

-----

Steps to Reproduce:

Assuming SAP's description of the parameters is correct, it's trivial to look at the check_for_primary() function and see that we're apparently querying the wrong one.

I believe the following steps will reproduce an issue occurring as a result of this:

1. Make the node with `mode = sync` the primary node via takeover.
2. Move the hdb* binaries to another location on that node so that the binaries are "missing."

-----

Actual results:

Both nodes end up in demoted state because the RA reads "mode = sync" from the global.ini file as a fallback on the primary.

-----

Expected results:

Pacemaker does not take any corrective action because the RA reads "actual mode = primary" from the global.ini file as a fallback.

-----

Additional info:

Related to closed BZ1783581.

--- Additional comment from Reid Wahl on 2020-07-10 20:02:01 UTC ---

I don't think this is an issue in the Scale Out RAs, but it wouldn't hurt to confirm.