Bug 2114013

Can you share your full cluster configuration? An sosreport would be ideal if possible, so that we can also see the log files. (Or you could attach logs manually if you'd prefer.)

---

See also this mailing list thread started by the reporter:
  - [ClusterLabs] resource agent OCF_HEARTBEAT_GALERA issue/broken - ? (https://lists.clusterlabs.org/pipermail/users/2022-July/030433.html)

> First issue I think is:
> - 'config' attr does not do much/anything? because if 'datadir' attr is not specified then resource/agent wants to start Mariadb from '/var/lib/mysql' - and !..
> Galera's '/apps/etc/mariadb-server.cnf' does contain those bits anyway.

As discussed in that thread, the config option specifies the `--defaults-file` option that we use in the mysql start command. Another CLI option may override what's in the file, I presume.

However, it's not obvious without looking at the script that if datadir is not specified, the resource agent explicitly passes --datadir=/var/lib/mysql.

I made a suggestion on the thread:

> It would be reasonable at least to add a note to the resource agent
> metadata, to say something like
> 
>   datadir: Directory containing databases. If this option is not
> specified, then --datadir=/var/lib/mysql will be used when starting
> the database.
> 
> instead of the current
> 
>   datadir: Directory containing databases
```

in logs outside of pcs cluster:
...
ERROR: Could not determine initial best node from galera name <10.0.1.7:5567>.
ERROR: MySQL is not running
...

but again, this galera cluster starts perfectly fine outside of pcs cluster control.

Can you share your full cluster configuration? An sosreport would be ideal if possible, so that we can also see the log files. (Or you could attach logs manually if you'd prefer.)

In an attempt to:
-> $ pcs resource debug-promote mariadb-apps --full

I do not see 'mysql_common_start' there.

Can you share your full cluster configuration? An sosreport would be ideal if possible, so that we can also see the log files. (Or you could attach logs manually if you'd prefer.)

Sorry, cannot just yet.
Also it's - as is the thread on list I started - against CentOS 8, should be, not RHEL, my bad. Feel free to change that.

And to cover all bits, here is the resource which works perfectly fine:
-> $ pcs resource config mariadb-clone
Clone: mariadb-clone
  Meta Attributes: mariadb-clone-meta_attributes
    promotable=true
    promoted-max=2
  Resource: mariadb (class=ocf provider=heartbeat type=galera)
    Attributes: mariadb-instance_attributes
      check_passwd=pacemaker#98
      check_user=pacemaker
      cluster_host_map=drunk.internal.ccn:10.0.1.6;sucker.internal.ccn:10.0.1.7
      group=mysql
      user=mysql
      wsrep_cluster_address=gcomm://10.0.1.6,10.0.1.7
    Meta Attributes: mariadb-meta_attributes
      failure-timeout=30s
    Operations:
...
So.. very same single pcs cluster, one galera works while the other does not.

Possibly interesting is:
When I change to this:
      wsrep_cluster_address=gcomm://drunk.internal.ccn:5567,sucker.internal.ccn:5567
and accordingly in '/apps/etc/mariadb-server.cnf'
then I get rid of 'cluster_host_map' as a result 'ERROR: Could not determine initial best node from galera name' disappears but..
...
ERROR: MySQL is not running
INFO: Waiting on node <drunk.internal.ccn:5567> to report database status before Master instances can start.
INFO: Waiting on node <sucker.internal.ccn:5567> to report database status before Master instances can start.
...

I'm quite certain I took care of SELinux - non-standard ports & path - even tried permissive/disabled.
Now when I look at it - do you really think 'global' config has anything to do with it - having one galera resource work but not the other?
Also if given then only difference between the two - in theory - are different IP ports & paths (thus extra attrs in second resource)

aha!!! this little thing seems to brake everything/resource/agent, here:

wsrep_cluster_address="gcomm://10.0.1.6:5567,10.0.1.7:5567" -> brakes down
VS
wsrep_cluster_address="gcomm://10.0.1.6,10.0.1.7" -> works okey

is that expected behaviour?
You mention docs/man which you say should be better - I say: with code/bins which have poor documentation or none at all - if such bins are meant for public-user use - we can wipe our arses with. Even if such code/bin is the smartest one.

'wsrep_cluster_address' - another thing - seems it's mandatory! and if is - also no mention of that in man page.

(In reply to lejeczek from comment #7)
> aha!!! this little thing seems to brake everything/resource/agent, here:
> 
> wsrep_cluster_address="gcomm://10.0.1.6:5567,10.0.1.7:5567" -> brakes down
> VS
> wsrep_cluster_address="gcomm://10.0.1.6,10.0.1.7" -> works okey
> 
> is that expected behaviour?

It may be improvable; I haven't gotten that far and will probably leave it to the resource-agents maintainer to decide. However, it makes sense that the current version of the agent can't properly parse the ports from the wsrep_cluster_address:
```
galera_to_pcmk_name()
{
    local galera=$1
    if [ -z "$OCF_RESKEY_cluster_host_map" ]; then
        echo $galera
    else
        echo "$OCF_RESKEY_cluster_host_map" | tr ';' '\n' | tr -d ' ' | sed 's/:/ /' | awk -F' ' '$2=="'"$galera"'" {print $1;exit}'
    fi
}
...
detect_first_master()
{
    ...
    all_nodes=$(echo "$OCF_RESKEY_wsrep_cluster_address" | sed 's/gcomm:\/\///g' | tr -d ' ' | tr -s ',' ' ')
    best_node_gcomm=$(echo "$all_nodes" | sed 's/^.* \(.*\)$/\1/')
    best_node=$(galera_to_pcmk_name $best_node_gcomm)
    if [ -z "$best_node" ]; then
        ocf_log err "Could not determine initial best node from galera name <${best_node_gcomm}>."
        return
    fi
...
}
```

> You mention docs/man which you say should be better - I say: with code/bins
> which have poor documentation or none at all - if such bins are meant for
> public-user use - we can wipe our arses with. Even if such code/bin is the
> smartest one.
> 
> 'wsrep_cluster_address' - another thing - seems it's mandatory! and if is -
> also no mention of that in man page.

It's listed as required, and it's listed as "gcomm://node,node,node" -- not "gcomm://node[:port],node[:port],node[:port]". The documentation also says that the wsrep_cluster_addresses "are expected to match valid pacemaker node names. If both names need to differ, you must provide a mapping in option cluster_host_map." In this case, the mapping didn't match the wsrep_cluster_adddress due to the ports appendd in the address.
```
[root@fastvm-rhel-8-0-23 ~]# rpm -q resource-agents
resource-agents-4.9.0-16.el8.x86_64
[root@fastvm-rhel-8-0-23 ~]# pcs resource describe ocf:heartbeat:galera
...
Resource options:
  ...
  wsrep_cluster_address (required): The galera cluster address. This takes the form of: gcomm://node,node,node Only nodes present in this node list will be allowed to start a galera instance. The galera node
                                    names listed in this address are expected to match valid pacemaker node names. If both names need to differ, you must provide a mapping in option cluster_host_map.

```

in configs mariadb/galera allow ports to be skipped - code set default ones - I guarantee that people coming to this(every other too) resource will assume that ports can be either: there or absent - just like I did.
Docs could cover it better, leaving no doubts.

The issue is back, in my comment #7 I wrote:

wsrep_cluster_address="gcomm://10.0.1.6,10.0.1.7" -> works okey

but it does not, a few rpm updates later, any more.

How that second resource "instance" fails now is with:

  * mariadb-apps_promote_0 on sucker.internal.ccn 'error' (1): call=1085, status='complete', exitreason='Failed initial monitor action', last-rc-change='Mon Jan  2 18:13:10 2023', queued=0ms, exec=9618ms

Everything, resource & mariadb configs remain the same as I showed them above.
Also ! again, when is Galera ("second/non-regular" resource) is started the same moment but outside of pcs/HA, with cmds as I showed above, then that Galera cluster runs just fine. 

What I noticed also - which I think might point to where the culprit is - is that when I start Galera first/regular (which was disabled) resource, then the second resource "amazingly" starts and sees no monitor problems, as soon as that first Galera resource elected/promoted Galara cluster. 

As soon as I disable that first Galara resource - when both resources worked fine - then pcs/HA stops/fails the second resource with the same:

  * mariadb-apps_promote_0 on sucker.internal.ccn 'error' (1): call=1207, status='complete', exitreason='Failed initial monitor action', last-rc-change='Mon Jan  2 18:24:30 2023', queued=0ms, exec=7555ms

"first/regular" resource:
-> $ pcs resource config mariadb
Resource: mariadb (class=ocf provider=heartbeat type=galera)
  Attributes: mariadb-instance_attributes
    check_passwd=pacemaker#98
    check_user=pacemaker
    cluster_host_map=drunk.internal.ccn:10.0.1.6;sucker.internal.ccn:10.0.1.7
    group=mysql
    user=mysql
    wsrep_cluster_address=gcomm://10.0.1.6,10.0.1.7
  Meta Attributes: mariadb-meta_attributes
    failure-timeout=30s
  Operations:
    demote: mariadb-demote-interval-0s
      interval=0s
      timeout=120s
    monitor: mariadb-monitor-interval-20s
      interval=20s
      timeout=30s
      OCF_CHECK_LEVEL=0
    monitor: mariadb-monitor-interval-10s
      interval=10s
      timeout=30s
      role=Master
      OCF_CHECK_LEVEL=0
    monitor: mariadb-monitor-interval-30s
      interval=30s
      timeout=30s
      role=Slave
      OCF_CHECK_LEVEL=0
    promote: mariadb-promote-interval-0s
      interval=0s
      timeout=300s
    start: mariadb-start-interval-0s
      interval=0s
      timeout=120s
    stop: mariadb-stop-interval-0s
      interval=0s
      timeout=120s

This "first/regular" resource is not affected, does not fail, if/when the "second/non-regular" resource is stopped/disabled.

many thanks, L.