Bug 1435982

Thanks to Damien for his big help on this!

So the reason the cluster is not forming properly on the remote node is that it never gets the list of all the nodes. The culprit is this line in the resource agent:
rmq_join_list()
{
    cibadmin -Q --xpath "//node_state[@crmd='online']//nvpair[@name='$RMQ_CRM_ATTR_COOKIE']" | grep "$RMQ_CRM_ATTR_COOKIE" | sed -n -e "s/^.*value=.\(.*\)\".*$/\1/p"
}

The above is supposed to collect all the nodes which wrote their name in attrd. The problem is that on remote nodes "crmd=online" is never true so the list is empty and so each node just starts on its own.

By changing the function to something like the following:
rmq_join_list()
{
        local join_list=$(cibadmin -Q --xpath "//node_state[@crmd='online']//nvpair[@name='$RMQ_CRM_ATTR_COOKIE']" | grep "$RMQ_CRM_ATTR_COOKIE" | sed -n -e "s/^.*value=.\(.*\)\".*$/\1/p")
        # If join_list is empty we want to check if there are any remote nodes
        # where rabbitmq is allowed to run (i.e. nodes without the crmd=online
        if [ -z "$join_list" ]; then
                # Get all the nodes written in the ATTR_COOKIE no matter if
                # they are online or not
                local remote_join_list=$(cibadmin -Q --xpath "//node_state//nvpair[@name='$RMQ_CRM_ATTR_COOKIE']" | grep "$RMQ_CRM_ATTR_COOKIE" | sed -n -e "s/^.*value=.\(.*\)\".*$/\1/p")
                # here we basically need to filter the nodes found above
                # and only use the nodes that have online=true and standby=false
                # basically we are mimicking crmd=online but for remotes
                local filter=$(crm_mon -r --as-xml | xmllint --format --xpath "//nodes//node[@online='true' and @standby='false']/@name" - | xargs -n1 echo | awk -F= '/rabbit/ {print "-e "
$2}')
                join_list="$(echo $remote_join_list | grep $filter)"
        fi
        echo $join_list
}


We were able to get a working cluster. Maybe we can get something a bit simpler than the above. Damien and I tried to mimick the crmd=online test for remotes. We did it because it was explicitly added for normal nodes via https://github.com/ClusterLabs/resource-agents/commit/ddb8e97e2860a3d375e9a422d48a0b9e0830416d 

Fabio, to your question: yes this is a rabbitmq resource agent issue on pacemaker remote only.

Am reassigning to the RA component

Link to patch: https://github.com/ClusterLabs/resource-agents/pull/959

New build with /rabbit/ removed from the awk-line.

Created attachment 1282298 [details]
undercloud sosreport

Created attachment 1282299 [details]
messaging sosreports

Overcloud failed:

Error: unable to push cib\nCall cib_apply_diff failed (-205): Update was older than existing configuration\n\n\u001b[1;31mError: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20170525-14352-79425q create failed: Error: unable to create resource/fence device 'messaging-0', 'messaging-0' already exists on this system\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Pacemaker/Pacemaker::Resource::Remote[messaging-0]/Pcmk_resource[messaging-0]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20170525-14352-79425q create failed: Error: unable to create resource/fence device 'messaging-0', 'messaging-0' already exists on this system\u001b[0m\n",

sosreports attached.

I am switching to VERIFIED.
First of all we have the patch in 7.3.z so we need to include that in 7.4 anyway + the problem is that the race (Bug 1448646) prevents to verify this bug easily but I applied the patch from that bug and was able to verify this bug which was already verified on 7.3.z.

Used build:

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1844