This should be a pretty simple fix if I understand what the problem
is.  What happens currently in the resource agent is roughly:

- Start node

- If it's the only one up so far, then do nothing and wait for other
  nodes.  If at least one other node is already up, join cluster with
  the other node.

- If the cluster join fails, then blow away mnesia (literally rm -rf
  /var/lib/rabbitmq/mnesia/*) and try again.  Unfortunately mnesia is
  also where the user config is stored, so this is where the users get
  lost.

So, we just need to be less heavy-handed when attempting to right the
cluster.  Some combination of the rabbitmqctl commands
forget_cluster_node [--offline] / update_cluster_nodes / force_boot
should let us patch things up without completely removing mnesia and
all of the users with it.

Indeed we shouldn't remove so much data while restarting.

Ok, we almost there. Thanks to the recently built resource-agents-3.9.5-54.el7_2.7.x86_64 it's much easy to fix the cluster now. No need to wipe out Mnesia entirely.

I'm going to propose a fix soon.

Hello All!
I've got a patch which fixes that. I'm going to test it a little and provide a test build in a 4-6 hours.

This patch restores only "users" (rabbit_user) table. Other tables will be purged.

Created attachment 1131859 [details]
Initial patch

A proposed solution

resource-agents-3.9.5-67.el7

tested according comment #7. User still exist after resource restart.

*** Bug 1297859 has been marked as a duplicate of this bug. ***

We use to add a monitoring user like:

rabbitmqctl add_user nagios PASSWORD
rabbitmqctl set_user_tags nagios monitoring
rabbitmqctl set_permissions nagios ".*" ".*" ".*"

Every time RabbitMQ is restarted, permissions are lost for this user.

PR to backup also this perms:
https://github.com/lemenkov/resource-agents/pull/1

(In reply to adrianlzt from comment #19)
> We use to add a monitoring user like:
> 
> rabbitmqctl add_user nagios PASSWORD
> rabbitmqctl set_user_tags nagios monitoring
> rabbitmqctl set_permissions nagios ".*" ".*" ".*"
> 
> Every time RabbitMQ is restarted, permissions are lost for this user.
> 
> PR to backup also this perms:
> https://github.com/lemenkov/resource-agents/pull/1

Thanks for the addition! I'll merge it shortly.

(In reply to Peter Lemenkov from comment #20)
> (In reply to adrianlzt from comment #19)
> > We use to add a monitoring user like:
> > 
> > rabbitmqctl add_user nagios PASSWORD
> > rabbitmqctl set_user_tags nagios monitoring
> > rabbitmqctl set_permissions nagios ".*" ".*" ".*"
> > 
> > Every time RabbitMQ is restarted, permissions are lost for this user.
> > 
> > PR to backup also this perms:
> > https://github.com/lemenkov/resource-agents/pull/1
> 
> Thanks for the addition! I'll merge it shortly.

I didn't saw the code about userperm in: 
https://github.com/ClusterLabs/resource-agents/master . 
Did we solved this problem there?

(In reply to xiaoxwan from comment #22)
 
> I didn't saw the code about userperm in: 
> https://github.com/ClusterLabs/resource-agents/master . 
> Did we solved this problem there?

No not yet.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2174.html

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days