Bug 1303803 - rabbitmq resource-agents wipes too much data on recovery and kills additional users configuration [NEEDINFO]
rabbitmq resource-agents wipes too much data on recovery and kills additional...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: resource-agents (Show other bugs)
7.0
x86_64 Linux
high Severity high
: pre-dev-freeze
: ---
Assigned To: Peter Lemenkov
Leonid Natapov
https://github.com/lemenkov/resource-...
: ZStream
: 1297859 (view as bug list)
Depends On:
Blocks: 1316633
  Show dependency treegraph
 
Reported: 2016-02-01 21:46 EST by Chris Lincoln
Modified: 2016-11-03 20:01 EDT (History)
17 users (show)

See Also:
Fixed In Version: resource-agents-3.9.5-65.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1316633 (view as bug list)
Environment:
Last Closed: 2016-11-03 20:01:28 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
mfuruta: needinfo? (plemenko)


Attachments (Terms of Use)
Initial patch (1.56 KB, patch)
2016-03-01 05:47 EST, Peter Lemenkov
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2374351 None None None 2016-10-14 04:37 EDT

  None (edit)
Description Chris Lincoln 2016-02-01 21:46:18 EST
Description of problem:
Ability to create RabbitMQ users that sustain across resource restarts. Justification is for monitoring capabilities that don't use the RabbitMQ admin user, and not using the guest account with its randomized password (i.e., create a specific monitor user with a specific password).
Comment 2 John Eckersberg 2016-02-10 13:17:41 EST
This should be a pretty simple fix if I understand what the problem
is.  What happens currently in the resource agent is roughly:

- Start node

- If it's the only one up so far, then do nothing and wait for other
  nodes.  If at least one other node is already up, join cluster with
  the other node.

- If the cluster join fails, then blow away mnesia (literally rm -rf
  /var/lib/rabbitmq/mnesia/*) and try again.  Unfortunately mnesia is
  also where the user config is stored, so this is where the users get
  lost.

So, we just need to be less heavy-handed when attempting to right the
cluster.  Some combination of the rabbitmqctl commands
forget_cluster_node [--offline] / update_cluster_nodes / force_boot
should let us patch things up without completely removing mnesia and
all of the users with it.
Comment 3 Peter Lemenkov 2016-02-23 09:59:43 EST
Indeed we shouldn't remove so much data while restarting.
Comment 4 Peter Lemenkov 2016-02-24 11:36:57 EST
Ok, we almost there. Thanks to the recently built resource-agents-3.9.5-54.el7_2.7.x86_64 it's much easy to fix the cluster now. No need to wipe out Mnesia entirely.

I'm going to propose a fix soon.
Comment 5 Peter Lemenkov 2016-02-29 06:19:34 EST
Hello All!
I've got a patch which fixes that. I'm going to test it a little and provide a test build in a 4-6 hours.

This patch restores only "users" (rabbit_user) table. Other tables will be purged.
Comment 8 Peter Lemenkov 2016-03-01 05:47 EST
Created attachment 1131859 [details]
Initial patch

A proposed solution
Comment 17 Leonid Natapov 2016-03-16 09:22:50 EDT
resource-agents-3.9.5-67.el7

tested according comment #7. User still exist after resource restart.
Comment 18 Peter Lemenkov 2016-04-11 05:03:50 EDT
*** Bug 1297859 has been marked as a duplicate of this bug. ***
Comment 19 adrianlzt 2016-05-10 11:38:33 EDT
We use to add a monitoring user like:

rabbitmqctl add_user nagios PASSWORD
rabbitmqctl set_user_tags nagios monitoring
rabbitmqctl set_permissions nagios ".*" ".*" ".*"

Every time RabbitMQ is restarted, permissions are lost for this user.

PR to backup also this perms:
https://github.com/lemenkov/resource-agents/pull/1
Comment 20 Peter Lemenkov 2016-05-18 08:59:42 EDT
(In reply to adrianlzt from comment #19)
> We use to add a monitoring user like:
> 
> rabbitmqctl add_user nagios PASSWORD
> rabbitmqctl set_user_tags nagios monitoring
> rabbitmqctl set_permissions nagios ".*" ".*" ".*"
> 
> Every time RabbitMQ is restarted, permissions are lost for this user.
> 
> PR to backup also this perms:
> https://github.com/lemenkov/resource-agents/pull/1

Thanks for the addition! I'll merge it shortly.
Comment 22 xiaoxwan 2016-08-31 05:21:57 EDT
(In reply to Peter Lemenkov from comment #20)
> (In reply to adrianlzt from comment #19)
> > We use to add a monitoring user like:
> > 
> > rabbitmqctl add_user nagios PASSWORD
> > rabbitmqctl set_user_tags nagios monitoring
> > rabbitmqctl set_permissions nagios ".*" ".*" ".*"
> > 
> > Every time RabbitMQ is restarted, permissions are lost for this user.
> > 
> > PR to backup also this perms:
> > https://github.com/lemenkov/resource-agents/pull/1
> 
> Thanks for the addition! I'll merge it shortly.

I didn't saw the code about userperm in: 
https://github.com/ClusterLabs/resource-agents/master . 
Did we solved this problem there?
Comment 23 Peter Lemenkov 2016-08-31 11:44:29 EDT
(In reply to xiaoxwan from comment #22)
 
> I didn't saw the code about userperm in: 
> https://github.com/ClusterLabs/resource-agents/master . 
> Did we solved this problem there?

No not yet.
Comment 26 errata-xmlrpc 2016-11-03 20:01:28 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2174.html

Note You need to log in before you can comment on or make changes to this bug.