RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1744467 - rabbitmq: users get deleted after restarting bundle
Summary: rabbitmq: users get deleted after restarting bundle
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: resource-agents
Version: 7.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 7.9
Assignee: Oyvind Albrigtsen
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1750352
TreeView+ depends on / blocked
 
Reported: 2019-08-22 09:08 UTC by Yadnesh Kulkarni
Modified: 2023-03-24 15:18 UTC (History)
9 users (show)

Fixed In Version: resource-agents-4.1.1-37.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1750352 (view as bug list)
Environment:
Last Closed: 2020-03-31 19:47:42 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ClusterLabs resource-agents issues 1392 0 'None' closed rabbitmq-cluster: also restore users/perms/policies when starting in single node mode 2021-01-07 00:36:53 UTC
Red Hat Knowledge Base (Solution) 4411111 0 None None None 2019-09-12 03:26:45 UTC
Red Hat Product Errata RHBA-2020:1067 0 None None None 2020-03-31 19:47:52 UTC

Description Yadnesh Kulkarni 2019-08-22 09:08:24 UTC 
Description of problem:
In a non-HA(single controller) environment, users get deleted after a restart of the rabbitmq bundle.

Steps to Reproduce:
[root@overcloudnmi-ctrl-0 rabbitmq]# docker ps | grep rabbit
9b78a85e19f3        192.x.x.1:8787/rhosp13/openstack-rabbitmq:pcmklatest          "/bin/bash /usr/lo..."   7 minutes ago       Up 7 minutes                                rabbitmq-bundle-docker-0

List users:
~~~
[root@overcloudnmi-ctrl-0 rabbitmq]# docker exec -it rabbitmq-bundle-docker-0 rabbitmqctl list_users
Listing users
guest   [administrator]
~~~

Create a user:
~~~
[root@overcloudnmi-ctrl-0 rabbitmq]# docker exec -it rabbitmq-bundle-docker-0 rabbitmqctl add_user appformix somepassword
Creating user "appformix"

[root@overcloudnmi-ctrl-0 rabbitmq]# docker exec -it rabbitmq-bundle-docker-0 rabbitmqctl set_user_tags appformix monitoring
Setting tags for user "appformix" to [monitoring]

[root@overcloudnmi-ctrl-0 rabbitmq]# docker exec -it rabbitmq-bundle-docker-0 rabbitmqctl set_permissions -p / appformix "" "" ".*"
Setting permissions for user "appformix" in vhost "/"
~~~

Before restarting rabbitmq-bundle:
~~~
[root@overcloudnmi-ctrl-0 rabbitmq]# docker exec -it rabbitmq-bundle-docker-0 rabbitmqctl list_users
Listing users
guest   [administrator]
appformix       [monitoring]
~~~

Restarting rabbitmq bundle
~~~
pcs resource restart  rabbitmq-bundle 
~~~

List users:
~~~
[root@overcloudnmi-ctrl-0 lib]# docker exec -it rabbitmq-bundle-docker-0 rabbitmqctl list_users
Listing users
guest   [administrator]
~~~

User `appformix` goes missing

Version-Release number of selected component (if applicable):
puppet-rabbitmq-8.1.1-0.20180216013831.d4b06b7.el7ost.noarch
rabbitmq-server-3.6.15-3.el7ost.noarch

Additional info:
Z-Stream release
~~~
cat /etc/rhosp-release 
Red Hat OpenStack Platform release 13.0.6 (Queens)
~~~

Adding users in an HA environment goes fine. The issue is with non-HA env
Comment 1 Michal Skalski 2019-08-22 09:24:06 UTC 
Just to add to what already was described:

The reason of this situation seems to be implementation of restoring users and their permissions: https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/rabbitmq-cluster#L440-L492
This part is not executed if the starting node is the first in the cluster: https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/rabbitmq-cluster#L409-L415 which in case of non-ha deployment will be always true.

Logs confirms that ocf scripts enters rmq_start_first function but never proceed further in rmq_try_start function:

[root@overcloudnmi-ctrl-0 log]# grep "Bootstrapping rabbitmq cluster" /var/log/messages
Aug 19 12:53:46 overcloudnmi-ctrl-0 rabbitmq-cluster(rabbitmq)[178]: INFO: Bootstrapping rabbitmq cluster
Aug 20 04:59:45 overcloudnmi-ctrl-0 rabbitmq-cluster(rabbitmq)[191]: INFO: Bootstrapping rabbitmq cluster
Aug 21 01:41:45 overcloudnmi-ctrl-0 rabbitmq-cluster(rabbitmq)[192]: INFO: Bootstrapping rabbitmq cluster
Aug 21 04:25:59 overcloudnmi-ctrl-0 rabbitmq-cluster(rabbitmq)[191]: INFO: Bootstrapping rabbitmq cluster
Aug 21 04:54:29 overcloudnmi-ctrl-0 rabbitmq-cluster(rabbitmq)[193]: INFO: Bootstrapping rabbitmq cluster
[root@overcloudnmi-ctrl-0 log]# grep "wiping data directory before joining" /var/log/messages
[root@overcloudnmi-ctrl-0 log]#
Comment 5 pkomarov 2019-12-31 13:55:20 UTC 
Verified , 
(undercloud) [stack@undercloud-0 ~]$ . stackrc 
(undercloud) [stack@undercloud-0 ~]$ rhos-release -L
Installed repositories (rhel-7.7):
  14
  ceph-3
  ceph-osd-3
  rhel-7.7
	

[root@controller-0 ~]# rpm -q resource-agents;docker exec `docker ps -f name=rabbitmq-bundle -q` sh -c "rpm -q resource-agents"
resource-agents-4.1.1-37.el7.x86_64
resource-agents-4.1.1-37.el7.x86_64


[root@controller-0 ~]# docker exec `docker ps -f name=rabbitmq-bundle -q` bash
^C
[root@controller-0 ~]# docker exec -it `docker ps -f name=rabbitmq-bundle -q` bash
()[root@controller-0 /]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@controller-0'
[{nodes,[{disc,['rabbit@controller-0']}]},
 {running_nodes,['rabbit@controller-0']},
 {cluster_name,<<"rabbit.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@controller-0',[]}]}]
()[root@controller-0 /]# rabbitmqctl list_users
Listing users
guest	[administrator]
()[root@controller-0 /]# rabbitmqctl add_user appformix somepassword
Creating user "appformix"
()[root@controller-0 /]# rabbitmqctl set_user_tags appformix monitoring
Setting tags for user "appformix" to [monitoring]
()[root@controller-0 /]# rabbitmqctl set_permissions -p / appformix "" "" ".*"
Setting permissions for user "appformix" in vhost "/"
()[root@controller-0 /]# #before rabbitmq restart 
()[root@controller-0 /]# rabbitmqctl list_users
Listing users
guest	[administrator]
appformix	[monitoring]
()[root@controller-0 /]# exit
[root@controller-0 ~]# #Restarting rabbitmq bundle

[root@controller-0 ~]# pcs resource restart  rabbitmq-bundle 
rabbitmq-bundle successfully restarted
[root@controller-0 ~]# docker ps |grep rabbit
f7051a70a40b        192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest                      "/bin/bash /usr/lo..."   42 seconds ago      Up 41 seconds                              rabbitmq-bundle-docker-0
[root@controller-0 ~]# docker exec -it `docker ps -f name=rabbitmq-bundle -q` bash
()[root@controller-0 /]#  rabbitmqctl list_users
Listing users
guest	[administrator]
appformix	[monitoring]
Comment 7 errata-xmlrpc 2020-03-31 19:47:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1067
Note You need to log in before you can comment on or make changes to this bug.