Bug 1144100 - rabbitmq restarts fail randomly
Summary: rabbitmq restarts fail randomly
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: rabbitmq-server
Version: epel7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: John Eckersberg
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1059028
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-18 16:25 UTC by Ben Nemec
Modified: 2015-07-07 19:58 UTC (History)
15 users (show)

Fixed In Version: rabbitmq-server-3.3.5-4.el7
Doc Type: Bug Fix
Doc Text:
Clone Of: 1059028
Environment:
Last Closed: 2015-01-06 02:09:08 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Ben Nemec 2014-09-18 16:25:02 UTC
+++ This bug was initially created as a clone of Bug #1059028 +++

This still appears to be a problem in the EPEL version of rabbitmq-server.

Description of problem:

I originally encountered this issue when installing OpenStack via devstack but have since been able to reproduce it by simply executing commands inside of a fresh F20 install.

Version-Release number of selected component (if applicable):

rabbitmq-server-3.1.5-1.fc20.noarch

How reproducible:

Occurs regularly but not 100% of the time

Steps to Reproduce:

/sbin/service rabbitmq-server stop
/sbin/service rabbitmq-server start
rabbitmqctl change_password guest newpassword


Actual results:

About half the time, on a freshly installed F20, this will fail, claiming the node cannot be contacted.  A typical error message:

[root@cob-dell5 ~]# rabbitmqctl change_password guest ozrootpw
Changing password for user "guest" ...
Error: unable to connect to node 'rabbit@cob-dell5': nodedown

DIAGNOSTICS
===========

nodes in question: ['rabbit@cob-dell5']

hosts, their running nodes and ports:
- cob-dell5: [{rabbitmqctl2648,48609}]

current node details:
- node name: 'rabbitmqctl2648@cob-dell5'
- home dir: /var/lib/rabbitmq
- cookie hash: 8DNoVu56TqDYWypW7YXDJw==

Expected results:

Changing password for user "guest" ...
...done.

--- Additional comment from Ian McLeod on 2014-01-28 21:36:45 EST ---

See the following thread on rdo-list for another example of this issue:

https://www.redhat.com/archives/rdo-list/2013-December/msg00058.html

As well as this more recent one, indicating tha the problem persists:

https://www.redhat.com/archives/rdo-list/2014-January/msg00164.html

--- Additional comment from Ian McLeod on 2014-02-27 17:51:21 EST ---

Note to anyone following.  The devstack installer has a commit that attempts to workaround this issue by retrying the rabbitmq restart:

https://github.com/openstack-dev/devstack/commit/f6c001faf6ac5728e07c6bbdaa67680395ed6f48

Still, it would be nice to sort this out on the rabbitmq side eventually.

--- Additional comment from Fedora Admin XMLRPC Client on 2014-07-01 13:58:36 EDT ---

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

--- Additional comment from Fedora Update System on 2014-08-10 04:14:16 EDT ---

rabbitmq-server-3.1.5-9.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-9.fc20

--- Additional comment from lnie on 2014-08-11 03:00:36 EDT ---

rabbitmq-server-3.1.5-9.fc20 works

--- Additional comment from Fedora Update System on 2014-08-14 22:35:38 EDT ---

Package rabbitmq-server-3.1.5-9.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing rabbitmq-server-3.1.5-9.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-9337/rabbitmq-server-3.1.5-9.fc20
then log in and leave karma (feedback).

--- Additional comment from Fedora Update System on 2014-08-15 20:28:02 EDT ---

rabbitmq-server-3.1.5-9.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 1 Nathan G. Grennan 2014-12-05 20:19:47 UTC
I just ran into this issue. RHEL 7 Openstack has upgraded to 3.3.5-2. I recompiled the rabbitmq-server 3.3.5-2 fc22 src.rpm from koji, and it fixed my issue.

Comment 2 Ian Wienand 2014-12-15 23:10:46 UTC
This is being seen in upstream testing for openstack

---
2014-12-15 19:16:20.015 | ++ sudo rabbitmqctl list_users
2014-12-15 19:16:20.239 | Error: unable to connect to node 'rabbit@devstack-centos7-rax-iad-100675': nodedown
2014-12-15 19:16:20.249 | 
2014-12-15 19:16:20.249 | DIAGNOSTICS
2014-12-15 19:16:20.249 | ===========
2014-12-15 19:16:20.249 | 
2014-12-15 19:16:20.249 | nodes in question: ['rabbit@devstack-centos7-rax-iad-100675']
2014-12-15 19:16:20.250 | 
2014-12-15 19:16:20.250 | hosts, their running nodes and ports:
2014-12-15 19:16:20.250 | - devstack-centos7-rax-iad-100675: [{rabbitmqctl29293,39511}]
2014-12-15 19:16:20.250 | 
2014-12-15 19:16:20.250 | current node details:
2014-12-15 19:16:20.250 | - node name: 'rabbitmqctl29293@devstack-centos7-rax-iad-100675'
2014-12-15 19:16:20.250 | - home dir: /var/lib/rabbitmq
2014-12-15 19:16:20.250 | - cookie hash: KieJnx1pnllKbHVihGcDqA==
2014-12-15 19:16:20.250 | 
2014-12-15 19:16:20.256 | + out='Listing users ...'
2014-12-15 19:16:20.256 | + echo 'failed to list users'
---

This happens somewhat frequently on our centos 7 jobs (can search on [1] to see examples)

[1] http://logstash.openstack.org

Comment 3 Fedora Update System 2014-12-16 16:44:16 UTC
rabbitmq-server-3.3.5-3.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.3.5-3.el7

Comment 4 Fedora Update System 2014-12-17 05:36:09 UTC
Package rabbitmq-server-3.3.5-3.el7:
* should fix your issue,
* was pushed to the Fedora EPEL 7 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing rabbitmq-server-3.3.5-3.el7'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2014-4742/rabbitmq-server-3.3.5-3.el7
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2014-12-17 19:37:51 UTC
rabbitmq-server-3.3.5-4.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/rabbitmq-server-3.3.5-4.el7

Comment 6 Fedora Update System 2015-01-06 02:09:08 UTC
rabbitmq-server-3.3.5-4.el7 has been pushed to the Fedora EPEL 7 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 David Hill 2015-07-07 19:58:17 UTC
This created bugs in Ubuntu Vivid.

See : 
https://review.openstack.org/#/c/191090/


Note You need to log in before you can comment on or make changes to this bug.