This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1300728 - rabbitmq report show Error: unable to connect to the local node
rabbitmq report show Error: unable to connect to the local node
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rabbitmq-server (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity high
: async
: 8.0 (Liberty)
Assigned To: Peter Lemenkov
Asaf Hirshberg
: Triaged, ZStream
Depends On:
Blocks: 1356169
  Show dependency treegraph
 
Reported: 2016-01-21 09:49 EST by Asaf Hirshberg
Modified: 2016-12-21 11:42 EST (History)
6 users (show)

See Also:
Fixed In Version: rabbitmq-server-3.3.5-28.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-21 11:42:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rabbitmq log controller0 (1.66 MB, text/plain)
2016-01-21 09:49 EST, Asaf Hirshberg
no flags Details
rabbitmq log controller1 (2.11 MB, text/plain)
2016-01-21 09:49 EST, Asaf Hirshberg
no flags Details
rabbitmq log controller2 (1.16 MB, text/plain)
2016-01-21 09:50 EST, Asaf Hirshberg
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1356169 None None None 2016-10-04 12:01 EDT
Github rabbitmq/rabbitmq-server/pull/894 None None None 2016-11-01 12:23 EDT

  None (edit)
Description Asaf Hirshberg 2016-01-21 09:49:06 EST
Created attachment 1116955 [details]
rabbitmq log controller0

Description of problem:
rabbitmqctl cluster_status" show all nodes in the cluster as connected but 
"rabbitmqctl report" command showing error that it cant connect to the local node on each node of the cluster. 
[root@overcloud-controller-0 ~]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@overcloud-controller-0' ...
[{nodes,[{disc,['rabbit@overcloud-controller-0',
                'rabbit@overcloud-controller-1',
                'rabbit@overcloud-controller-2']}]},
 {running_nodes,['rabbit@overcloud-controller-2',
                 'rabbit@overcloud-controller-1',
                 'rabbit@overcloud-controller-0']},
 {cluster_name,<<"rabbit@overcloud-controller-0">>},
 {partitions,[]}]
...done.

[root@overcloud-controller-0 ~]# rabbitmqctl report 
Reporting server status on {{2016,1,21},{13,29,54}}

 ...
Status of node 'rabbit@overcloud-controller-2' ...
Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@overcloud-controller-0']

rabbit@overcloud-controller-0:
  * connected to epmd (port 4369) on overcloud-controller-0
  * node rabbit@overcloud-controller-0 up, 'rabbit' application running

current node details:
- node name: 'rabbitmqctl14611@overcloud-controller-0'
- home dir: /var/lib/rabbitmq
- cookie hash: EdBLcRP4/rUPcYrgeGx7RA==

[root@overcloud-controller-0 ~]# rabbitmqctl report 
Reporting server status on {{2016,1,21},{13,32,56}}

 ...
Status of node 'rabbit@overcloud-controller-2' ...
Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@overcloud-controller-0']

rabbit@overcloud-controller-0:
  * connected to epmd (port 4369) on overcloud-controller-0
  * node rabbit@overcloud-controller-0 up, 'rabbit' application running

current node details:
- node name: 'rabbitmqctl22338@overcloud-controller-0'
- home dir: /var/lib/rabbitmq
- cookie hash: EdBLcRP4/rUPcYrgeGx7RA==

[root@overcloud-controller-0 ~]# 


[root@overcloud-controller-1 ~]# rabbitmqctl report
Reporting server status on {{2016,1,21},{13,30,26}}

 ...
Status of node 'rabbit@overcloud-controller-2' ...
Error: unable to connect to node 'rabbit@overcloud-controller-1': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@overcloud-controller-1']

rabbit@overcloud-controller-1:
  * connected to epmd (port 4369) on overcloud-controller-1
  * node rabbit@overcloud-controller-1 up, 'rabbit' application running

current node details:
- node name: 'rabbitmqctl14349@overcloud-controller-1'
- home dir: /var/lib/rabbitmq
- cookie hash: EdBLcRP4/rUPcYrgeGx7RA==

[root@overcloud-controller-1 ~]# 

[root@overcloud-controller-2 ~]# rabbitmqctl report
Reporting server status on {{2016,1,21},{13,33,31}}

 ...
Status of node 'rabbit@overcloud-controller-0' ...
Error: unable to connect to node 'rabbit@overcloud-controller-2': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@overcloud-controller-2']

rabbit@overcloud-controller-2:
  * connected to epmd (port 4369) on overcloud-controller-2
  * node rabbit@overcloud-controller-2 up, 'rabbit' application running

current node details:
- node name: 'rabbitmqctl14888@overcloud-controller-2'
- home dir: /var/lib/rabbitmq
- cookie hash: EdBLcRP4/rUPcYrgeGx7RA==

[root@overcloud-controller-2 ~]#


How reproducible:
10/10


Additional info:
Comment 1 Asaf Hirshberg 2016-01-21 09:49 EST
Created attachment 1116957 [details]
rabbitmq log  controller1
Comment 2 Asaf Hirshberg 2016-01-21 09:50 EST
Created attachment 1116958 [details]
rabbitmq log controller2
Comment 3 Asaf Hirshberg 2016-01-21 11:28:42 EST
From what I have test, it doesn't seems to affect the openstack services, I created new instances, uploaded new images and created networks without interruptions.
Comment 6 Peter Lemenkov 2016-10-04 12:01:56 EDT
We've finally know what's going on there. A similar issue for the recent version (3.6.x) was fixed in bug 1356169.
Comment 8 Udi Shkalim 2016-12-11 11:40:35 EST
Verified on: rabbitmq-server-3.3.5-29.el7ost.noarch

#rabbitmqctl report
Reporting server status on {{2016,12,11},{16,39,48}}

 ...
Status of node 'rabbit@controller-2' ...
[{pid,2901},
 {running_applications,[{rabbit,"RabbitMQ","3.3.5"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {xmerl,"XML parser","1.3.6"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,[{total,121367160},
          {connection_procs,3898552},
          {queue_procs,4603760},
          {plugins,0},
          {other_proc,13752488},
          {mnesia,1168664},
          {mgmt_db,0},
          {msg_index,185184},
          {other_ets,1285968},
          {binary,69706256},
          {code,16700794},
          {atom,654217},
          {other_system,9411277}]},
 {alarms,[]},
Comment 10 errata-xmlrpc 2016-12-21 11:42:56 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2990.html

Note You need to log in before you can comment on or make changes to this bug.