Bug 1300728
| Summary: | rabbitmq report show Error: unable to connect to the local node | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Asaf Hirshberg <ahirshbe> | ||||||||
| Component: | rabbitmq-server | Assignee: | Peter Lemenkov <plemenko> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Asaf Hirshberg <ahirshbe> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 7.0 (Kilo) | CC: | apevec, jeckersb, lhh, oblaut, srevivo, ushkalim | ||||||||
| Target Milestone: | async | Keywords: | Triaged, ZStream | ||||||||
| Target Release: | 8.0 (Liberty) | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | rabbitmq-server-3.3.5-28.el7ost | Doc Type: | Bug Fix | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2016-12-21 16:42:56 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1356169 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 1116957 [details]
rabbitmq log controller1
Created attachment 1116958 [details]
rabbitmq log controller2
From what I have test, it doesn't seems to affect the openstack services, I created new instances, uploaded new images and created networks without interruptions. We've finally know what's going on there. A similar issue for the recent version (3.6.x) was fixed in bug 1356169. Verified on: rabbitmq-server-3.3.5-29.el7ost.noarch
#rabbitmqctl report
Reporting server status on {{2016,12,11},{16,39,48}}
...
Status of node 'rabbit@controller-2' ...
[{pid,2901},
{running_applications,[{rabbit,"RabbitMQ","3.3.5"},
{os_mon,"CPO CXC 138 46","2.2.14"},
{mnesia,"MNESIA CXC 138 12","4.11"},
{xmerl,"XML parser","1.3.6"},
{sasl,"SASL CXC 138 11","2.3.4"},
{stdlib,"ERTS CXC 138 10","1.19.4"},
{kernel,"ERTS CXC 138 10","2.16.4"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,[{total,121367160},
{connection_procs,3898552},
{queue_procs,4603760},
{plugins,0},
{other_proc,13752488},
{mnesia,1168664},
{mgmt_db,0},
{msg_index,185184},
{other_ets,1285968},
{binary,69706256},
{code,16700794},
{atom,654217},
{other_system,9411277}]},
{alarms,[]},
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2990.html |
Created attachment 1116955 [details] rabbitmq log controller0 Description of problem: rabbitmqctl cluster_status" show all nodes in the cluster as connected but "rabbitmqctl report" command showing error that it cant connect to the local node on each node of the cluster. [root@overcloud-controller-0 ~]# rabbitmqctl cluster_status Cluster status of node 'rabbit@overcloud-controller-0' ... [{nodes,[{disc,['rabbit@overcloud-controller-0', 'rabbit@overcloud-controller-1', 'rabbit@overcloud-controller-2']}]}, {running_nodes,['rabbit@overcloud-controller-2', 'rabbit@overcloud-controller-1', 'rabbit@overcloud-controller-0']}, {cluster_name,<<"rabbit@overcloud-controller-0">>}, {partitions,[]}] ...done. [root@overcloud-controller-0 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,29,54}} ... Status of node 'rabbit@overcloud-controller-2' ... Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-0'] rabbit@overcloud-controller-0: * connected to epmd (port 4369) on overcloud-controller-0 * node rabbit@overcloud-controller-0 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl14611@overcloud-controller-0' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-0 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,32,56}} ... Status of node 'rabbit@overcloud-controller-2' ... Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-0'] rabbit@overcloud-controller-0: * connected to epmd (port 4369) on overcloud-controller-0 * node rabbit@overcloud-controller-0 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl22338@overcloud-controller-0' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-0 ~]# [root@overcloud-controller-1 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,30,26}} ... Status of node 'rabbit@overcloud-controller-2' ... Error: unable to connect to node 'rabbit@overcloud-controller-1': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-1'] rabbit@overcloud-controller-1: * connected to epmd (port 4369) on overcloud-controller-1 * node rabbit@overcloud-controller-1 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl14349@overcloud-controller-1' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-1 ~]# [root@overcloud-controller-2 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,33,31}} ... Status of node 'rabbit@overcloud-controller-0' ... Error: unable to connect to node 'rabbit@overcloud-controller-2': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-2'] rabbit@overcloud-controller-2: * connected to epmd (port 4369) on overcloud-controller-2 * node rabbit@overcloud-controller-2 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl14888@overcloud-controller-2' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-2 ~]# How reproducible: 10/10 Additional info: