Created attachment 1116955 [details] rabbitmq log controller0 Description of problem: rabbitmqctl cluster_status" show all nodes in the cluster as connected but "rabbitmqctl report" command showing error that it cant connect to the local node on each node of the cluster. [root@overcloud-controller-0 ~]# rabbitmqctl cluster_status Cluster status of node 'rabbit@overcloud-controller-0' ... [{nodes,[{disc,['rabbit@overcloud-controller-0', 'rabbit@overcloud-controller-1', 'rabbit@overcloud-controller-2']}]}, {running_nodes,['rabbit@overcloud-controller-2', 'rabbit@overcloud-controller-1', 'rabbit@overcloud-controller-0']}, {cluster_name,<<"rabbit@overcloud-controller-0">>}, {partitions,[]}] ...done. [root@overcloud-controller-0 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,29,54}} ... Status of node 'rabbit@overcloud-controller-2' ... Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-0'] rabbit@overcloud-controller-0: * connected to epmd (port 4369) on overcloud-controller-0 * node rabbit@overcloud-controller-0 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl14611@overcloud-controller-0' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-0 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,32,56}} ... Status of node 'rabbit@overcloud-controller-2' ... Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-0'] rabbit@overcloud-controller-0: * connected to epmd (port 4369) on overcloud-controller-0 * node rabbit@overcloud-controller-0 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl22338@overcloud-controller-0' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-0 ~]# [root@overcloud-controller-1 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,30,26}} ... Status of node 'rabbit@overcloud-controller-2' ... Error: unable to connect to node 'rabbit@overcloud-controller-1': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-1'] rabbit@overcloud-controller-1: * connected to epmd (port 4369) on overcloud-controller-1 * node rabbit@overcloud-controller-1 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl14349@overcloud-controller-1' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-1 ~]# [root@overcloud-controller-2 ~]# rabbitmqctl report Reporting server status on {{2016,1,21},{13,33,31}} ... Status of node 'rabbit@overcloud-controller-0' ... Error: unable to connect to node 'rabbit@overcloud-controller-2': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit@overcloud-controller-2'] rabbit@overcloud-controller-2: * connected to epmd (port 4369) on overcloud-controller-2 * node rabbit@overcloud-controller-2 up, 'rabbit' application running current node details: - node name: 'rabbitmqctl14888@overcloud-controller-2' - home dir: /var/lib/rabbitmq - cookie hash: EdBLcRP4/rUPcYrgeGx7RA== [root@overcloud-controller-2 ~]# How reproducible: 10/10 Additional info:
Created attachment 1116957 [details] rabbitmq log controller1
Created attachment 1116958 [details] rabbitmq log controller2
From what I have test, it doesn't seems to affect the openstack services, I created new instances, uploaded new images and created networks without interruptions.
We've finally know what's going on there. A similar issue for the recent version (3.6.x) was fixed in bug 1356169.
Verified on: rabbitmq-server-3.3.5-29.el7ost.noarch #rabbitmqctl report Reporting server status on {{2016,12,11},{16,39,48}} ... Status of node 'rabbit@controller-2' ... [{pid,2901}, {running_applications,[{rabbit,"RabbitMQ","3.3.5"}, {os_mon,"CPO CXC 138 46","2.2.14"}, {mnesia,"MNESIA CXC 138 12","4.11"}, {xmerl,"XML parser","1.3.6"}, {sasl,"SASL CXC 138 11","2.3.4"}, {stdlib,"ERTS CXC 138 10","1.19.4"}, {kernel,"ERTS CXC 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"}, {memory,[{total,121367160}, {connection_procs,3898552}, {queue_procs,4603760}, {plugins,0}, {other_proc,13752488}, {mnesia,1168664}, {mgmt_db,0}, {msg_index,185184}, {other_ets,1285968}, {binary,69706256}, {code,16700794}, {atom,654217}, {other_system,9411277}]}, {alarms,[]},
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2990.html