Description of problem: glusterd : after rpm upgrade glusterd is not coming up(even 'service glusterd start' is failing) on 2 RHS node of cluster Version-Release number of selected component (if applicable): 3.4.0.7rhs-1.el6rhs.x86_64 How reproducible: Steps to Reproduce: 1.had 3.4.0.6rhs-1.el6rhs.x86_64 rpm in all RHS node 2.upgrade rpm to 3.4.0.7rhs-1.el6rhs.x86_64 (rpm -Uvh ) and did not stop glusterd before upgrading 3.after upgrade on 4 RHS server, checked glusterd status on them [root@cutlass glusterd]# service glusterd status glusterd (pid 13851) is running... [root@fan rpm]# service glusterd status glusterd (pid 21300) is running... [root@fred ~]# service glusterd status glusterd is stopped [root@mia rpm]# service glusterd status glusterd is stopped try to start glusterd but it is always failing [root@fred ~]# service glusterd start Starting glusterd: [FAILED] [root@fred ~]# service glusterd status glusterd is stopped [root@fred ~]# service glusterd start Starting glusterd: [FAILED] [root@fred glusterd]# cat /var/lib/glusterd/glusterd.info Actual results: Expected results: Additional info: less /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <snip> [2013-05-14 10:39:30.700132] I [glusterfsd.c:1919:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.0.7rhs (/u sr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2013-05-14 10:39:30.702926] I [glusterd.c:961:init] 0-management: Using /var/lib/glusterd as working directory [2013-05-14 10:39:30.705883] I [socket.c:3482:socket_init] 0-socket.management: SSL support is NOT enabled [2013-05-14 10:39:30.705938] I [socket.c:3497:socket_init] 0-socket.management: using system polling thread [2013-05-14 10:39:30.706477] C [rdma.c:4099:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices [2013-05-14 10:39:30.706564] E [rdma.c:4990:init] 0-rdma.management: Failed to initialize IB Device [2013-05-14 10:39:30.706582] E [rpc-transport.c:320:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2013-05-14 10:39:30.706634] W [rpcsvc.c:1387:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2013-05-14 10:39:32.084514] I [glusterd-store.c:1772:glusterd_restore_op_version] 0-management: op-version not found in store, setting it to minimum op-version : 1 [2013-05-14 10:39:32.086042] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2013-05-14 10:39:32.086100] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2013-05-14 10:39:32.086121] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2013-05-14 10:39:32.086136] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2013-05-14 10:39:32.091315] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2013-05-14 10:39:32.091373] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2013-05-14 10:39:32.091419] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2013-05-14 10:39:32.091452] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2013-05-14 10:39:32.383177] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2013-05-14 10:39:32.383243] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2013-05-14 10:39:32.383264] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2013-05-14 10:39:32.387636] I [glusterd-handler.c:2722:glusterd_friend_add] 0-management: connect returned 0 [2013-05-14 10:39:32.388495] I [glusterd-handler.c:2722:glusterd_friend_add] 0-management: connect returned 0 [2013-05-14 10:39:32.388621] I [rpc-clnt.c:961:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-05-14 10:39:32.388734] I [socket.c:3482:socket_init] 0-management: SSL support is NOT enabled [2013-05-14 10:39:32.388774] I [socket.c:3497:socket_init] 0-management: using system polling thread [2013-05-14 10:39:32.395760] I [rpc-clnt.c:961:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-05-14 10:39:32.395820] I [socket.c:3482:socket_init] 0-management: SSL support is NOT enabled [2013-05-14 10:39:32.395872] I [socket.c:3497:socket_init] 0-management: using system polling thread [2013-05-14 10:39:32.840617] E [glusterd-store.c:3135:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2013-05-14 10:39:32.840648] E [xlator.c:442:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2013-05-14 10:39:32.840662] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed [2013-05-14 10:39:32.840672] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed [2013-05-14 10:39:32.840848] W [glusterfsd.c:1011:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x5b0) [0x406760] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x405107] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x106) [0x405016]))) 0-: received signum (0), shutting down ' <snip>
Rachana, how is the situation with 3.4.0.8rhs alone? I know there was the issue with 3.4.0.7rhs->3.4.0.8rhs, but now that 3.4.0.8rhs fixes most of the issues, should we mark this closed?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html