Bug 962692 - glusterd : after rpm upgrade glusterd is not coming up(even 'service glusterd start' is failing) on 2 RHS node of cluster
glusterd : after rpm upgrade glusterd is not coming up(even 'service glusterd...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
x86_64 Linux
medium Severity high
: ---
: ---
Assigned To: krishnan parthasarathi
amainkar
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-14 05:22 EDT by Rachana Patel
Modified: 2015-11-03 18:05 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 18:39:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rachana Patel 2013-05-14 05:22:38 EDT
Description of problem:
glusterd : after rpm upgrade glusterd is not coming up(even 'service glusterd start' is failing) on 2 RHS node of cluster

Version-Release number of selected component (if applicable):
3.4.0.7rhs-1.el6rhs.x86_64

How reproducible:


Steps to Reproduce:
1.had 3.4.0.6rhs-1.el6rhs.x86_64 rpm in all RHS node
2.upgrade rpm to 3.4.0.7rhs-1.el6rhs.x86_64 (rpm -Uvh ) and did not stop glusterd before upgrading
3.after upgrade on 4 RHS server, checked glusterd status on them

[root@cutlass glusterd]# service glusterd status
glusterd (pid  13851) is running...

[root@fan rpm]# service glusterd status
glusterd (pid  21300) is running...

[root@fred ~]# service glusterd status
glusterd is stopped


[root@mia rpm]# service glusterd status
glusterd is stopped

try to start glusterd but it is always failing
[root@fred ~]# service glusterd start
Starting glusterd:                                         [FAILED]
[root@fred ~]# service glusterd status
glusterd is stopped
[root@fred ~]# service glusterd start
Starting glusterd:                                         [FAILED]

[root@fred glusterd]# cat /var/lib/glusterd/glusterd.info 


  
Actual results:


Expected results:


Additional info:
less /var/log/glusterfs/etc-glusterfs-glusterd.vol.log

<snip>

[2013-05-14 10:39:30.700132] I [glusterfsd.c:1919:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.0.7rhs (/u
sr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2013-05-14 10:39:30.702926] I [glusterd.c:961:init] 0-management: Using /var/lib/glusterd as working directory
[2013-05-14 10:39:30.705883] I [socket.c:3482:socket_init] 0-socket.management: SSL support is NOT enabled
[2013-05-14 10:39:30.705938] I [socket.c:3497:socket_init] 0-socket.management: using system polling thread
[2013-05-14 10:39:30.706477] C [rdma.c:4099:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
[2013-05-14 10:39:30.706564] E [rdma.c:4990:init] 0-rdma.management: Failed to initialize IB Device
[2013-05-14 10:39:30.706582] E [rpc-transport.c:320:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2013-05-14 10:39:30.706634] W [rpcsvc.c:1387:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2013-05-14 10:39:32.084514] I [glusterd-store.c:1772:glusterd_restore_op_version] 0-management: op-version not found in store, setting it to minimum op-version : 1
[2013-05-14 10:39:32.086042] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2013-05-14 10:39:32.086100] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2013-05-14 10:39:32.086121] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2013-05-14 10:39:32.086136] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2013-05-14 10:39:32.091315] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2013-05-14 10:39:32.091373] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2013-05-14 10:39:32.091419] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2013-05-14 10:39:32.091452] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2013-05-14 10:39:32.383177] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2013-05-14 10:39:32.383243] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2013-05-14 10:39:32.383264] E [glusterd-store.c:2505:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2013-05-14 10:39:32.387636] I [glusterd-handler.c:2722:glusterd_friend_add] 0-management: connect returned 0
[2013-05-14 10:39:32.388495] I [glusterd-handler.c:2722:glusterd_friend_add] 0-management: connect returned 0
[2013-05-14 10:39:32.388621] I [rpc-clnt.c:961:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2013-05-14 10:39:32.388734] I [socket.c:3482:socket_init] 0-management: SSL support is NOT enabled
[2013-05-14 10:39:32.388774] I [socket.c:3497:socket_init] 0-management: using system polling thread
[2013-05-14 10:39:32.395760] I [rpc-clnt.c:961:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2013-05-14 10:39:32.395820] I [socket.c:3482:socket_init] 0-management: SSL support is NOT enabled
[2013-05-14 10:39:32.395872] I [socket.c:3497:socket_init] 0-management: using system polling thread
[2013-05-14 10:39:32.840617] E [glusterd-store.c:3135:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
[2013-05-14 10:39:32.840648] E [xlator.c:442:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2013-05-14 10:39:32.840662] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed
[2013-05-14 10:39:32.840672] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed
[2013-05-14 10:39:32.840848] W [glusterfsd.c:1011:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x5b0) [0x406760] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x405107] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x106) [0x405016]))) 0-: received signum (0), shutting down
'
<snip>
Comment 4 Amar Tumballi 2013-05-16 04:47:37 EDT
Rachana, how is the situation with 3.4.0.8rhs alone? I know there was the issue with 3.4.0.7rhs->3.4.0.8rhs, but now that 3.4.0.8rhs fixes most of the issues, should we mark this closed?
Comment 5 Scott Haines 2013-09-23 18:39:47 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html
Comment 6 Scott Haines 2013-09-23 18:43:48 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.