Description of problem: While updating glusterfs package to latest version the update is getting hang on glusterfs-server package and is not continuing after that. Updating : glusterfs-server-3.5qa2-0.340.gitc193996.el6rhs.x86_64 5/16 tried with glusterd stop and glusterd start.Even after rebooting the issue persist. Version-Release number of selected component (if applicable): glusterfs-server-3.5qa2-0.340.gitc193996.el6rhs.x86_ How reproducible: Always Steps to Reproduce: 1.yum update 2. 3. Actual results: Dependencies Resolved =================================================================================== Package Arch Version Repository Size =================================================================================== Updating: glusterfs x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 1.2 M Updating for dependencies: glusterfs-api x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 50 k glusterfs-cli x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 107 k glusterfs-fuse x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 74 k glusterfs-geo-replication x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 137 k glusterfs-libs x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 243 k glusterfs-rdma x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 31 k glusterfs-server x86_64 3.5qa2-0.340.gitc193996.el6rhs rhs3.0-server 628 k Transaction Summary =================================================================================== Upgrade 8 Package(s) Total download size: 2.4 M Is this ok [y/N]: y Downloading Packages: (1/8): glusterfs-3.5qa2-0.340.gitc193996.el6rhs.x86_64.rpm | 1.2 MB 00:04 (2/8): glusterfs-api-3.5qa2-0.340.gitc193996.el6rhs.x86_64. | 50 kB 00:01 (3/8): glusterfs-cli-3.5qa2-0.340.gitc193996.el6rhs.x86_64. | 107 kB 00:01 (4/8): glusterfs-fuse-3.5qa2-0.340.gitc193996.el6rhs.x86_64 | 74 kB 00:01 (5/8): glusterfs-geo-replication-3.5qa2-0.340.gitc193996.el | 137 kB 00:01 (6/8): glusterfs-libs-3.5qa2-0.340.gitc193996.el6rhs.x86_64 | 243 kB 00:02 (7/8): glusterfs-rdma-3.5qa2-0.340.gitc193996.el6rhs.x86_64 | 31 kB 00:00 (8/8): glusterfs-server-3.5qa2-0.340.gitc193996.el6rhs.x86_ | 628 kB 00:04 ----------------------------------------------------------------------------------- Total 87 kB/s | 2.4 MB 00:28 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : glusterfs-libs-3.5qa2-0.340.gitc193996.el6rhs.x86_64 1/16 Updating : glusterfs-3.5qa2-0.340.gitc193996.el6rhs.x86_64 2/16 Updating : glusterfs-fuse-3.5qa2-0.340.gitc193996.el6rhs.x86_64 3/16 Updating : glusterfs-cli-3.5qa2-0.340.gitc193996.el6rhs.x86_64 4/16 Updating : glusterfs-server-3.5qa2-0.340.gitc193996.el6rhs.x86_64 5/16 Expected results: Gluster package installation should not fail. Additional info:
We found that when the yum upgrade process is hung, glusterd process started in 'upgrade' mode is also hung. On running the glusterd in upgrade mode, outside of yum upgrade, we saw the following output: [root@rhsauto069 ~]# glusterd --xlator-option *.upgrade=on -N librdmacm: Warning: couldn't read ABI version. librdmacm: Warning: assuming: 4 librdmacm: Fatal: unable to get RDMA device list Following is the stack trace where glusterd is 'waiting' on, Thread 4 (Thread 0x7fb704f3a700 (LWP 26870)): #0 0x00000033ea00e264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00000033ea009508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00000033ea0093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fb6fae8d5e6 in gf_rdma_init (this=0x6f62f0) at rdma.c:4249 #4 init (this=0x6f62f0) at rdma.c:4480 #5 0x00007fb705541a47 in rpc_transport_load (ctx=<value optimized out>, options=0x7fb703bc602c, trans_name=<value optimized out>) at rpc-transport.c:330 #6 0x00007fb70553ddc2 in rpcsvc_transport_create (svc=0x6f1640, options=<value optimized out>, name=<value optimized out>) at rpcsvc.c:1474 #7 0x00007fb70553deca in rpcsvc_create_listener (svc=0x6f1640, options=<value optimized out>, name=<value optimized out>) at rpcsvc.c:1541 #8 0x00007fb705540130 in rpcsvc_create_listeners (svc=0x6f1640, options=0x7fb703bc602c, name=0x6ec000 "management") at rpcsvc.c:1620 #9 0x00007fb6fba5ccf8 in init (this=0x6eb5d0) at glusterd.c:1337 #10 0x00007fb70576a1c2 in __xlator_init (xl=0x6eb5d0) at xlator.c:377 #11 xlator_init (xl=0x6eb5d0) at xlator.c:400 #12 0x00007fb7057a3b81 in glusterfs_graph_init (graph=<value optimized out>) at graph.c:304 #13 0x00007fb7057a3ca2 in glusterfs_graph_activate (graph=0x6e2fd0, ctx=0x6c5010) at graph.c:500 #14 0x00000000004051cb in glusterfs_process_volfp (ctx=0x6c5010, fp=0x6e70b0) at glusterfsd.c:1828 #15 0x0000000000405302 in glusterfs_volumes_init (ctx=0x6c5010) at glusterfsd.c:1880 #16 0x0000000000407b8c in main (argc=4, argv=0x7fff86c07348) at glusterfsd.c:1961
Provisioned a new RHS system and tried yum update again, it hungs at the glusterfs-server package update.Given the machine details to Dev team for debugging.
The glusterfs server update is getting hang from the following version : glusterfs-server-3.5qa2-0.323.git6567d14.el6rhs.x86_64 to glusterfs-server-3.5qa2-0.340.gitc193996.el6rhs.x86_64 But it is successfull from any older version to the latest version glusterfs-server-3.5qa2-0.369.git500a656.el6rhs.x86_64 so reducing the priority of BZ.
surabhi, Could you check if you observe yum upgrade hanging with the recent releases?
I am seeing the issue again with the new build. glusterfs-server-3.5qa2-0.450.gita05c579.el6rhs.x86_64
The update from glusterfs-server-3.5qa2-0.425.git9360107.el6rhs.x86_64.rpm to glusterfs-server-3.5qa2-0.450.gita05c579.el6rhs.x86_64 hangs.Also getting the following error: librdmacm: Warning: couldn't read ABI version. librdmacm: Warning: assuming: 4 librdmacm: Fatal: unable to get RDMA device list
(In reply to surabhi from comment #7) > The update from glusterfs-server-3.5qa2-0.425.git9360107.el6rhs.x86_64.rpm > to glusterfs-server-3.5qa2-0.450.gita05c579.el6rhs.x86_64 hangs.Also getting > the following error: > Correcting the version number so the hung was seen from glusterfs-server-3.5qa2-0.340.gitc193996.el6rhs.x86_64 to glusterfs-server-3.5qa2-0.425.git9360107.el6rhs.x86_64. > librdmacm: Warning: couldn't read ABI version. > librdmacm: Warning: assuming: 4 > librdmacm: Fatal: unable to get RDMA device list Also saw the following errors. Non-fatal POSTUN scriptlet failure in rpm package glusterfs-libs Cleanup : glusterfs-debuginfo-3.5qa2-0.340.gitc193996.el6rhs.x86_64 22/22 /sbin/ldconfig: relative path `1' used to build cache warning: %postun(glusterfs-libs-3.5qa2-0.425.git9360107.el6rhs.x86_64) scriptlet failed, exit status 1
With the latest downstream nightly builds the hang issue is no longer seen. I tried upgrading glusterfs packages in 2 iteration Iteration 1 - glusterfs-3.6.0-1.0.el6rhs to glusterfs-3.6.0.1-1.el6rhs Iteration 2 - glusterfs-3.6.0.3-1.el6rhs to glusterfs-3.6.0.4-1.el6rhs On both these upgrades, I was not seeing any issues like upgrade hang. But I see 2 warnings, though the upgrade was successful. This may be the concern a little, as the upgrade is not exactly clean enough Below is the snip from the upgrade, <snip> Total 46 MB/s | 2.5 MB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : glusterfs-libs-3.6.0.4-1.el6rhs.x86_64 1/16 Updating : glusterfs-3.6.0.4-1.el6rhs.x86_64 2/16 Updating : glusterfs-api-3.6.0.4-1.el6rhs.x86_64 3/16 Updating : glusterfs-fuse-3.6.0.4-1.el6rhs.x86_64 4/16 Updating : glusterfs-cli-3.6.0.4-1.el6rhs.x86_64 5/16 Updating : glusterfs-server-3.6.0.4-1.el6rhs.x86_64 6/16 error reading information on service glusterfsd: No such file or directory Updating : glusterfs-geo-replication-3.6.0.4-1.el6rhs.x86_64 7/16 Updating : glusterfs-rdma-3.6.0.4-1.el6rhs.x86_64 8/16 Cleanup : glusterfs-geo-replication-3.6.0.3-1.el6rhs.x86_64 9/16 Cleanup : glusterfs-server-3.6.0.3-1.el6rhs.x86_64 10/16 Cleanup : glusterfs-api-3.6.0.3-1.el6rhs.x86_64 11/16 Cleanup : glusterfs-rdma-3.6.0.3-1.el6rhs.x86_64 12/16 Cleanup : glusterfs-cli-3.6.0.3-1.el6rhs.x86_64 13/16 Cleanup : glusterfs-fuse-3.6.0.3-1.el6rhs.x86_64 14/16 Cleanup : glusterfs-3.6.0.3-1.el6rhs.x86_64 15/16 Cleanup : glusterfs-libs-3.6.0.3-1.el6rhs.x86_64 16/16 Non-fatal POSTUN scriptlet failure in rpm package glusterfs-libs Verifying : glusterfs-rdma-3.6.0.4-1.el6rhs.x86_64 1/16 Verifying : glusterfs-geo-replication-3.6.0.4-1.el6rhs.x86_64 2/16 Verifying : glusterfs-server-3.6.0.4-1.el6rhs.x86_64 3/16 Verifying : glusterfs-api-3.6.0.4-1.el6rhs.x86_64 4/16 Verifying : glusterfs-cli-3.6.0.4-1.el6rhs.x86_64 5/16 Verifying : glusterfs-libs-3.6.0.4-1.el6rhs.x86_64 6/16 Verifying : glusterfs-fuse-3.6.0.4-1.el6rhs.x86_64 7/16 Verifying : glusterfs-3.6.0.4-1.el6rhs.x86_64 8/16 Verifying : glusterfs-geo-replication-3.6.0.3-1.el6rhs.x86_64 9/16 Verifying : glusterfs-fuse-3.6.0.3-1.el6rhs.x86_64 10/16 Verifying : glusterfs-api-3.6.0.3-1.el6rhs.x86_64 11/16 Verifying : glusterfs-server-3.6.0.3-1.el6rhs.x86_64 12/16 Verifying : glusterfs-libs-3.6.0.3-1.el6rhs.x86_64 13/16 Verifying : glusterfs-3.6.0.3-1.el6rhs.x86_64 14/16 Verifying : glusterfs-cli-3.6.0.3-1.el6rhs.x86_64 15/16 Verifying : glusterfs-rdma-3.6.0.3-1.el6rhs.x86_64 </snip>
Updating the native client from RHS 2.1 U2 to latest downstream nightly build ( glusterfs-3.6.0.5-1.el6 ) also throws some warnings, Running Transaction Updating : glusterfs-libs-3.6.0.5-1.el6.x86_64 1/6 Non-fatal POSTIN scriptlet failure in rpm package glusterfs-libs-3.6.0.5-1.el6.x86_64 /sbin/ldconfig: relative path `2' used to build cache warning: %post(glusterfs-libs-3.6.0.5-1.el6.x86_64) scriptlet failed, exit status 1 Updating : glusterfs-3.6.0.5-1.el6.x86_64
This is a spec file related issue. Assigning it to Bala for further investigation.
KP, as we discussed about this problem, this issue is not at build side. This should be taken and fixed in respective component.
(In reply to SATHEESARAN from comment #10) > Updating the native client from RHS 2.1 U2 to latest downstream nightly > build ( glusterfs-3.6.0.5-1.el6 ) also throws some warnings, > > Running Transaction > Updating : glusterfs-libs-3.6.0.5-1.el6.x86_64 > 1/6 > Non-fatal POSTIN scriptlet failure in rpm package > glusterfs-libs-3.6.0.5-1.el6.x86_64 > /sbin/ldconfig: relative path `2' used to build cache > warning: %post(glusterfs-libs-3.6.0.5-1.el6.x86_64) scriptlet failed, exit > status 1 > Updating : glusterfs-3.6.0.5-1.el6.x86_64 I am slightly confused here in upgrading rhs client 2.1u2 to rhs client 3.0. Both are for different RHEL ie rhs client 2.1u2 for rhel 6.4 and rhs client 3.0 for rhel 6.5. How was the upgrade done?
Closing as duplicate as per comment #11. Other issues were already resolved. Please open a separate bug for each issue. *** This bug has been marked as a duplicate of bug 1100214 ***