Created attachment 1029873 [details] core dump Description of problem: When glusterd is running glusterfs update fails. However glusterd service stops to run and it cannot be started again. Some glusterfs processes are still running. Moreover after call of some gluster command it fails with segmentation fault. Version-Release number of selected component (if applicable): glusterfs-3.6.0.53-1 How reproducible: 100% Steps to Reproduce: 1. try to update glusterfs whilst glusterd is running 2. run some gluster command e.g. # gluster volume status Segmentation fault (core dumped) Actual results: Glusterd service is stopped and gluster commands fail with segmentation fault Expected results: glusterd could still be stopped correctly and no segmentation fault Additional info:
The core file seems to be corrupted, I am unable to print the backtrace of it and unfortunately this doesn't get reproduced as well. Could you try to reproduce it? If not then we would need to close it.
Created attachment 1034433 [details] core dump
(In reply to Atin Mukherjee from comment #5) > The core file seems to be corrupted, I am unable to print the backtrace of > it and unfortunately this doesn't get reproduced as well. Could you try to > reproduce it? If not then we would need to close it. I tried with success. However I had there something more, something which I forget to mention in scenario. I had some started volume on the machine, before glusterfs update. Sorry for that. I added a new core file to this bug.
This issue is because of the issue faced in https://bugzilla.redhat.com/show_bug.cgi?id=1211839 While the brick process or glusterd is up, the in-service software update should be blocked. In this case, while it was tried to upgrade from RHGS 3.0.4 ( glusterfs-3.6.0.53-1.el6rhs ) to RHGS 3.1 ( glusterfs-3.7.0-3.el6rhs ), few packages are upgraded, even though there was gluster brick process up. The upgraded packages are glusterfs-client-xlators, glusterfs-cli, glusterfs-geo-replication. This is evident from the list of rpm packages listed in comment12 Here is the console output, when I tried to upgrade from RHGS 3.0.4 to RHGS 3.1 : <snip> Total 18 MB/s | 3.2 MB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Error in PRETRANS scriptlet in rpm package glusterfs-libs-3.7.1-1.el6rhs.x86_64 error: lua script failed: Detected running glusterfs processes ERROR: Distribute volumes detected. In-service rolling upgrade requires distribute volume(s) to be stopped. ERROR: Please stop distribute volume(s) before proceeding... exiting! Error in PRETRANS scriptlet in rpm package glusterfs-3.7.1-1.el6rhs.x86_64 error: lua script failed: Detected running glusterfs processes Error in PRETRANS scriptlet in rpm package glusterfs-fuse-3.7.1-1.el6rhs.x86_64 error: lua script failed: Detected running glusterfs processes Error in PRETRANS scriptlet in rpm package glusterfs-api-3.7.1-1.el6rhs.x86_64 error: lua script failed: Detected running glusterfs processes Error in PRETRANS scriptlet in rpm package glusterfs-server-3.7.1-1.el6rhs.x86_64 error: lua script failed: Detected running glusterfs processes Error in PRETRANS scriptlet in rpm package glusterfs-rdma-3.7.1-1.el6rhs.x86_64 error: lua script failed: Detected running glusterfs processes Installing : glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64 1/17 Updating : glusterfs-cli-3.7.1-1.el6rhs.x86_64 2/17 Updating : glusterfs-geo-replication-3.7.1-1.el6rhs.x86_64 3/17 Non-fatal POSTIN scriptlet failure in rpm package glusterfs-geo-replication-3.7.1-1.el6rhs.x86_64 warning: %post(glusterfs-geo-replication-3.7.1-1.el6rhs.x86_64) scriptlet failed, exit status 1 Cleanup : glusterfs-geo-replication-3.6.0.53-1.el6rhs.x86_64 4/17 Cleanup : glusterfs-cli-3.6.0.53-1.el6rhs.x86_64 5/17 Verifying : glusterfs-cli-3.7.1-1.el6rhs.x86_64 1/17 Verifying : glusterfs-geo-replication-3.7.1-1.el6rhs.x86_64 2/17 Verifying : glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64 3/17 glusterfs-server-3.6.0.53-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-server-3.6.0.53-1.el6rhs.x86_64 4/17 glusterfs-3.6.0.53-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-3.6.0.53-1.el6rhs.x86_64 5/17 glusterfs-api-3.6.0.53-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-api-3.6.0.53-1.el6rhs.x86_64 6/17 Verifying : glusterfs-cli-3.6.0.53-1.el6rhs.x86_64 7/17 Verifying : glusterfs-libs-3.7.1-1.el6rhs.x86_64 8/17 glusterfs-libs-3.6.0.53-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-libs-3.6.0.53-1.el6rhs.x86_64 9/17 glusterfs-fuse-3.6.0.53-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-fuse-3.6.0.53-1.el6rhs.x86_64 10/17 Verifying : glusterfs-server-3.7.1-1.el6rhs.x86_64 11/17 Verifying : glusterfs-rdma-3.7.1-1.el6rhs.x86_64 12/17 Verifying : glusterfs-fuse-3.7.1-1.el6rhs.x86_64 13/17 glusterfs-rdma-3.6.0.53-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-rdma-3.6.0.53-1.el6rhs.x86_64 14/17 Verifying : glusterfs-geo-replication-3.6.0.53-1.el6rhs.x86_64 15/17 Verifying : glusterfs-3.7.1-1.el6rhs.x86_64 16/17 Verifying : glusterfs-api-3.7.1-1.el6rhs.x86_64 17/17 Dependency Installed: glusterfs-client-xlators.x86_64 0:3.7.1-1.el6rhs Dependency Updated: glusterfs-cli.x86_64 0:3.7.1-1.el6rhs glusterfs-geo-replication.x86_64 0:3.7.1-1.el6rhs Failed: glusterfs.x86_64 0:3.6.0.53-1.el6rhs glusterfs.x86_64 0:3.7.1-1.el6rhs glusterfs-api.x86_64 0:3.6.0.53-1.el6rhs glusterfs-api.x86_64 0:3.7.1-1.el6rhs glusterfs-fuse.x86_64 0:3.6.0.53-1.el6rhs glusterfs-fuse.x86_64 0:3.7.1-1.el6rhs glusterfs-libs.x86_64 0:3.6.0.53-1.el6rhs glusterfs-libs.x86_64 0:3.7.1-1.el6rhs glusterfs-rdma.x86_64 0:3.6.0.53-1.el6rhs glusterfs-rdma.x86_64 0:3.7.1-1.el6rhs glusterfs-server.x86_64 0:3.6.0.53-1.el6rhs glusterfs-server.x86_64 0:3.7.1-1.el6rhs Complete! [root@ ~]# service glusterd status glusterd is stopped [root@ ~]# gluster volume status Segmentation fault (core dumped) </snip>
My suggestion would, the while performing software upgrade or update, if any of the gluster processes are running, then upgrade all the gluster packages must be abadoned. I have tested the upgrade after killing gluster brick processes, the upgrade was smooth with no errors
As per #comment 13 marking this bug as duplicate of #BZ 1211839. Feel free to reopen if you think otherwise. *** This bug has been marked as a duplicate of bug 1211839 ***