Bug 1198887

Summary: While performing in-service software update, glusterfs-geo-replication and glusterfs-cli packages are updated even when glusterfsd or distributed volume is up
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: SATHEESARAN <sasundar>
Component: coreAssignee: Niels de Vos <ndevos>
Status: CLOSED CURRENTRELEASE QA Contact: SATHEESARAN <sasundar>
Severity: urgent Docs Contact:
Priority: high    
Version: rhgs-3.0CC: barumuga, ndevos, nlevinki, rcyriac, rhs-bugs, storage-qa-internal, vagarwal, vikumar
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1211839 (view as bug list) Environment:
Last Closed: 2015-08-10 09:37:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1211839    

Description SATHEESARAN 2015-03-05 03:16:33 UTC 
Description of problem:
-----------------------
While performing 'In-service Software Upgrade', glusterfs package update should be prevented, when there are glusterfsd, glusterfs, glusterd processes are running.

The same is the case when there are distributed volume is running.

But while upgrading from RHS 3.0.3-async to latest RHS 3.0.4 build, 
glusterfs-geo-replication and glusterfs-cli packages are getting updated, even when there is a distributed volume or glusterfsd( brick process ) of replicate/distributed-replicate volumes up / running.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
glusterfs-3.6.0.48-1.el6rhs
glusterfs-3.6.0.42-1.1.el6rhs

How reproducible:
-----------------
Always

Steps to Reproduce:
-------------------
1. Install RHS 3.0.3 async update ( glusterfs-3.6.0.42-1.1.el6rhs )
2. Create a distributed volume and start it
3. Upgrade to latest RHS-3.0.4 candidate ( glusterfs-3.6.0.48-1.el6rhs )

[or]

1. Install RHS 3.0.3 async update ( glusterfs-3.6.0.42-1.1.el6rhs )
2. Create a distributed-replicate or replicate volume and start it
3. Upgrade to latest RHS-3.0.4 candidate ( glusterfs-3.6.0.48-1.el6rhs )

Actual results:
---------------
glusterfs-geo-replication and glusterfs-cli packages are updated

Expected results:
-----------------
Updating all glusterfs packages should be prevented, as there are distribute volume is not stopped.

The same holds true for any existing replicate or distributed-replicate volume

Additional info:
----------------
Refer - https://bugzilla.redhat.com/show_bug.cgi?id=1016385#c14
Comment 1 SATHEESARAN 2015-03-05 03:19:38 UTC 
Providinf the console log, when upgrading the setup with replicate volume.
You can observe that the glusterfs-geo-replication and glusterfs-cli packages are updated.

[root@ ~]# yum update glusterfs* --disablerepo=* --enablerepo=rhs
Loaded plugins: aliases, changelog, downloadonly, product-id, security,
              : subscription-manager, tmprepo, verify, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Update Process
rhs                                                        | 2.9 kB     00:00     
rhs/primary_db                                             | 9.8 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-api.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-api.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-cli.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-cli.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-fuse.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-fuse.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-geo-replication.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-geo-replication.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-libs.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-libs.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-rdma.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-rdma.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-server.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-server.x86_64 0:3.6.0.48-1.el6rhs will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================
 Package                        Arch        Version                Repository
                                                                             Size
==================================================================================
Updating:
 glusterfs                      x86_64      3.6.0.48-1.el6rhs      rhs      1.3 M
 glusterfs-api                  x86_64      3.6.0.48-1.el6rhs      rhs       59 k
 glusterfs-cli                  x86_64      3.6.0.48-1.el6rhs      rhs      118 k
 glusterfs-fuse                 x86_64      3.6.0.48-1.el6rhs      rhs       80 k
 glusterfs-geo-replication      x86_64      3.6.0.48-1.el6rhs      rhs      154 k
 glusterfs-libs                 x86_64      3.6.0.48-1.el6rhs      rhs      269 k
 glusterfs-rdma                 x86_64      3.6.0.48-1.el6rhs      rhs       38 k
 glusterfs-server               x86_64      3.6.0.48-1.el6rhs      rhs      704 k

Transaction Summary
==================================================================================
Upgrade       8 Package(s)

Total download size: 2.7 M
Is this ok [y/N]: y
Downloading Packages:
(1/8): glusterfs-3.6.0.48-1.el6rhs.x86_64.rpm              | 1.3 MB     00:00     
(2/8): glusterfs-api-3.6.0.48-1.el6rhs.x86_64.rpm          |  59 kB     00:00     
(3/8): glusterfs-cli-3.6.0.48-1.el6rhs.x86_64.rpm          | 118 kB     00:00     
(4/8): glusterfs-fuse-3.6.0.48-1.el6rhs.x86_64.rpm         |  80 kB     00:00     
(5/8): glusterfs-geo-replication-3.6.0.48-1.el6rhs.x86_64. | 154 kB     00:00     
(6/8): glusterfs-libs-3.6.0.48-1.el6rhs.x86_64.rpm         | 269 kB     00:00     
(7/8): glusterfs-rdma-3.6.0.48-1.el6rhs.x86_64.rpm         |  38 kB     00:00     
(8/8): glusterfs-server-3.6.0.48-1.el6rhs.x86_64.rpm       | 704 kB     00:00     
----------------------------------------------------------------------------------
Total                                              42 MB/s | 2.7 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Error in PRETRANS scriptlet in rpm package glusterfs-libs-3.6.0.48-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
WARNING: Updating glusterfs requires its processes to be killed. This action does NOT incur downtime.
WARNING: Ensure to wait for the upgraded server to finish healing before proceeding.
WARNING: Refer upgrade section of install guide for more details
Please run # service glusterd stop; pkill glusterfs; pkill glusterfsd; pkill gsyncd.py;
Error in PRETRANS scriptlet in rpm package glusterfs-3.6.0.48-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-api-3.6.0.48-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-fuse-3.6.0.48-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-server-3.6.0.48-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-rdma-3.6.0.48-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
  Updating   : glusterfs-cli-3.6.0.48-1.el6rhs.x86_64                        1/16 
  Updating   : glusterfs-geo-replication-3.6.0.48-1.el6rhs.x86_64            2/16 
  Cleanup    : glusterfs-geo-replication-3.6.0.42-1.el6rhs.x86_64            3/16 
  Cleanup    : glusterfs-cli-3.6.0.42-1.el6rhs.x86_64                        4/16 
  Verifying  : glusterfs-cli-3.6.0.48-1.el6rhs.x86_64                        1/16 
  Verifying  : glusterfs-geo-replication-3.6.0.48-1.el6rhs.x86_64            2/16 
  Verifying  : glusterfs-geo-replication-3.6.0.42-1.el6rhs.x86_64            3/16 
glusterfs-server-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-server-3.6.0.42-1.el6rhs.x86_64                     4/16 
glusterfs-libs-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-libs-3.6.0.42-1.el6rhs.x86_64                       5/16 
glusterfs-api-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-api-3.6.0.42-1.el6rhs.x86_64                        6/16 
  Verifying  : glusterfs-libs-3.6.0.48-1.el6rhs.x86_64                       7/16 
  Verifying  : glusterfs-fuse-3.6.0.48-1.el6rhs.x86_64                       8/16 
glusterfs-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-3.6.0.42-1.el6rhs.x86_64                            9/16 
glusterfs-fuse-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-fuse-3.6.0.42-1.el6rhs.x86_64                      10/16 
  Verifying  : glusterfs-3.6.0.48-1.el6rhs.x86_64                           11/16 
  Verifying  : glusterfs-cli-3.6.0.42-1.el6rhs.x86_64                       12/16 
  Verifying  : glusterfs-server-3.6.0.48-1.el6rhs.x86_64                    13/16 
  Verifying  : glusterfs-api-3.6.0.48-1.el6rhs.x86_64                       14/16 
glusterfs-rdma-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-rdma-3.6.0.42-1.el6rhs.x86_64                      15/16 
  Verifying  : glusterfs-rdma-3.6.0.48-1.el6rhs.x86_64                      16/16 

Updated:
  glusterfs-cli.x86_64 0:3.6.0.48-1.el6rhs                                        
  glusterfs-geo-replication.x86_64 0:3.6.0.48-1.el6rhs                            

Failed:
  glusterfs.x86_64 0:3.6.0.42-1.el6rhs                                            
  glusterfs.x86_64 0:3.6.0.48-1.el6rhs                                            
  glusterfs-api.x86_64 0:3.6.0.42-1.el6rhs                                        
  glusterfs-api.x86_64 0:3.6.0.48-1.el6rhs                                        
  glusterfs-fuse.x86_64 0:3.6.0.42-1.el6rhs                                       
  glusterfs-fuse.x86_64 0:3.6.0.48-1.el6rhs                                       
  glusterfs-libs.x86_64 0:3.6.0.42-1.el6rhs                                       
  glusterfs-libs.x86_64 0:3.6.0.48-1.el6rhs                                       
  glusterfs-rdma.x86_64 0:3.6.0.42-1.el6rhs                                       
  glusterfs-rdma.x86_64 0:3.6.0.48-1.el6rhs                                       
  glusterfs-server.x86_64 0:3.6.0.42-1.el6rhs                                     
  glusterfs-server.x86_64 0:3.6.0.48-1.el6rhs
Comment 2 SATHEESARAN 2015-03-05 03:21:28 UTC 
Providing the console log, when upgrading the setup with distribute volume.
You can observe that the glusterfs-geo-replication and glusterfs-cli packages are updated.

[root@ ~]# yum update glusterfs* --disablerepo=* --enablerepo=rhs-async
Loaded plugins: aliases, changelog, downloadonly, product-id, security,
              : subscription-manager, tmprepo, verify, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Update Process
rhs-async                                                  | 2.9 kB     00:00     
rhs-async/primary_db                                       | 9.5 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-api.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-api.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-cli.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-cli.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-fuse.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-fuse.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-geo-replication.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-geo-replication.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-libs.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-libs.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-rdma.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-rdma.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
---> Package glusterfs-server.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-server.x86_64 0:3.6.0.42.1-1.el6rhs will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================
 Package                      Arch      Version                Repository    Size
==================================================================================
Updating:
 glusterfs                    x86_64    3.6.0.42.1-1.el6rhs    rhs-async    1.3 M
 glusterfs-api                x86_64    3.6.0.42.1-1.el6rhs    rhs-async     59 k
 glusterfs-cli                x86_64    3.6.0.42.1-1.el6rhs    rhs-async    116 k
 glusterfs-fuse               x86_64    3.6.0.42.1-1.el6rhs    rhs-async     80 k
 glusterfs-geo-replication    x86_64    3.6.0.42.1-1.el6rhs    rhs-async    152 k
 glusterfs-libs               x86_64    3.6.0.42.1-1.el6rhs    rhs-async    266 k
 glusterfs-rdma               x86_64    3.6.0.42.1-1.el6rhs    rhs-async     37 k
 glusterfs-server             x86_64    3.6.0.42.1-1.el6rhs    rhs-async    699 k

Transaction Summary
==================================================================================
Upgrade       8 Package(s)

Total download size: 2.7 M
Is this ok [y/N]: y
Downloading Packages:
(1/8): glusterfs-3.6.0.42.1-1.el6rhs.x86_64.rpm            | 1.3 MB     00:00     
(2/8): glusterfs-api-3.6.0.42.1-1.el6rhs.x86_64.rpm        |  59 kB     00:00     
(3/8): glusterfs-cli-3.6.0.42.1-1.el6rhs.x86_64.rpm        | 116 kB     00:00     
(4/8): glusterfs-fuse-3.6.0.42.1-1.el6rhs.x86_64.rpm       |  80 kB     00:00     
(5/8): glusterfs-geo-replication-3.6.0.42.1-1.el6rhs.x86_6 | 152 kB     00:00     
(6/8): glusterfs-libs-3.6.0.42.1-1.el6rhs.x86_64.rpm       | 266 kB     00:00     
(7/8): glusterfs-rdma-3.6.0.42.1-1.el6rhs.x86_64.rpm       |  37 kB     00:00     
(8/8): glusterfs-server-3.6.0.42.1-1.el6rhs.x86_64.rpm     | 699 kB     00:00     
----------------------------------------------------------------------------------
Total                                              44 MB/s | 2.7 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Error in PRETRANS scriptlet in rpm package glusterfs-libs-3.6.0.42.1-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
ERROR: Distribute volumes detected. In-service rolling upgrade requires distribute volume(s) to be stopped.
ERROR: Please stop distribute volume(s) before proceeding... exiting!
Error in PRETRANS scriptlet in rpm package glusterfs-3.6.0.42.1-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-api-3.6.0.42.1-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-fuse-3.6.0.42.1-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-server-3.6.0.42.1-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
Error in PRETRANS scriptlet in rpm package glusterfs-rdma-3.6.0.42.1-1.el6rhs.x86_64
error: lua script failed: Detected running glusterfs processes
  Updating   : glusterfs-cli-3.6.0.42.1-1.el6rhs.x86_64                      1/16 
  Updating   : glusterfs-geo-replication-3.6.0.42.1-1.el6rhs.x86_64          2/16 
  Cleanup    : glusterfs-geo-replication-3.6.0.42-1.el6rhs.x86_64            3/16 
  Cleanup    : glusterfs-cli-3.6.0.42-1.el6rhs.x86_64                        4/16 
  Verifying  : glusterfs-geo-replication-3.6.0.42.1-1.el6rhs.x86_64          1/16 
  Verifying  : glusterfs-cli-3.6.0.42.1-1.el6rhs.x86_64                      2/16 
  Verifying  : glusterfs-cli-3.6.0.42-1.el6rhs.x86_64                        3/16 
glusterfs-server-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-server-3.6.0.42-1.el6rhs.x86_64                     4/16 
glusterfs-libs-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-libs-3.6.0.42-1.el6rhs.x86_64                       5/16 
  Verifying  : glusterfs-api-3.6.0.42.1-1.el6rhs.x86_64                      6/16 
  Verifying  : glusterfs-rdma-3.6.0.42.1-1.el6rhs.x86_64                     7/16 
glusterfs-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-3.6.0.42-1.el6rhs.x86_64                            8/16 
glusterfs-fuse-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-fuse-3.6.0.42-1.el6rhs.x86_64                       9/16 
  Verifying  : glusterfs-libs-3.6.0.42.1-1.el6rhs.x86_64                    10/16 
glusterfs-api-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-api-3.6.0.42-1.el6rhs.x86_64                       11/16 
  Verifying  : glusterfs-server-3.6.0.42.1-1.el6rhs.x86_64                  12/16 
  Verifying  : glusterfs-geo-replication-3.6.0.42-1.el6rhs.x86_64           13/16 
  Verifying  : glusterfs-3.6.0.42.1-1.el6rhs.x86_64                         14/16 
glusterfs-rdma-3.6.0.42-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-rdma-3.6.0.42-1.el6rhs.x86_64                      15/16 
  Verifying  : glusterfs-fuse-3.6.0.42.1-1.el6rhs.x86_64                    16/16 

Updated:
  glusterfs-cli.x86_64 0:3.6.0.42.1-1.el6rhs                                      
  glusterfs-geo-replication.x86_64 0:3.6.0.42.1-1.el6rhs                          

Failed:
  glusterfs.x86_64 0:3.6.0.42-1.el6rhs                                            
  glusterfs.x86_64 0:3.6.0.42.1-1.el6rhs                                          
  glusterfs-api.x86_64 0:3.6.0.42-1.el6rhs                                        
  glusterfs-api.x86_64 0:3.6.0.42.1-1.el6rhs                                      
  glusterfs-fuse.x86_64 0:3.6.0.42-1.el6rhs                                       
  glusterfs-fuse.x86_64 0:3.6.0.42.1-1.el6rhs                                     
  glusterfs-libs.x86_64 0:3.6.0.42-1.el6rhs                                       
  glusterfs-libs.x86_64 0:3.6.0.42.1-1.el6rhs                                     
  glusterfs-rdma.x86_64 0:3.6.0.42-1.el6rhs                                       
  glusterfs-rdma.x86_64 0:3.6.0.42.1-1.el6rhs                                     
  glusterfs-server.x86_64 0:3.6.0.42-1.el6rhs                                     
  glusterfs-server.x86_64 0:3.6.0.42.1-1.el6rhs                                   

Complete!
Comment 3 SATHEESARAN 2015-03-05 03:27:21 UTC 
RHSS Cli stopped respnding post comment1 and comment2, due to the mix of packages now.

[root@d~]# gluster volume status
[root@~]# gluster volume list
[root@~]# gluster pe s

[root@~]# ps aux | grep glusterfsd
root      5050  0.0  2.5 811436 26412 ?        Ssl  Mar03   0:04 /usr/sbin/glusterfsd -s dhcp37-220.lab.eng.blr.redhat.com --volfile-id rep.dhcp37-220.lab.eng.blr.redhat.com.tmp-6 -p /var/lib/glusterd/vols/rep/run/dhcp37-220.lab.eng.blr.redhat.com-tmp-6.pid -S /var/run/7145c16dc42f2275cbddd2178c14b44b.socket --brick-name /tmp/6 -l /var/log/glusterfs/bricks/tmp-6.log --xlator-option *-posix.glusterd-uuid=1e12a8b3-32a8-4cf6-aab1-7e551a12be72 --brick-port 49152 --xlator-option rep-server.listen-port=49152
root      6572  0.0  0.0 103252   840 pts/0    S+   02:30   0:00 grep glusterfsd

[root@ ~]# pidof glusterfsd
5050


I have killed this brick process and proceeded to upgrade once again. All worked well.

[root@~]# kill 5050
[root@~]# pkill glusterfsd
[root@~]# pkill glusterfs
[root@~]# service glusterd stop
                                     [  OK  ]
[root@~]# yum update glusterfs* --disablerepo=* --enablerepo=rhs
Loaded plugins: aliases, changelog, downloadonly, product-id, security,
              : subscription-manager, tmprepo, verify, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Repository 'rhs' is missing name in configuration, using id
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-api.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-api.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-fuse.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-fuse.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-libs.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-libs.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-rdma.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-rdma.x86_64 0:3.6.0.48-1.el6rhs will be an update
---> Package glusterfs-server.x86_64 0:3.6.0.42-1.el6rhs will be updated
---> Package glusterfs-server.x86_64 0:3.6.0.48-1.el6rhs will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================
 Package                 Arch          Version                   Repository  Size
==================================================================================
Updating:
 glusterfs               x86_64        3.6.0.48-1.el6rhs         rhs        1.3 M
 glusterfs-api           x86_64        3.6.0.48-1.el6rhs         rhs         59 k
 glusterfs-fuse          x86_64        3.6.0.48-1.el6rhs         rhs         80 k
 glusterfs-libs          x86_64        3.6.0.48-1.el6rhs         rhs        269 k
 glusterfs-rdma          x86_64        3.6.0.48-1.el6rhs         rhs         38 k
 glusterfs-server        x86_64        3.6.0.48-1.el6rhs         rhs        704 k

Transaction Summary
==================================================================================
Upgrade       6 Package(s)

Total download size: 2.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/6): glusterfs-3.6.0.48-1.el6rhs.x86_64.rpm              | 1.3 MB     00:00     
(2/6): glusterfs-api-3.6.0.48-1.el6rhs.x86_64.rpm          |  59 kB     00:00     
(3/6): glusterfs-fuse-3.6.0.48-1.el6rhs.x86_64.rpm         |  80 kB     00:00     
(4/6): glusterfs-libs-3.6.0.48-1.el6rhs.x86_64.rpm         | 269 kB     00:00     
(5/6): glusterfs-rdma-3.6.0.48-1.el6rhs.x86_64.rpm         |  38 kB     00:00     
(6/6): glusterfs-server-3.6.0.48-1.el6rhs.x86_64.rpm       | 704 kB     00:00     
----------------------------------------------------------------------------------
Total                                              46 MB/s | 2.4 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : glusterfs-libs-3.6.0.48-1.el6rhs.x86_64                       1/12 
  Updating   : glusterfs-3.6.0.48-1.el6rhs.x86_64                            2/12 
  Updating   : glusterfs-api-3.6.0.48-1.el6rhs.x86_64                        3/12 
  Updating   : glusterfs-fuse-3.6.0.48-1.el6rhs.x86_64                       4/12 
  Updating   : glusterfs-server-3.6.0.48-1.el6rhs.x86_64                     5/12 
error reading information on service glusterfsd: No such file or directory
warning: /var/lib/glusterd/vols/rep/rep.tcp-fuse.vol saved as /var/lib/glusterd/vols/rep/rep.tcp-fuse.vol.rpmsave
warning: /var/lib/glusterd/vols/rep/rep.dhcp37-170.lab.eng.blr.redhat.com.tmp-6.vol saved as /var/lib/glusterd/vols/rep/rep.dhcp37-170.lab.eng.blr.redhat.com.tmp-6.vol.rpmsave
warning: /var/lib/glusterd/vols/rep/trusted-rep.tcp-fuse.vol saved as /var/lib/glusterd/vols/rep/trusted-rep.tcp-fuse.vol.rpmsave
warning: /var/lib/glusterd/vols/rep/rep-rebalance.vol saved as /var/lib/glusterd/vols/rep/rep-rebalance.vol.rpmsave
warning: /var/lib/glusterd/vols/rep/rep.dhcp37-220.lab.eng.blr.redhat.com.tmp-6.vol saved as /var/lib/glusterd/vols/rep/rep.dhcp37-220.lab.eng.blr.redhat.com.tmp-6.vol.rpmsave
  Updating   : glusterfs-rdma-3.6.0.48-1.el6rhs.x86_64                       6/12 
  Cleanup    : glusterfs-server-3.6.0.42-1.el6rhs.x86_64                     7/12 
  Cleanup    : glusterfs-rdma-3.6.0.42-1.el6rhs.x86_64                       8/12 
  Cleanup    : glusterfs-fuse-3.6.0.42-1.el6rhs.x86_64                       9/12 
  Cleanup    : glusterfs-api-3.6.0.42-1.el6rhs.x86_64                       10/12 
  Cleanup    : glusterfs-3.6.0.42-1.el6rhs.x86_64                           11/12 
  Cleanup    : glusterfs-libs-3.6.0.42-1.el6rhs.x86_64                      12/12 
  Verifying  : glusterfs-libs-3.6.0.48-1.el6rhs.x86_64                       1/12 
  Verifying  : glusterfs-fuse-3.6.0.48-1.el6rhs.x86_64                       2/12 
  Verifying  : glusterfs-3.6.0.48-1.el6rhs.x86_64                            3/12 
  Verifying  : glusterfs-api-3.6.0.48-1.el6rhs.x86_64                        4/12 
  Verifying  : glusterfs-server-3.6.0.48-1.el6rhs.x86_64                     5/12 
  Verifying  : glusterfs-rdma-3.6.0.48-1.el6rhs.x86_64                       6/12 
  Verifying  : glusterfs-server-3.6.0.42-1.el6rhs.x86_64                     7/12 
  Verifying  : glusterfs-libs-3.6.0.42-1.el6rhs.x86_64                       8/12 
  Verifying  : glusterfs-3.6.0.42-1.el6rhs.x86_64                            9/12 
  Verifying  : glusterfs-fuse-3.6.0.42-1.el6rhs.x86_64                      10/12 
  Verifying  : glusterfs-api-3.6.0.42-1.el6rhs.x86_64                       11/12 
  Verifying  : glusterfs-rdma-3.6.0.42-1.el6rhs.x86_64                      12/12 

Updated:
  glusterfs.x86_64 0:3.6.0.48-1.el6rhs                                            
  glusterfs-api.x86_64 0:3.6.0.48-1.el6rhs                                        
  glusterfs-fuse.x86_64 0:3.6.0.48-1.el6rhs                                       
  glusterfs-libs.x86_64 0:3.6.0.48-1.el6rhs                                       
  glusterfs-rdma.x86_64 0:3.6.0.48-1.el6rhs                                       
  glusterfs-server.x86_64 0:3.6.0.48-1.el6rhs                                     

Complete!
Comment 4 SATHEESARAN 2015-04-08 08:29:06 UTC 
I have seen the same issue while upgrading from RHS 3.0.3 ( glusterfs-3.6.0.42-1.el6rhs ) to RHS 3.0.4 ( glusterfs-3.6.0.53-1.el6rhs )

gluster cli package and glusterfs-geo-replication package got updated and gluster-cli is no longer responding

cli.log had the following:
--------------------------

[2015-04-08 13:41:40.973056] T [cli.c:264:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT
[2015-04-08 13:41:40.973080] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT
[2015-04-08 13:41:40.973090] I [socket.c:2263:socket_event_handler] 0-transport: disconnecting now
[2015-04-08 13:41:40.973111] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT
[2015-04-08 13:41:40.973358] T [rpc-clnt.c:1381:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
[2015-04-08 13:41:40.973427] T [rpc-clnt.c:1238:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
[2015-04-08 13:41:40.973807] T [socket.c:2732:socket_connect] (-->gluster(cli_cmd_submit+0x8e) [0x40b38e] (-->gluster(cli_submit_request+0xdb) [0x40a57b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x468) [0x35c880ece8]))) 0-glusterfs: connect () called on transport already connected
[2015-04-08 13:41:40.973929] T [rpc-clnt.c:1573:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 27) to rpc-transport (glusterfs)
[2015-04-08 13:41:40.973965] D [rpc-clnt-ping.c:231:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning
[2015-04-08 13:41:40.980219] T [rpc-clnt.c:660:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 27) from rpc-transport (glusterfs)
[2015-04-08 13:41:40.980279] D [cli-rpc-ops.c:6498:gf_cli_status_cbk] 0-cli: Received response to status cmd
[2015-04-08 13:41:40.980343] D [cli-cmd.c:388:cli_cmd_submit] 0-cli: Returning 0
[2015-04-08 13:41:40.980365] D [cli-rpc-ops.c:6761:gf_cli_status_volume] 0-cli: Returning: 0
[2015-04-08 13:41:40.980405] T [rpc-clnt.c:1381:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
[2015-04-08 13:41:40.980427] T [rpc-clnt.c:1238:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 144, payload: 80, rpc hdr: 64
[2015-04-08 13:41:40.980499] T [socket.c:2732:socket_connect] (-->gluster(cli_cmd_submit+0x8e) [0x40b38e] (-->gluster(cli_submit_request+0xdb) [0x40a57b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x468) [0x35c880ece8]))) 0-glusterfs: connect () called on transport already connected
[2015-04-08 13:41:40.980534] T [rpc-clnt.c:1573:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2 Program: Gluster CLI, ProgVers: 2, Proc: 27) to rpc-transport (glusterfs)
[2015-04-08 13:41:40.980555] D [rpc-clnt-ping.c:231:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning
[2015-04-08 13:41:40.991180] T [rpc-clnt.c:660:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x2 Program: Gluster CLI, ProgVers: 2, Proc: 27) from rpc-transport (glusterfs)
[2015-04-08 13:41:40.991256] D [cli-rpc-ops.c:6498:gf_cli_status_cbk] 0-cli: Received response to status cmd
[2015-04-08 13:41:40.991579] D [cli-cmd.c:388:cli_cmd_submit] 0-cli: Returning 0
[2015-04-08 13:41:40.991604] D [cli-rpc-ops.c:6761:gf_cli_status_volume] 0-cli: Returning: 0
[2015-04-08 13:41:40.991630] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2015-04-08 13:46:14.532670] E [mem-pool.c:353:mem_get0] (-->gluster(cli_rpc_init+0x2c) [0x40987c] (-->/usr/lib64/libglusterfs.so.0(dict_new+0xb) [0x35c801bd6b] (-->/usr/lib64/libglusterfs.so.0(get_new_dict_full+0x25) [0x35c801bc25]))) 0-mem-pool: invalid argument
[2015-04-08 13:46:21.163614] E [mem-pool.c:353:mem_get0] (-->gluster(cli_rpc_init+0x2c) [0x40987c] (-->/usr/lib64/libglusterfs.so.0(dict_new+0xb) [0x35c801bd6b] (-->/usr/lib64/libglusterfs.so.0(get_new_dict_full+0x25) [0x35c801bc25]))) 0-mem-pool: invalid argument
[2015-04-08 13:46:25.613171] E [mem-pool.c:353:mem_get0] (-->gluster(cli_rpc_init+0x2c) [0x40987c] (-->/usr/lib64/libglusterfs.so.0(dict_new+0xb) [0x35c801bd6b] (-->/usr/lib64/libglusterfs.so.0(get_new_dict_full+0x25) [0x35c801bc25]))) 0-mem-pool: invalid argument
[2015-04-08 13:46:29.868359] E [mem-pool.c:353:mem_get0] (-->gluster(cli_rpc_init+0x2c) [0x40987c] (-->/usr/lib64/libglusterfs.so.0(dict_new+0xb) [0x35c801bd6b] (-->/usr/lib64/libglusterfs.so.0(get_new_dict_full+0x25) [0x35c801bc25]))) 0-mem-pool: invalid argument
Comment 5 SATHEESARAN 2015-04-08 09:00:44 UTC 
Re-assigning the component as this problem has nothing to do with build.
Comment 6 SATHEESARAN 2015-04-13 07:06:13 UTC 
There is a customer, who hit this issue, while performing the update.
Refer - https://bugzilla.redhat.com/show_bug.cgi?id=1201221

Raising the severity to Urgent
Comment 7 Vimal Kumar 2015-04-14 14:06:26 UTC 
*** Bug 1201221 has been marked as a duplicate of this bug. ***