Bug 1016385 - Rolling upgrade using yum fails from Big Bend to Big Bend Update 1
Rolling upgrade using yum fails from Big Bend to Big Bend Update 1
Status: CLOSED CURRENTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: build (Show other bugs)
2.1
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Pranith Kumar K
Gowrishankar Rajaiyan
: ZStream
Depends On:
Blocks: 1022822
  Show dependency treegraph
 
Reported: 2013-10-08 01:29 EDT by Gowrishankar Rajaiyan
Modified: 2013-12-09 11:45 EST (History)
8 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.36rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1022822 (view as bug list)
Environment:
Last Closed: 2013-12-09 11:45:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Gowrishankar Rajaiyan 2013-10-08 01:29:01 EDT
Description of problem:


Version-Release number of selected component (if applicable):
glusterfs-3.4.0.34rhs-1.el6rhs.x86_64.rpm

How reproducible: Always


Steps to Reproduce:
1. Install Big Bend (glusterfs-3.4.0.33rhs-1.el6rhs.x86_64)
2. Yum update to glusterfs-3.4.0.34

Actual results: Update fails.


Expected results: Update succeeds.


Additional info:
[root@ninja ~]# yum update
Loaded plugins: aliases, changelog, downloadonly, fastestmirror, filter-data, keys, list-data, merge-conf, priorities, product-id, protectbase,
              : security, subscription-manager, tmprepo, tsflags, upgrade-helper, verify, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
Skipping filters plugin, no data
0 packages excluded due to repository protections
Setting up Update Process
Resolving Dependencies
Skipping filters plugin, no data
--> Running transaction check
---> Package glusterfs.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
---> Package glusterfs-api.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs-api.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
---> Package glusterfs-fuse.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs-fuse.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
---> Package glusterfs-geo-replication.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs-geo-replication.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
---> Package glusterfs-libs.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs-libs.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
---> Package glusterfs-rdma.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs-rdma.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
---> Package glusterfs-server.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated
---> Package glusterfs-server.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================
 Package                                     Arch                     Version                                   Repository                       Size
======================================================================================================================================================
Updating:
 glusterfs                                   x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                   1.8 M
 glusterfs-api                               x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                    43 k
 glusterfs-fuse                              x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                    88 k
 glusterfs-geo-replication                   x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                   136 k
 glusterfs-libs                              x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                   227 k
 glusterfs-rdma                              x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                    43 k
 glusterfs-server                            x86_64                   3.4.0.34rhs-1.el6rhs                      rhs2.1-server                   612 k

Transaction Summary
======================================================================================================================================================
Upgrade       7 Package(s)

Total download size: 2.9 M
Is this ok [y/N]: y
Downloading Packages:
(1/7): glusterfs-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                                               | 1.8 MB     00:00     
(2/7): glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                                           |  43 kB     00:00     
(3/7): glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                                          |  88 kB     00:00     
(4/7): glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                               | 136 kB     00:00     
(5/7): glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                                          | 227 kB     00:00     
(6/7): glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                                          |  43 kB     00:00     
(7/7): glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64.rpm                                                                        | 612 kB     00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                  46 MB/s | 2.9 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Error in PREIN scriptlet in rpm package glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-libs-3.4.0.34rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-3.4.0.34rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-fuse-3.4.0.34rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-server-3.4.0.34rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-rdma-3.4.0.34rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64
ERROR: Please stop the volume.. installation cannot proceed
Usage: $ gluster volume stop <volname> 
error: %pre(glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-api-3.4.0.34rhs-1.el6rhs
glusterfs-api-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-api-3.4.0.33rhs-1.el6rhs.x86_64                                                                                         1/14 
glusterfs-geo-replication-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-geo-replication-3.4.0.33rhs-1.el6rhs.x86_64                                                                             2/14 
glusterfs-server-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-server-3.4.0.33rhs-1.el6rhs.x86_64                                                                                      3/14 
  Verifying  : glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64                                                                                        4/14 
glusterfs-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-3.4.0.33rhs-1.el6rhs.x86_64                                                                                             5/14 
  Verifying  : glusterfs-3.4.0.34rhs-1.el6rhs.x86_64                                                                                             6/14 
  Verifying  : glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64                                                                                        7/14 
glusterfs-rdma-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-rdma-3.4.0.33rhs-1.el6rhs.x86_64                                                                                        8/14 
  Verifying  : glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64                                                                                      9/14 
  Verifying  : glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64                                                                                       10/14 
glusterfs-libs-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-libs-3.4.0.33rhs-1.el6rhs.x86_64                                                                                       11/14 
glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not!
  Verifying  : glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64                                                                                       12/14 
  Verifying  : glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64                                                                                        13/14 
  Verifying  : glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64                                                                            14/14 

Failed:
  glusterfs.x86_64 0:3.4.0.33rhs-1.el6rhs                                   glusterfs.x86_64 0:3.4.0.34rhs-1.el6rhs                                  
  glusterfs-api.x86_64 0:3.4.0.33rhs-1.el6rhs                               glusterfs-api.x86_64 0:3.4.0.34rhs-1.el6rhs                              
  glusterfs-fuse.x86_64 0:3.4.0.33rhs-1.el6rhs                              glusterfs-fuse.x86_64 0:3.4.0.34rhs-1.el6rhs                             
  glusterfs-geo-replication.x86_64 0:3.4.0.33rhs-1.el6rhs                   glusterfs-geo-replication.x86_64 0:3.4.0.34rhs-1.el6rhs                  
  glusterfs-libs.x86_64 0:3.4.0.33rhs-1.el6rhs                              glusterfs-libs.x86_64 0:3.4.0.34rhs-1.el6rhs                             
  glusterfs-rdma.x86_64 0:3.4.0.33rhs-1.el6rhs                              glusterfs-rdma.x86_64 0:3.4.0.34rhs-1.el6rhs                             
  glusterfs-server.x86_64 0:3.4.0.33rhs-1.el6rhs                            glusterfs-server.x86_64 0:3.4.0.34rhs-1.el6rhs                           

Complete!
[root@ninja ~]#
Comment 3 Amar Tumballi 2013-10-08 04:13:48 EDT
Bala, this is for handling issues with volume stop... need to remove it from glusterfs.spec in gist-git.
Comment 4 Bala.FA 2013-10-08 04:34:44 EDT
Harsha, as we introduce this behaviour in Big Bend, now the behaviour is treated as bug.  We would need to short out this.
Comment 5 Harshavardhana 2013-10-08 06:12:12 EDT
(In reply to Bala.FA from comment #4)
> Harsha, as we introduce this behaviour in Big Bend, now the behaviour is
> treated as bug.  We would need to short out this.

Are we already supporting rolling upgrades? i thought we had to come with much more concrete plan inside the code. Removing spec file checks - we will expose customers again for the issues seen with RHS 2.0 releases
Comment 6 Bala.FA 2013-10-08 07:20:59 EDT
Sayan, I would like to get your thoughts on about rolling upgrades.
Comment 7 Sayan Saha 2013-10-08 22:47:10 EDT
Yes we need to support rolling upgrade for all subsequent updates of Big Bend.
Comment 8 Harshavardhana 2013-10-08 23:48:20 EDT
https://code.engineering.redhat.com/gerrit/#/c/13821/ - RHS 2.1 Update 1

https://code.engineering.redhat.com/gerrit/#/c/13822/ - RHS 2.1 

Fix sent
Comment 9 Anthony Towns 2013-10-11 01:21:41 EDT
The description of in-service upgrades in https://bugzilla.redhat.com/show_bug.cgi?id=1009151 involves first stopping the gluster server, and only then running yum. Isn't this a test case error?
Comment 10 Harshavardhana 2013-10-11 02:58:03 EDT
(In reply to Anthony Towns from comment #9)
> The description of in-service upgrades in
> https://bugzilla.redhat.com/show_bug.cgi?id=1009151 involves first stopping
> the gluster server, and only then running yum. Isn't this a test case error?

Now that you talk about its correct - i just provided a patch regarding the complain that it shouldn't happen. If rolling upgrades are what we going to supported then my patch stays since that is necessary to make sure people don't update without taking down 'glusterfs' server daemons.

I will abandon it right way.
Comment 11 Harshavardhana 2013-10-11 03:09:20 EDT
So here is what i see

'Rolling upgrades' where one server is updated at a given time --> i.e making sure to take that server down (server i.e "glusterfsd" not the OS itself) - client doesn't see disruption in i/o - caveat pure distributed, pure stripe volumes will not be accessible. Only supported mechanism is 'distributed+replicated' volume

'In place upgrades' where no servers are taken down and client doesn't see any disruption in i/o --> i.e making sure that no servers are down, no clients see i/o disruption -- good to have but futuristic.

Current downstream RPM needs to verify for "Rolling Upgrades" where "glusterfsd" should be down - i will change the wording saying "pkill glusterfsd please.." rather than "gluster volume stop <volname>" since latter seems misleading..

-Harsha
Comment 12 Gowrishankar Rajaiyan 2013-10-11 04:42:56 EDT
Agree. 

"ERROR: Please stop the volume.. installation cannot proceed" is indeed misleading.

Thanks for the explanation Harsha and Aj.
Comment 13 Harshavardhana 2013-10-14 17:23:50 EDT
(In reply to Gowrishankar Rajaiyan from comment #12)
> Agree. 
> 
> "ERROR: Please stop the volume.. installation cannot proceed" is indeed
> misleading.
> 
> Thanks for the explanation Harsha and Aj.

https://code.engineering.redhat.com/gerrit/#/c/13982/

https://code.engineering.redhat.com/gerrit/#/c/13981/

Here are the patches sent for review - this should be merged since the "messaging" becomes correct in accordance with rolling upgrade procedure listed at - https://bugzilla.redhat.com/show_bug.cgi?id=1009151
Comment 14 Gowrishankar Rajaiyan 2013-10-16 02:52:51 EDT
The patch checks for "pidof glusterfsd" and generalizes with an output as "ERROR: Please kill gluster processes.. installation cannot proceed", which is not true if the customer has a pure distribute volume.

Even though we would document that only replicated or distributed-replicated would be supported for such rolling upgrades, there are chances of user killing gluster processes even for a distribute volume since our message during upgrade is generic.

IMHO, in %pre section of spec file we should have something like:

if [[ $distribute_vol_type -eq 0 && $volume_status -eq 1 ]] ; then
echo "In-service rolling upgrade requires that there exist no started distribute volume(s)."
echo "Please stop distribute volume(s) before proceeding."
exit 1;
fi

Even better to have it interactive when the above criteria is met:
echo "Updating glusterfs requires its processes to be killed. This action does NOT incur downtime." (since as per the above check we are sure there exists no distribute volume which is started)
echo "Ensure to wait for the upgraded server to finish healing before proceeding. Refer install guide for more details". And stop glusterd and kill gluster processes as part of %pre.


for example:
<snip>
#/bin/bash

pidof -c -o %PPID -x glusterfsd &>/dev/null

if [ $? -eq 0 ]; then

pushd . > /dev/null 2>&1
for volume in /var/lib/glusterd/vols/*; do cd $volume;

	distribute=`grep '^type=' info | awk -F'=' '{print $2}'`
	volume_started=`grep '^status=' info | awk -F'=' '{print $2}'`

	if [[ $distribute -eq 0 && $volume_started -eq 1 ]] ; then
		echo "In-service rolling upgrade requires that there exist no started distribute volume(s)."
		echo "Please stop distribute volume(s) before proceeding."
		exit 1;
	fi
done
popd > /dev/null 2>&1

	echo "Updating glusterfs requires its processes to be killed. This action does NOT incur downtime."
	echo "Ensure to wait for the upgraded server to finish healing before proceeding. Refer install guide for more details"

	while true; do

		read -p "Are you sure you want to stop glusterd and kill glusterfs processes (yes/no)? " yn

		case $yn in
			yes ) service glusterd stop; pkill glusterfs; pkill glusterfsd; break;;
			no ) exit 1;;
			*) echo "Please answer yes or no.";;
 		esac
	done
fi
</snip>
Comment 15 Harshavardhana 2013-10-16 05:43:31 EDT
> 
> Even better to have it interactive when the above criteria is met:
> echo "Updating glusterfs requires its processes to be killed. This action
> does NOT incur downtime." (since as per the above check we are sure there
> exists no distribute volume which is started)
> echo "Ensure to wait for the upgraded server to finish healing before
> proceeding. Refer install guide for more details". And stop glusterd and
> kill gluster processes as part of %pre.
> 
> 
> for example:
> <snip>
> #/bin/bash
> 
> pidof -c -o %PPID -x glusterfsd &>/dev/null
> 
> if [ $? -eq 0 ]; then
> 
> pushd . > /dev/null 2>&1
> for volume in /var/lib/glusterd/vols/*; do cd $volume;
> 
> 	distribute=`grep '^type=' info | awk -F'=' '{print $2}'`
> 	volume_started=`grep '^status=' info | awk -F'=' '{print $2}'`
> 
> 	if [[ $distribute -eq 0 && $volume_started -eq 1 ]] ; then
> 		echo "In-service rolling upgrade requires that there exist no started
> distribute volume(s)."
> 		echo "Please stop distribute volume(s) before proceeding."
> 		exit 1;
> 	fi
> done
> popd > /dev/null 2>&1
> 
> 	echo "Updating glusterfs requires its processes to be killed. This action
> does NOT incur downtime."
> 	echo "Ensure to wait for the upgraded server to finish healing before
> proceeding. Refer install guide for more details"
> 
> 	while true; do
> 
> 		read -p "Are you sure you want to stop glusterd and kill glusterfs
> processes (yes/no)? " yn
> 
> 		case $yn in
> 			yes ) service glusterd stop; pkill glusterfs; pkill glusterfsd; break;;
> 			no ) exit 1;;
> 			*) echo "Please answer yes or no.";;
>  		esac
> 	done
> fi
> </snip>

This is interesting, appreciate your inputs - i will incorporate this and send across a new patch
Comment 18 Gowrishankar Rajaiyan 2013-10-24 00:45:43 EDT
For U1 this bug shall be verified for patch mentioned in comment #13.
Comment 19 Gowrishankar Rajaiyan 2013-10-24 04:26:56 EDT
Running Transaction
Error in PREIN scriptlet in rpm package glusterfs-libs-3.4.0.36rhs-1.el6rhs.x86_64
ERROR: Please kill gluster processes.. installation cannot proceed
Usage: $ pkill gluster 
error: %pre(glusterfs-libs-3.4.0.36rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-libs-3.4.0.36rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-3.4.0.36rhs-1.el6rhs.x86_64
ERROR: Please kill gluster processes.. installation cannot proceed
Usage: $ pkill gluster 
error: %pre(glusterfs-3.4.0.36rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-3.4.0.36rhs-1.el6rhs
  Updating   : gluster-swift-1.8.0-6.12memc.el6rhs.noarch                                                                                        3/24 
Error in PREIN scriptlet in rpm package glusterfs-fuse-3.4.0.36rhs-1.el6rhs.x86_64
ERROR: Please kill gluster processes.. installation cannot proceed
Usage: $ pkill gluster 
error: %pre(glusterfs-fuse-3.4.0.36rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping glusterfs-fuse-3.4.0.36rhs-1.el6rhs
Error in PREIN scriptlet in rpm package glusterfs-geo-replication-3.4.0.36rhs-1.el6rhs.x86_64

Note You need to log in before you can comment on or make changes to this bug.