Description of problem: Version-Release number of selected component (if applicable): glusterfs-3.4.0.34rhs-1.el6rhs.x86_64.rpm How reproducible: Always Steps to Reproduce: 1. Install Big Bend (glusterfs-3.4.0.33rhs-1.el6rhs.x86_64) 2. Yum update to glusterfs-3.4.0.34 Actual results: Update fails. Expected results: Update succeeds. Additional info: [root@ninja ~]# yum update Loaded plugins: aliases, changelog, downloadonly, fastestmirror, filter-data, keys, list-data, merge-conf, priorities, product-id, protectbase, : security, subscription-manager, tmprepo, tsflags, upgrade-helper, verify, versionlock This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Loading mirror speeds from cached hostfile Skipping filters plugin, no data 0 packages excluded due to repository protections Setting up Update Process Resolving Dependencies Skipping filters plugin, no data --> Running transaction check ---> Package glusterfs.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update ---> Package glusterfs-api.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs-api.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update ---> Package glusterfs-fuse.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs-fuse.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update ---> Package glusterfs-geo-replication.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs-geo-replication.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update ---> Package glusterfs-libs.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs-libs.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update ---> Package glusterfs-rdma.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs-rdma.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update ---> Package glusterfs-server.x86_64 0:3.4.0.33rhs-1.el6rhs will be updated ---> Package glusterfs-server.x86_64 0:3.4.0.34rhs-1.el6rhs will be an update --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================== Updating: glusterfs x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 1.8 M glusterfs-api x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 43 k glusterfs-fuse x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 88 k glusterfs-geo-replication x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 136 k glusterfs-libs x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 227 k glusterfs-rdma x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 43 k glusterfs-server x86_64 3.4.0.34rhs-1.el6rhs rhs2.1-server 612 k Transaction Summary ====================================================================================================================================================== Upgrade 7 Package(s) Total download size: 2.9 M Is this ok [y/N]: y Downloading Packages: (1/7): glusterfs-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 1.8 MB 00:00 (2/7): glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 43 kB 00:00 (3/7): glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 88 kB 00:00 (4/7): glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 136 kB 00:00 (5/7): glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 227 kB 00:00 (6/7): glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 43 kB 00:00 (7/7): glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64.rpm | 612 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------ Total 46 MB/s | 2.9 MB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Error in PREIN scriptlet in rpm package glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-libs-3.4.0.34rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-3.4.0.34rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-fuse-3.4.0.34rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-server-3.4.0.34rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-rdma-3.4.0.34rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64 ERROR: Please stop the volume.. installation cannot proceed Usage: $ gluster volume stop <volname> error: %pre(glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-api-3.4.0.34rhs-1.el6rhs glusterfs-api-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-api-3.4.0.33rhs-1.el6rhs.x86_64 1/14 glusterfs-geo-replication-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-geo-replication-3.4.0.33rhs-1.el6rhs.x86_64 2/14 glusterfs-server-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-server-3.4.0.33rhs-1.el6rhs.x86_64 3/14 Verifying : glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64 4/14 glusterfs-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-3.4.0.33rhs-1.el6rhs.x86_64 5/14 Verifying : glusterfs-3.4.0.34rhs-1.el6rhs.x86_64 6/14 Verifying : glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64 7/14 glusterfs-rdma-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-rdma-3.4.0.33rhs-1.el6rhs.x86_64 8/14 Verifying : glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64 9/14 Verifying : glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64 10/14 glusterfs-libs-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-libs-3.4.0.33rhs-1.el6rhs.x86_64 11/14 glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64 was supposed to be removed but is not! Verifying : glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64 12/14 Verifying : glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64 13/14 Verifying : glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64 14/14 Failed: glusterfs.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs.x86_64 0:3.4.0.34rhs-1.el6rhs glusterfs-api.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs-api.x86_64 0:3.4.0.34rhs-1.el6rhs glusterfs-fuse.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs-fuse.x86_64 0:3.4.0.34rhs-1.el6rhs glusterfs-geo-replication.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs-geo-replication.x86_64 0:3.4.0.34rhs-1.el6rhs glusterfs-libs.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs-libs.x86_64 0:3.4.0.34rhs-1.el6rhs glusterfs-rdma.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs-rdma.x86_64 0:3.4.0.34rhs-1.el6rhs glusterfs-server.x86_64 0:3.4.0.33rhs-1.el6rhs glusterfs-server.x86_64 0:3.4.0.34rhs-1.el6rhs Complete! [root@ninja ~]#
Bala, this is for handling issues with volume stop... need to remove it from glusterfs.spec in gist-git.
Harsha, as we introduce this behaviour in Big Bend, now the behaviour is treated as bug. We would need to short out this.
(In reply to Bala.FA from comment #4) > Harsha, as we introduce this behaviour in Big Bend, now the behaviour is > treated as bug. We would need to short out this. Are we already supporting rolling upgrades? i thought we had to come with much more concrete plan inside the code. Removing spec file checks - we will expose customers again for the issues seen with RHS 2.0 releases
Sayan, I would like to get your thoughts on about rolling upgrades.
Yes we need to support rolling upgrade for all subsequent updates of Big Bend.
https://code.engineering.redhat.com/gerrit/#/c/13821/ - RHS 2.1 Update 1 https://code.engineering.redhat.com/gerrit/#/c/13822/ - RHS 2.1 Fix sent
The description of in-service upgrades in https://bugzilla.redhat.com/show_bug.cgi?id=1009151 involves first stopping the gluster server, and only then running yum. Isn't this a test case error?
(In reply to Anthony Towns from comment #9) > The description of in-service upgrades in > https://bugzilla.redhat.com/show_bug.cgi?id=1009151 involves first stopping > the gluster server, and only then running yum. Isn't this a test case error? Now that you talk about its correct - i just provided a patch regarding the complain that it shouldn't happen. If rolling upgrades are what we going to supported then my patch stays since that is necessary to make sure people don't update without taking down 'glusterfs' server daemons. I will abandon it right way.
So here is what i see 'Rolling upgrades' where one server is updated at a given time --> i.e making sure to take that server down (server i.e "glusterfsd" not the OS itself) - client doesn't see disruption in i/o - caveat pure distributed, pure stripe volumes will not be accessible. Only supported mechanism is 'distributed+replicated' volume 'In place upgrades' where no servers are taken down and client doesn't see any disruption in i/o --> i.e making sure that no servers are down, no clients see i/o disruption -- good to have but futuristic. Current downstream RPM needs to verify for "Rolling Upgrades" where "glusterfsd" should be down - i will change the wording saying "pkill glusterfsd please.." rather than "gluster volume stop <volname>" since latter seems misleading.. -Harsha
Agree. "ERROR: Please stop the volume.. installation cannot proceed" is indeed misleading. Thanks for the explanation Harsha and Aj.
(In reply to Gowrishankar Rajaiyan from comment #12) > Agree. > > "ERROR: Please stop the volume.. installation cannot proceed" is indeed > misleading. > > Thanks for the explanation Harsha and Aj. https://code.engineering.redhat.com/gerrit/#/c/13982/ https://code.engineering.redhat.com/gerrit/#/c/13981/ Here are the patches sent for review - this should be merged since the "messaging" becomes correct in accordance with rolling upgrade procedure listed at - https://bugzilla.redhat.com/show_bug.cgi?id=1009151
The patch checks for "pidof glusterfsd" and generalizes with an output as "ERROR: Please kill gluster processes.. installation cannot proceed", which is not true if the customer has a pure distribute volume. Even though we would document that only replicated or distributed-replicated would be supported for such rolling upgrades, there are chances of user killing gluster processes even for a distribute volume since our message during upgrade is generic. IMHO, in %pre section of spec file we should have something like: if [[ $distribute_vol_type -eq 0 && $volume_status -eq 1 ]] ; then echo "In-service rolling upgrade requires that there exist no started distribute volume(s)." echo "Please stop distribute volume(s) before proceeding." exit 1; fi Even better to have it interactive when the above criteria is met: echo "Updating glusterfs requires its processes to be killed. This action does NOT incur downtime." (since as per the above check we are sure there exists no distribute volume which is started) echo "Ensure to wait for the upgraded server to finish healing before proceeding. Refer install guide for more details". And stop glusterd and kill gluster processes as part of %pre. for example: <snip> #/bin/bash pidof -c -o %PPID -x glusterfsd &>/dev/null if [ $? -eq 0 ]; then pushd . > /dev/null 2>&1 for volume in /var/lib/glusterd/vols/*; do cd $volume; distribute=`grep '^type=' info | awk -F'=' '{print $2}'` volume_started=`grep '^status=' info | awk -F'=' '{print $2}'` if [[ $distribute -eq 0 && $volume_started -eq 1 ]] ; then echo "In-service rolling upgrade requires that there exist no started distribute volume(s)." echo "Please stop distribute volume(s) before proceeding." exit 1; fi done popd > /dev/null 2>&1 echo "Updating glusterfs requires its processes to be killed. This action does NOT incur downtime." echo "Ensure to wait for the upgraded server to finish healing before proceeding. Refer install guide for more details" while true; do read -p "Are you sure you want to stop glusterd and kill glusterfs processes (yes/no)? " yn case $yn in yes ) service glusterd stop; pkill glusterfs; pkill glusterfsd; break;; no ) exit 1;; *) echo "Please answer yes or no.";; esac done fi </snip>
> > Even better to have it interactive when the above criteria is met: > echo "Updating glusterfs requires its processes to be killed. This action > does NOT incur downtime." (since as per the above check we are sure there > exists no distribute volume which is started) > echo "Ensure to wait for the upgraded server to finish healing before > proceeding. Refer install guide for more details". And stop glusterd and > kill gluster processes as part of %pre. > > > for example: > <snip> > #/bin/bash > > pidof -c -o %PPID -x glusterfsd &>/dev/null > > if [ $? -eq 0 ]; then > > pushd . > /dev/null 2>&1 > for volume in /var/lib/glusterd/vols/*; do cd $volume; > > distribute=`grep '^type=' info | awk -F'=' '{print $2}'` > volume_started=`grep '^status=' info | awk -F'=' '{print $2}'` > > if [[ $distribute -eq 0 && $volume_started -eq 1 ]] ; then > echo "In-service rolling upgrade requires that there exist no started > distribute volume(s)." > echo "Please stop distribute volume(s) before proceeding." > exit 1; > fi > done > popd > /dev/null 2>&1 > > echo "Updating glusterfs requires its processes to be killed. This action > does NOT incur downtime." > echo "Ensure to wait for the upgraded server to finish healing before > proceeding. Refer install guide for more details" > > while true; do > > read -p "Are you sure you want to stop glusterd and kill glusterfs > processes (yes/no)? " yn > > case $yn in > yes ) service glusterd stop; pkill glusterfs; pkill glusterfsd; break;; > no ) exit 1;; > *) echo "Please answer yes or no.";; > esac > done > fi > </snip> This is interesting, appreciate your inputs - i will incorporate this and send across a new patch
For U1 this bug shall be verified for patch mentioned in comment #13.
Running Transaction Error in PREIN scriptlet in rpm package glusterfs-libs-3.4.0.36rhs-1.el6rhs.x86_64 ERROR: Please kill gluster processes.. installation cannot proceed Usage: $ pkill gluster error: %pre(glusterfs-libs-3.4.0.36rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-libs-3.4.0.36rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-3.4.0.36rhs-1.el6rhs.x86_64 ERROR: Please kill gluster processes.. installation cannot proceed Usage: $ pkill gluster error: %pre(glusterfs-3.4.0.36rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-3.4.0.36rhs-1.el6rhs Updating : gluster-swift-1.8.0-6.12memc.el6rhs.noarch 3/24 Error in PREIN scriptlet in rpm package glusterfs-fuse-3.4.0.36rhs-1.el6rhs.x86_64 ERROR: Please kill gluster processes.. installation cannot proceed Usage: $ pkill gluster error: %pre(glusterfs-fuse-3.4.0.36rhs-1.el6rhs.x86_64) scriptlet failed, exit status 1 error: install: %pre scriptlet failed (2), skipping glusterfs-fuse-3.4.0.36rhs-1.el6rhs Error in PREIN scriptlet in rpm package glusterfs-geo-replication-3.4.0.36rhs-1.el6rhs.x86_64