Bug 1498543

Summary: after upgrade from cns 3.5 to 3.6, heketi block-provisioning fails
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: krishnaram Karthick <kramdoss>
Component: rhgs-server-containerAssignee: Raghavendra Talur <rtalur>
Status: CLOSED ERRATA QA Contact: krishnaram Karthick <kramdoss>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: cns-3.6CC: akhakhar, annair, asriram, asrivast, hchiramm, jarrpa, madam, mliyazud, mzywusko, pprakash, rhs-bugs, rreddy, rtalur, sankarshan, vinug
Target Milestone: ---Keywords: TestBlocker
Target Release: CNS 3.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhgs-server-docker-3.3.0-362 Doc Type: Bug Fix
Doc Text:
On upgrading RHGS server containers, new files in /var/lib/glusterd/groups were not available which prevented creation of block hosting volume. This fix updates files in /var/lib/glusted/groups directory.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-11 07:00:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1445448    

Description krishnaram Karthick 2017-10-04 15:13:27 UTC
Description of problem:
After performing a cns upgrade from 3.5 to 3.6, gluster-block provisioning fails.

From the error message, looks like the gluster-block related volume option file ('/var/lib/glusterd/groups/gluster-block) is missing and hence the error.

Events:
  FirstSeen     LastSeen        Count   From                                                            SubObjectPath   Type            Reason   Message
  ---------     --------        -----   ----                                                            -------------   --------        ------   -------
  33s           33s             1       gluster.org/glusterblock a60cd6e0-a90f-11e7-9a01-0a580a810022                   Warning         ProvisioningFailed        Failed to provision volume with StorageClass "gluster-block":  failed to create volume: [heketi] error creating volume Unable to execute command on glusterfs-p1k16: Unable to open file '/var/lib/glusterd/groups/gluster-block'. Error: No such file or directory
  1m            30s             2       gluster.org/glusterblock a60cd6e0-a90f-11e7-9a01-0a580a810022                   Normal          Provisioning              External provisioner is provisioning volume for claim "storage-project/claim"
  1m            22s             25      persistentvolume-controller                                                     Normal          ExternalProvisioning      cannot find provisioner "gluster.org/glusterblock", expecting that a volume for the claim is provisioned either manually or via external software

Version-Release number of selected component (if applicable):
cns-deploy-5.0.0-53.el7rhgs.x86_64

How reproducible:
1/1

Steps to Reproduce:
1. upgrade cns from 3.5 to 3.6
2. provision gluster-block via dynamic provisioning


Actual results:
provisioning fails

Expected results:
provisioning should succeed

Additional info:

Comment 2 Humble Chirammal 2017-10-04 15:34:28 UTC
The block profile which is tried from Heketi to set on the block volume is not present in upgrade scenarios so the error :

Failed to provision volume with StorageClass "gluster-block":  failed to create volume: [heketi] error creating volume Unable to execute command on glusterfs-p1k16: Unable to open file '/var/lib/glusterd/groups/gluster-block'. Error: No such file or directory


2 options :

1) Either document the file entry and make a note to keep it in upgrade section of the doc.

2) Or supply it through the container.

Option 2 is the preferred the method:

Comment 6 krishnaram Karthick 2017-10-09 06:48:06 UTC
Verified the bug in build - cns-deploy-5.0.0-54.el7rhgs.x86_64

Fresh block device creation is successful after the upgrade.

Moving the bug to verified.

Comment 7 errata-xmlrpc 2017-10-11 07:00:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2877