Bug 1639253

Summary: When OCS is deployed with management encryption enabled, block device creation failures are observed.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rachael <rgeorge>
Component: rhgs-server-containerAssignee: Saravanakumar <sarumuga>
Status: CLOSED ERRATA QA Contact: vinutha <vinug>
Severity: high Docs Contact:
Priority: high    
Version: ocs-3.11CC: hchiramm, knarra, kramdoss, madam, rhs-bugs, rtalur, sankarshan, sselvan, vinug
Target Milestone: ---Keywords: ZStream
Target Release: OCS 3.11.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ocs/rhgs-server-rhel7:3.11.1-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-07 04:12:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1662298    
Bug Blocks: 1640001, 1641915, 1644154    

Description Rachael 2018-10-15 11:55:29 UTC
Description of problem:

On a 3 node OCS setup, management encryption was enabled using the steps mentioned in 
https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html-single/operations_guide/index#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Enabling_Encryption-New_Setup

After the deployment was successful and encryption was enabled successfully as seen from the following logs:

[2018-10-15 09:02:43.093752] I [socket.c:4306:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2018-10-15 09:02:43.093790] I [socket.c:4309:socket_init] 0-socket.management: SSL support for glusterd is ENABLED

However when I tried creating a block device, it failed with the following error:

# heketi-cli blockvolume create --size=2
Error: Unable to open file '/var/lib/glusterd/groups/gluster-block'. Error: No such file or directory


Version-Release number of selected component (if applicable):
oc v3.11.16
kubernetes v1.11.0+d4cacc0

glusterfs-libs-3.12.2-18.1.el7rhgs.x86_64
glusterfs-3.12.2-18.1.el7rhgs.x86_64
glusterfs-api-3.12.2-18.1.el7rhgs.x86_64
python2-gluster-3.12.2-18.1.el7rhgs.x86_64
glusterfs-fuse-3.12.2-18.1.el7rhgs.x86_64
glusterfs-server-3.12.2-18.1.el7rhgs.x86_64
gluster-block-0.2.1-27.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-18.1.el7rhgs.x86_64
glusterfs-cli-3.12.2-18.1.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-18.1.el7rhgs.x86_64

heketi-client-7.0.0-14.el7rhgs.x86_64
heketi-7.0.0-14.el7rhgs.x86_64


How reproducible: 2/2


Steps to Reproduce:
1. Configure management encryption on a fresh deployment of OCS using steps mentioned here:

https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html-single/operations_guide/index#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Enabling_Encryption-New_Setup

2. Once deployment is successful, create a block device using heketi-cli command:

heketi-cli blockvolume create --size=2


Actual results:
Block device creation fails with the following error:

Error: Unable to open file '/var/lib/glusterd/groups/gluster-block'. Error: No such file or directory

Expected results:
Block device creation should be successful

Comment 4 krishnaram Karthick 2018-11-02 10:02:39 UTC
providing qa_ack to take this bug into 3.11.1

Comment 9 Raghavendra Talur 2018-11-30 11:20:54 UTC
As the error message says, the volume create failed because gluster-block group option file did not exist on the node/pod.

The script does the following things
1. it always copies hooks and groups dir
2. it copies all dirs if there are no files in the bind mount
3. it checks for existence of the files in bind mount using ls

The sosreports attached don't have any logging from gluster-setup.sh script.

Also, they have the hooks dir but not the groups dir in the sosreport.

I think it is still a case of copy logic gone wrong in the script.

Comment 10 Saravanakumar 2018-11-30 12:56:49 UTC
(In reply to Raghavendra Talur from comment #9)
> As the error message says, the volume create failed because gluster-block
> group option file did not exist on the node/pod.
> 
> The script does the following things
> 1. it always copies hooks and groups dir

Thanks Talur. identified issue while doing groups directory copying.

So, we have in code:
<<        cp -r ${i}_bkp/hooks/* ${i}/hooks/
<<        cp -r ${i}_bkp/groups/* ${i}/groups


While doing hooks directory copy from backup, it worked fine as the content it had is a directory (named "1").

While doing groups directory copy from backup, it *fails* as the content it had is files.  
copy operation fails with "cp: target '/tmp/groups/' is not a directory". 

Better way to copy here is using: cp -a -f ${i}_bkp/groups/* ${i}/groups 
(should be done for hooks as well)

I will update this in the script, gluster-setup.sh

Comment 21 errata-xmlrpc 2019-02-07 04:12:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0287