Description of problem: On a 3 node OCS setup, management encryption was enabled using the steps mentioned in https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html-single/operations_guide/index#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Enabling_Encryption-New_Setup After the deployment was successful and encryption was enabled successfully as seen from the following logs: [2018-10-15 09:02:43.093752] I [socket.c:4306:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED [2018-10-15 09:02:43.093790] I [socket.c:4309:socket_init] 0-socket.management: SSL support for glusterd is ENABLED However when I tried creating a block device, it failed with the following error: # heketi-cli blockvolume create --size=2 Error: Unable to open file '/var/lib/glusterd/groups/gluster-block'. Error: No such file or directory Version-Release number of selected component (if applicable): oc v3.11.16 kubernetes v1.11.0+d4cacc0 glusterfs-libs-3.12.2-18.1.el7rhgs.x86_64 glusterfs-3.12.2-18.1.el7rhgs.x86_64 glusterfs-api-3.12.2-18.1.el7rhgs.x86_64 python2-gluster-3.12.2-18.1.el7rhgs.x86_64 glusterfs-fuse-3.12.2-18.1.el7rhgs.x86_64 glusterfs-server-3.12.2-18.1.el7rhgs.x86_64 gluster-block-0.2.1-27.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-18.1.el7rhgs.x86_64 glusterfs-cli-3.12.2-18.1.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-18.1.el7rhgs.x86_64 heketi-client-7.0.0-14.el7rhgs.x86_64 heketi-7.0.0-14.el7rhgs.x86_64 How reproducible: 2/2 Steps to Reproduce: 1. Configure management encryption on a fresh deployment of OCS using steps mentioned here: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html-single/operations_guide/index#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Enabling_Encryption-New_Setup 2. Once deployment is successful, create a block device using heketi-cli command: heketi-cli blockvolume create --size=2 Actual results: Block device creation fails with the following error: Error: Unable to open file '/var/lib/glusterd/groups/gluster-block'. Error: No such file or directory Expected results: Block device creation should be successful
providing qa_ack to take this bug into 3.11.1
As the error message says, the volume create failed because gluster-block group option file did not exist on the node/pod. The script does the following things 1. it always copies hooks and groups dir 2. it copies all dirs if there are no files in the bind mount 3. it checks for existence of the files in bind mount using ls The sosreports attached don't have any logging from gluster-setup.sh script. Also, they have the hooks dir but not the groups dir in the sosreport. I think it is still a case of copy logic gone wrong in the script.
(In reply to Raghavendra Talur from comment #9) > As the error message says, the volume create failed because gluster-block > group option file did not exist on the node/pod. > > The script does the following things > 1. it always copies hooks and groups dir Thanks Talur. identified issue while doing groups directory copying. So, we have in code: << cp -r ${i}_bkp/hooks/* ${i}/hooks/ << cp -r ${i}_bkp/groups/* ${i}/groups While doing hooks directory copy from backup, it worked fine as the content it had is a directory (named "1"). While doing groups directory copy from backup, it *fails* as the content it had is files. copy operation fails with "cp: target '/tmp/groups/' is not a directory". Better way to copy here is using: cp -a -f ${i}_bkp/groups/* ${i}/groups (should be done for hooks as well) I will update this in the script, gluster-setup.sh
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0287