Bug 1444820

Summary: Script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh failing
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Cal Calhoun <ccalhoun>
Component: coreAssignee: Anoop C S <anoopcs>
Status: CLOSED ERRATA QA Contact: Vivek Das <vdas>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: amukherj, anoopcs, bkunal, ccalhoun, gdeschner, madam, rhs-bugs, rhs-smb, sankarshan, sheggodu, storage-qa-internal, vdas
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 06:32:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1408949, 1472361, 1474007, 1503135    

Description Cal Calhoun 2017-04-24 11:45:29 UTC
Description of problem:

Error messages in glusterd.log when starting volume with enabled CIFS support:

glusterd.log:[2017-04-18 07:59:52.086854] E [run.c:191:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd0de2) [0x7f0d1b3b6de2] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd07f6) [0x7f0d1b3b67f6] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f0d26bdea95] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=brand_cifs_poc --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd

Version-Release number of selected component (if applicable):

Red Hat Enterprise Linux Server release 7.3 (Maipo)
Red Hat Gluster Storage Server 3.2.0

gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-3.8.4-18.el7rhgs.x86_64
glusterfs-api-3.8.4-18.el7rhgs.x86_64
glusterfs-cli-3.8.4-18.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-18.el7rhgs.x86_64
glusterfs-fuse-3.8.4-18.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-18.el7rhgs.x86_64
glusterfs-libs-3.8.4-18.el7rhgs.x86_64
glusterfs-rdma-3.8.4-18.el7rhgs.x86_64
glusterfs-server-3.8.4-18.el7rhgs.x86_64
samba-4.4.6-4.el7rhgs.x86_64
samba-client-4.4.6-4.el7rhgs.x86_64
samba-client-libs-4.4.6-4.el7rhgs.x86_64
samba-common-4.4.6-4.el7rhgs.noarch
samba-common-libs-4.4.6-4.el7rhgs.x86_64
samba-common-tools-4.4.6-4.el7rhgs.x86_64
samba-libs-4.4.6-4.el7rhgs.x86_64
samba-vfs-glusterfs-4.4.6-4.el7rhgs.x86_64

How reproducible:

Consistently

Steps to Reproduce:

Script can be manually executed with the same parameters as displayed in the error message:

1. # /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=brand_cifs_poc --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
ame: unrecognized option '--first=no'
ame: unrecognized option '--version=1'
ame: unrecognized option '--volume-op=start'
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh: line 105: [: =: unary operator expected

Actual results:

Script errors with the indicated message.

Expected results:

Would expect the script to run correctly with the supplied parameters.

Additional info:

1. Script is called with unexpected parameters:

first, version, volume-op are not defined

function parse_args () {
        ARGS=$(getopt -l $OPTSPEC  -name $PROGNAME $@)
        eval set -- "$ARGS"

        while true; do
            case $1 in
                --volname)
                    shift
                    VOL=$1
                    ;;
                --gd-workdir)
                    shift
                    GLUSTERD_WORKDIR=$1
                    ;;
                *)
                    shift
                    break
                    ;;
            esac
            shift
        done
}

2. Syntax issues:

if [ $(get_smb "$VOL") = "disable" ]; then
        exit 0
fi

Comment 8 Anoop C S 2017-04-27 11:30:15 UTC
Hi Cal,

I don't think the above mentioned issues with respect to S30samba-start.sh is the real cause of failure to execute the script. Because those errors will not interrupt the execution of remainder of script. So I have the following queries:

[1] Can you please verify whether smb.conf contains a section called [gluster-brand_cifs_poc] or not?

[2] Check for any errors in smbd logs i.e, inside /var/log/samba/log.smbd

[3] Immediately after executing the script manually, what's the exit status?
# echo $?

I strongly suspect for failures in sighup function inside hook script where we either send HUP signal to smbd or condrestart smb.

Comment 9 Pan Ousley 2017-05-09 14:29:08 UTC
Hi Anoop,


The customer wanted to clarify that the issue is not with the script failing, but rather the following:

"
script is working ok, the thing is that 

1) gluster calls this helper script it wit non-existent parameters:

[root@rhs03 glusterfs]# /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=brand_cifs_poc --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
ame: unrecognized option '--first=no'
ame: unrecognized option '--version=1'
ame: unrecognized option '--volume-op=start'

2) script itself has syntax issues:

Error	ShellCheck	Double quote array expansions to avoid re-splitting elements. [SC2068]	32:48
Error	ShellCheck	Double quote array expansions to avoid re-splitting elements. [SC2068]	104:12
Warning	ShellCheck	This { is literal. Check expression (missing ;/\n?) or quote it. [SC1083]	60:36
Warning	ShellCheck	This } is literal. Check expression (missing ;/\n?) or quote it. [SC1083]	60:47
Warning	ShellCheck	This { is literal. Check expression (missing ;/\n?) or quote it. [SC1083]	61:41
Warning	ShellCheck	This } is literal. Check expression (missing ;/\n?) or quote it. [SC1083]	61:52


Script itself is working, but due to above reasons, useless & misleading error messages are present in glusterd logs:

glusterd.log:[2017-04-18 07:59:52.086854] E [run.c:191:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd0de2) [0x7f0d1b3b6de2] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd07f6) [0x7f0d1b3b67f6] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f0d26bdea95] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=brand_cifs_poc --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
"

Any thoughts on this, or is there any more information that you need us to provide?

Thank you.

Comment 10 Anoop C S 2017-05-10 09:47:56 UTC
Hi Pamela,

(In reply to Pamela Ousley from comment #9)
> Script itself is working, but due to above reasons, useless & misleading
> error messages are present in glusterd logs:
> 
> glusterd.log:[2017-04-18 07:59:52.086854] E [run.c:191:runner_log]
> (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd0de2)
> [0x7f0d1b3b6de2]
> -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd07f6)
> [0x7f0d1b3b67f6] -->/lib64/libglusterfs.so.0(runner_log+0x115)
> [0x7f0d26bdea95] ) 0-management: Failed to execute script:
> /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
> --volname=brand_cifs_poc --first=no --version=1 --volume-op=start
> --gd-workdir=/var/lib/glusterd

The above error log entry stating the failure to execute the script is not because of the reasons mentioned by the customer. That's why I asked to provide the output of `echo $?` after executing the script manually from outside(see [3] in comment #8). If exit status is non-zero then that itself explains the cause. Or if exit status is zero then we need to see why that log entry is still seen in glusterd logs.

Comment 12 Anoop C S 2017-05-12 08:53:17 UTC
Hi Cal,

Ok. Things are more clearer now. Still I can assure you that gluster does not check for std{out,err} errors/warnings. As far as I can understand, we determine the outcome of execution of hook scripts solely based on the exit status. I have a vague idea of what might have happened.

If by any chance customer had tried to start the volume before installing the required samba packages, S30samba-start.sh is expected to be shown as "Failed to execute" in glusterd logs. In order to make sure that my guess is correct or not I would like to have the latest glusterd logs after trying a stop and start of the volume in question(assuming all required samba packages are installed i.e, smbd binary, smb service init file etc).

Comment 23 Cal Calhoun 2017-10-06 14:11:08 UTC
@Anoop, I have queried the customer for a list of the currently installed samba and gluster packages as well as glusterd logs after a volume stop/start.  This should show us exactly what package versions are installed and demonstrate whether the logging is still occurring.
-Cal-

Comment 25 Cal Calhoun 2017-10-13 12:44:40 UTC
@Anoop: From the customer:

yes, that looks better, but there's still the problem with parameters

Script is called: 
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=brand_cifs_poc --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd 

Result is (error out) 
ame: unrecognized option '--first=no' ame: unrecognized option '--version=1' ame: unrecognized option '--volume-op=start'

This needs to be fixed also:
2) Remove undefined parameters from script call "--first=no --version=1 --volume-op=start" or define them properly in parse_args () function

...
function parse_args () {
        ARGS=$(getopt -l $OPTSPEC  -name $PROGNAME $@)
        eval set -- "$ARGS"

        while true; do
            case $1 in
                --volname)
                    shift
                    VOL=$1
                    ;;
                --gd-workdir)
                    shift
                    GLUSTERD_WORKDIR=$1
                    ;;
                ####<ADD CODE HERE>###
                *)
                    shift
                    break
                    ;;
            esac
            shift
        done
}
...

or call script with:

/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=brand_cifs_poc  --gd-workdir=/var/lib/glusterd

Comment 27 Cal Calhoun 2017-10-18 13:16:24 UTC
@Michael: Moved to core unless someone can suggest a better component.

Comment 33 Cal Calhoun 2018-01-18 15:55:28 UTC
@sankarshan, I'm not sure.  The customer had added this reference to the case so I wanted to make you aware of it.

Comment 35 Cal Calhoun 2018-01-25 15:00:00 UTC
@Anoop,

  I'll convey this to the customer and have them open a new case for the shellcheck warnings.

-Cal

Comment 36 Vivek Das 2018-05-08 15:04:20 UTC
Executed S30samba-start.sh script and I am not seeing the issue any more.
Moving this bug to verified.

Version
-------
samba-4.7.5-103.el7rhgs.x86_64
glusterfs-server-3.12.2-8.el7rhgs.x86_64

Comment 37 errata-xmlrpc 2018-09-04 06:32:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607