Description of problem:

On Linux setups where the install prefix (./configure --prefix=...) is mounted read-only, such as NixOS, one gets the following glusterfind failure:

  $ glusterfind query myvol outfile --full
  Fail to create dir /nix/store/w6204cvmxi7qkzvx11ivgq5hgrm2f2wm-glusterfs-3.10.2/var/lib/misc/glusterfsd/glusterfind/default/myvol/20170907-215100-679121-mTMVz3: [Errno 30] Read-only file system: '/nix/store/w6204cvmxi7qkzvx11ivgq5hgrm2f2wm-glusterfs-3.10.2/var/lib/misc'

This is because glusterfind tries to put variable data into $PREFIX, instead of into /var (or whatever the --localstatedir given to ./configure is), where all modifiable data should land.

In detail, it happens this way:

configure.ac has:

    GLUSTERFSD_MISCDIR="$(eval echo $prefix)/var/lib/misc/glusterfsd"

Then, tools/glusterfind/src/tool.conf.in has:

   [vars]
    session_dir=@GLUSTERD_WORKDIR@/glusterfind/
    working_dir=@GLUSTERFSD_MISCDIR@/glusterfind/
    log_dir=/var/log/glusterfs/glusterfind/

relevant here is `working_dir=@GLUSTERFSD_MISCDIR@/`. (Note `log_dir=/var/...` is probably also wrong and should use `@LOCALSTATEDIR@`, but that's not relevant for this issue).

That expands to (NixOS in my example):

    # cat /nix/store/w6204cvmxi7qkzvx11ivgq5hgrm2f2wm-glusterfs-3.10.2/libexec/glusterfs/glusterfind/tool.conf 
    [vars]
    session_dir=/var/lib/glusterd/glusterfind/
    working_dir=/nix/store/w6204cvmxi7qkzvx11ivgq5hgrm2f2wm-glusterfs-3.10.2/var/lib/misc/glusterfsd/glusterfind/
    log_dir=/var/log/glusterfs/glusterfind/

Finally, tools/glusterfind/src/main.py has:

    node_outfile = os.path.join(conf.get_opt("working_dir"),
                                args.session, args.volume,
                                tmpfilename,
                                "tmp_output_%s" % num)
    ...
    # remote file will be copied into this directory
    mkdirp(os.path.dirname(node_outfile),
           exit_on_err=True, logger=logger)

And that's where `mkdirp` is run on a directory in $PREFIX, namely fully expanded through the layer:

    mkdirp($(eval echo $prefix)/var/lib/misc/glusterfsd/glusterfind/args.session/args.volume)

I think it is wrong that this is done in $prefix; it should be done in $localstatedir, as all other components of glusterfs do it.


Version-Release number of selected component: glusterfs 3.10 and 3.12

How reproducible: Always


Steps to Reproduce:
1. Install glusterfs with ./configure --prefix=/path/to/prefix --localstatedir=/path/to/var
2. Mount /path/to/prefix readonly
3. Set up glusterfs volume "myvol"
4. Run `glusterfind query myvol outfile --full`