diff --git a/scripts/hfs_start_volume.py b/scripts/hfs_start_volume.py
index a85886c..6f28dc5 100755
--- a/scripts/hfs_start_volume.py
+++ b/scripts/hfs_start_volume.py
@@ -186,6 +186,7 @@ def create_tenant_dirs(vol_file):
                        os.mkdir(ppath)
                if not os.path.exists(path):
                        os.mkdir(path)
+                       os.chmod(path, 0777)
        opt_dir_lines.close()
 
 def start_local (vol_name):

I think we need to look into this a little further.  The server daemons run as root.  There's code there to have them call setfsuid/setfsgid, but that code's conditional on HAVE_SET_FSID which is explicitly undefined (posix.c line 64).  Therefore we should still be root when we actually make I/O calls, and with mode 0755 the OS shouldn't be rejecting them - unless it's an SELinux thing, in which case let's fix that.  The other possibility is that accesses are being rejected by the posix-acl translator, because the UID has been mapped to something that's not recognized as super-user.  This is a known issue with access-control since 3.2.4 or so, and is why http://review.gluster.com/#change,332 exists.  That patch allows some UID other than zero to be treated as super-user, which I believe would fix this problem, but nothing in the HekaFS volume/tenant management sets the necessary option.

Even if neither the SELinux nor posix-acl explanations turns out to be the case, we should be able to do better than to create world-writable directories.  Why can't we chown the directory to the root UID for that tenant instead?

Enabling HAVE_SET_FSID made no difference.

And there's nothing in /var/log/setroubleshoot logs, and the SELinux Troubleshooter does not indicate any problems from glusterfsd

Just to capture an in-person conversation between Kaleb and myself about this...

We seem to be dealing with two issues here.  One is almost a philosophical question question: should the tenant root have "tmp-like" behavior (world writable) or "root-like" behavior (only writable by the per-tenant root user)?  Giving it tmp-like behavior just means setting its mode to 01777 - sticky bit set to prevent users from removing each others' files.  Giving it root-like (or any other) behavior is more complicated, but is probably what we want in the long term.  The problem there is that we need to set things up so that posix-acl will work, so the root-directory creation would look something like this:

   tenant_root = hfs_utils.get_tenant_root(tenant)
   os.mkdir(path)
   os.chown(path,tenant_root)
   os.chmod(path,0777)

There's also another important piece, when we're modifying the volfile:

   posix_acl_xlator.options["super-uid"] = tenant_root

That way, posix_acl will do the correct permissions checks, with a result just like the root directory of a local filesystem.  If the volume were mounted on /home, for example then useradd etc. would run as tenant-root and be able to mkdir/chown per-user subdirectories on the volume root directory which it owns.

For now, tmp-like behavior is OK.  Longer term, we'll have to hack the management code to implement root-like behavior.

fixed in ssh://git.fedorahosted.org/git/CloudFS.git

commit 500f3f0a59d7f73aff9f4848f0025e77a287d314
Author: Kaleb S. KEITHLEY <kkeithle>
Date:   Thu Oct 20 10:51:32 2011 -0400

hekafs-0.7-16.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/hekafs-0.7-16.fc16

Package hekafs-0.7-16.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing hekafs-0.7-16.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14711
then log in and leave karma (feedback).

hekafs-0.7-16.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.