As a proof of concept I'm trying to write a xlator that does nothing, I call it "nop". The code for "nop.c" is simply: #include "config.h" #include "call-stub.h" struct xlator_fops fops = {}; struct xlator_cbks cbks = {}; struct xlator_dumpops dumpops = {}; struct volume_options options[] = {{.key={NULL}},}; int32_t init (xlator_t *this){return 0;} int fini (xlator_t *this){return 0;} And I compile it with: $ gcc -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -DGF_LINUX_HOST_OS -shared -nostartfiles -lglusterfs -lpthread -I${GFS} -I${GFS}/libglusterfs/src -I${GFS}/contrib/uuid nop.c -o nop.so Then, if I try a test.vol file like that: volume test-posix type storage/posix option directory /node0-data end-volume volume test-nop type features/nop subvolumes test-posix end-volume volume test-cache type performance/io-cache subvolumes test-nop end-volume and mount it with: $ glusterfs --debug -f test.vol /mount/point It seems to work fine, doing nothing. However, when used together with the stripe xlator as follows: volume test-posix0 type storage/posix option directory /node0-data end-volume volume test-posix1 type storage/posix option directory /node1-data end-volume volume test-nop0 type features/nop subvolumes test-posix0 end-volume volume test-nop1 type features/nop subvolumes test-posix1 end-volume volume test-stripe type cluster/stripe subvolumes test-nop0 test-nop1 end-volume Glusterfs hangs during the initial fuse lookups and /mount/point looks unmounted with ???? permissions etc.
To see that this is a "stripe" bug you can replace the stripe xlator for a distribute xlator and see how example mounts and works perfectly. Similarly, again with the stripe xlator, if instead of using the nop translator you use for example the rot-13 you will experience the same problem. Somehow stripe xlator does not work with the "default" functions.
Hi Jeff, is this something you want to check out? Or do you know someone that would be willing to do so?
Yes, I'll take it. This is almost certainly something to do with the fact that the default functions use STACK_WIND_TAIL, so callbacks can come from somewhere "down the stack" (e.g. protocol/client) instead of stripe's immediate child (in this case the whatever/nop). By default (unless STACK_WIND_COOKIE was used) the identity of this immediately previous translator is passed to callbacks as the "cookie" argument. A cursory examination of the stripe code shows that there is a dependency on this to issue further calls, e.g. in stripe_sh_make_entry_cbk. When STACK_WIND_TAIL was first implemented there was a similar dependency in DHT, which caused similar problems, so it seems highly likely that we're seeing a repeat of the same thing in stripe.
(In reply to Jeff Darcy from comment #3) > Yes, I'll take it. Thanks!
Looks like my theory was correct. The specific culprit, in current master, is this check in stripe_lookup_cbk (stripe.c:215). > if (FIRST_CHILD(this) == prev->this) { If "prev" doesn't match the first child, which it never will if STACK_WIND_TAIL gets involved, then we don't fill in the attr structure and return absolute garbage to FUSE. On my system this resulted in subsequent calls returning EIO instead of hanging, but close enough. Sure enough, when I changed the defaults-generation code to use STACK_WIND instead of STACK_WIND_TAIL, the problem went away. I can generate a patch pretty easily to do a smarter ancestry check in this case, but must add two caveats. First, there might be other problems of the same general nature. Second, this kind of single-volfile client and server in the same process isn't really supported (and stripe will soon be deprecated in favor of shard) so we're kind of on shaky ground here. I'm glad to fix the immediate problem, but can't promise to fix others that might be specific to this usage pattern.
REVIEW: http://review.gluster.org/13178 (stripe: fix FIRST_CHILD checks to be more general) posted (#1) for review on master by Jeff Darcy (jdarcy)
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Stripe is deprecated.