+++ This bug was initially created as a clone of Bug #1417915 +++

+++ This bug was initially created as a clone of Bug #1417913 +++

+++ This bug was initially created as a clone of Bug #1416684 +++

Description of problem:
Since 3.9.0 command
/usr/sbin/glusterfs --debug --volfile-server=SERVER --volfile-id=VOLID /mount/path
hangs on 32 bit system

Additional info:

It hangs on INCREMENT_ATOMIC from this commit:
commit 3cc7f6588c281846f8c590553da03dd16f150e8a
Author: Poornima G <pgurusid>
Date:   Wed Aug 17 12:55:37 2016 +0530

    md-cache: Add cache hit and miss counters
    
    These counters can be accessed either by .meta interface
    or statedump.
    
    From meta: cat on the private file in md-cache directory.
    Eg: cat /mnt/glusterfs/0/.meta/graphs/active/patchy-md-cache/private
    [performance/md-cache.patchy-md-cache]


         if (xdata) {
                 ret = mdc_inode_xatt_get (this, loc->inode, &xattr_rsp);
-                if (ret != 0)
+                if (ret != 0) {
+                        INCREMENT_ATOMIC (conf->mdc_counter.lock,
+                                          conf->mdc_counter.xattr_miss);
                         goto uncached;
+                }
 

comment out "!defined(__i386__)" in INCREMENT_ATOMIC definition:

+#if (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 1)) && !defined(__i386__)
+# define INCREMENT_ATOMIC(lk, op) __sync_add_and_fetch(&op, 1)

fix the problem.

I see two problem here:
1. incorrect workaround for INCREMENT_ATOMIC
2. incorrect detection of RHEL 5 32bit

Backtrace:
#0  0xb7e55362 in pthread_spin_lock () from /lib/libpthread.so.0
(gdb) bt
#0  0xb7e55362 in pthread_spin_lock () from /lib/libpthread.so.0
#1  0xb3541ffd in mdc_lookup (frame=0xb6df2300, this=0xb37165b8, loc=0xb03fd220, xdata=0xb66417b4) at md-cache.c:1085
#2  0xb350e3e5 in io_stats_lookup (frame=0xb6df2218, this=0xb3717680, loc=0xb03fd220, xdata=0xb66417b4) at io-stats.c:2617
#3  0xb7f3c225 in default_lookup (frame=0xb6df2218, this=0xb3702f60, loc=0xb03fd220, xdata=0xb66417b4) at defaults.c:2572
#4  0xb34fb63a in meta_lookup (frame=0xb6df2218, this=0xb3702f60, loc=0xb03fd220, xdata=0xb66417b4) at meta.c:44
#5  0xb63282d1 in fuse_first_lookup (this=0x8089400) at fuse-bridge.c:4262

--- Additional comment from Vitaly Lipatov on 2017-01-26 10:43:20 CET ---

LOCK hangs due missed LOCK_INIT == missed pthread_spin_init == use gf_lock_t lock uninitialized with zero;

--- Additional comment from Vitaly Lipatov on 2017-01-26 11:04:25 CET ---

fix:

--- a/xlators/performance/md-cache/src/md-cache.c
+++ b/xlators/performance/md-cache/src/md-cache.c
@@ -2905,6 +2905,7 @@ init (xlator_t *this)
         GF_OPTION_INIT("cache-invalidation", conf->mdc_invalidation, bool, out);
 
         LOCK_INIT (&conf->lock);
+        LOCK_INIT (&conf->mdc_counter.lock);
         time (&conf->last_child_down);