Bug 1422431
| Summary: | multiple glusterfsd process crashed making the complete subvolume unavailable | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rahul Hinduja <rhinduja> | |
| Component: | marker | Assignee: | Poornima G <pgurusid> | |
| Status: | CLOSED ERRATA | QA Contact: | Rahul Hinduja <rhinduja> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.2 | CC: | amukherj, asrivast, pgurusid, rcyriac, rhs-bugs, skoduri, storage-qa-internal | |
| Target Milestone: | --- | |||
| Target Release: | RHGS 3.2.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8.4-15 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1422776 (view as bug list) | Environment: | ||
| Last Closed: | 2017-03-23 06:05:09 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1351528, 1422776, 1424937 | |||
Upcall invalidations for XATTR operations were added as part of md-cache optimizations. The crash looks similar to bug1387204. Request poornima to take a look. Thanks! The simple reproducer for this issue: Create a plain distribute volume, enable cache-invalidation and marker feature on the server side: gluster vol set <VOLNAME> features.cache-invalidation on gluster vol ser <VOLNAME> indexing on gluster vol quota <VOLNAME> enable And from the fuse mount point, create a file and rename the file. After this the bricks will crash. The reason for the crash is, on recieving a rename fop, marker_rename() stores the, oldloc and newloc in its 'local' struct, once the rename is done, the xtime marker(last updated time) is set on the file, but sending a setxattr fop. When upcall receives the setxattr fop, the loc->inode is NULL and it crashes. The loc->inode can be NULL only in one valid case, i.e. in rename case where the inode of new loc will be NULL. Hence, marker should have got the inode of the new_loc and filled it before issuing a setxattr. Hence moving the component to marker. This is similar to BZ: 1387204 that is already fixed, but when quota is enabled it takes a different code path. Will send the patch in marker-quota to fix the same. downstream patch : https://code.engineering.redhat.com/gerrit/#/c/98136 Verified with build: glusterfs-geo-replication-3.8.4-15.el7rhgs.x86_64 Ran the same test suit which does create,chmod,chown,chgrp,symlink,hardlink,truncate,rename and remove in different crawl method with quota and md-cache enabled. All cases are passed and no crashes are seen: [geo_rahul@skywalker ~]$ grep -ri "quota" /home/geo_rahul/regression/3.2-Regression/3.8.4-15-RHEL7.3-rsync-fuse-with-mdcache.log 2017-02-20 23:55:08,502 INFO run Executing gluster volume quota master enable on 10.70.42.7 2017-02-20 23:55:42,794 INFO run "gluster volume quota master enable" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:42,794 INFO run "gluster volume quota master enable" on 10.70.42.7: STDOUT is volume quota : success 2017-02-20 23:55:42,795 INFO run Executing gluster volume quota master limit-usage / 100GB on 10.70.42.7 2017-02-20 23:55:47,671 INFO run "gluster volume quota master limit-usage / 100GB" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:47,671 INFO run "gluster volume quota master limit-usage / 100GB" on 10.70.42.7: STDOUT is volume quota : success 2017-02-20 23:56:20,384 INFO run Executing gluster volume quota slave enable on 10.70.43.249 2017-02-20 23:56:54,749 INFO run "gluster volume quota slave enable" on 10.70.43.249: RETCODE is 0 2017-02-20 23:56:54,749 INFO run "gluster volume quota slave enable" on 10.70.43.249: STDOUT is volume quota : success 2017-02-20 23:56:54,749 INFO run Executing gluster volume quota slave limit-usage / 100GB on 10.70.43.249 2017-02-20 23:56:59,809 INFO run "gluster volume quota slave limit-usage / 100GB" on 10.70.43.249: RETCODE is 0 2017-02-20 23:56:59,809 INFO run "gluster volume quota slave limit-usage / 100GB" on 10.70.43.249: STDOUT is volume quota : success [geo_rahul@skywalker ~]$ [geo_rahul@skywalker ~]$ [geo_rahul@skywalker ~]$ grep -ri "volume set" /home/geo_rahul/regression/3.2-Regression/3.8.4-15-RHEL7.3-rsync-fuse-with-mdcache.log 2017-02-20 23:55:47,671 INFO run Executing gluster volume set master performance.cache-invalidation on on 10.70.42.7 2017-02-20 23:55:48,710 INFO run "gluster volume set master performance.cache-invalidation on" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:48,711 INFO run "gluster volume set master performance.cache-invalidation on" on 10.70.42.7: STDOUT is volume set: success 2017-02-20 23:55:48,711 INFO run Executing gluster volume set master features.cache-invalidation on on 10.70.42.7 2017-02-20 23:55:49,666 INFO run "gluster volume set master features.cache-invalidation on" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:49,666 INFO run "gluster volume set master features.cache-invalidation on" on 10.70.42.7: STDOUT is volume set: success 2017-02-20 23:55:49,667 INFO run Executing gluster volume set master performance.md-cache-timeout 600 on 10.70.42.7 2017-02-20 23:55:50,652 INFO run "gluster volume set master performance.md-cache-timeout 600" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:50,652 INFO run "gluster volume set master performance.md-cache-timeout 600" on 10.70.42.7: STDOUT is volume set: success 2017-02-20 23:55:50,653 INFO run Executing gluster volume set master performance.stat-prefetch on on 10.70.42.7 2017-02-20 23:55:51,615 INFO run "gluster volume set master performance.stat-prefetch on" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:51,615 INFO run "gluster volume set master performance.stat-prefetch on" on 10.70.42.7: STDOUT is volume set: success 2017-02-20 23:55:51,616 INFO run Executing gluster volume set master features.cache-invalidation-timeout 600 on 10.70.42.7 2017-02-20 23:55:52,584 INFO run "gluster volume set master features.cache-invalidation-timeout 600" on 10.70.42.7: RETCODE is 0 2017-02-20 23:55:52,584 INFO run "gluster volume set master features.cache-invalidation-timeout 600" on 10.70.42.7: STDOUT is volume set: success 2017-02-20 23:56:59,810 INFO run Executing gluster volume set slave performance.cache-invalidation on on 10.70.43.249 2017-02-20 23:57:01,095 INFO run "gluster volume set slave performance.cache-invalidation on" on 10.70.43.249: RETCODE is 0 2017-02-20 23:57:01,095 INFO run "gluster volume set slave performance.cache-invalidation on" on 10.70.43.249: STDOUT is volume set: success 2017-02-20 23:57:01,095 INFO run Executing gluster volume set slave features.cache-invalidation on on 10.70.43.249 2017-02-20 23:57:02,355 INFO run "gluster volume set slave features.cache-invalidation on" on 10.70.43.249: RETCODE is 0 2017-02-20 23:57:02,355 INFO run "gluster volume set slave features.cache-invalidation on" on 10.70.43.249: STDOUT is volume set: success 2017-02-20 23:57:02,355 INFO run Executing gluster volume set slave performance.md-cache-timeout 600 on 10.70.43.249 2017-02-20 23:57:03,531 INFO run "gluster volume set slave performance.md-cache-timeout 600" on 10.70.43.249: RETCODE is 0 2017-02-20 23:57:03,531 INFO run "gluster volume set slave performance.md-cache-timeout 600" on 10.70.43.249: STDOUT is volume set: success 2017-02-20 23:57:03,531 INFO run Executing gluster volume set slave performance.stat-prefetch on on 10.70.43.249 2017-02-20 23:57:04,780 INFO run "gluster volume set slave performance.stat-prefetch on" on 10.70.43.249: RETCODE is 0 2017-02-20 23:57:04,780 INFO run "gluster volume set slave performance.stat-prefetch on" on 10.70.43.249: STDOUT is volume set: success 2017-02-20 23:57:04,780 INFO run Executing gluster volume set slave features.cache-invalidation-timeout 600 on 10.70.43.249 2017-02-20 23:57:06,014 INFO run "gluster volume set slave features.cache-invalidation-timeout 600" on 10.70.43.249: RETCODE is 0 2017-02-20 23:57:06,014 INFO run "gluster volume set slave features.cache-invalidation-timeout 600" on 10.70.43.249: STDOUT is volume set: success [geo_rahul@skywalker ~]$ [root@dhcp42-7 ~]# gluster volume info master Volume Name: master Type: Distributed-Replicate Volume ID: 69c38f0f-c27b-47fe-b02a-8927cfa68eec Status: Started Snapshot Count: 0 Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: 10.70.42.7:/bricks/brick0/master_brick0 Brick2: 10.70.41.211:/bricks/brick0/master_brick1 Brick3: 10.70.43.141:/bricks/brick0/master_brick2 Brick4: 10.70.43.156:/bricks/brick0/master_brick3 Brick5: 10.70.42.7:/bricks/brick1/master_brick4 Brick6: 10.70.41.211:/bricks/brick1/master_brick5 Brick7: 10.70.43.141:/bricks/brick1/master_brick6 Brick8: 10.70.43.156:/bricks/brick1/master_brick7 Brick9: 10.70.42.7:/bricks/brick2/master_brick8 Brick10: 10.70.41.211:/bricks/brick2/master_brick9 Brick11: 10.70.43.141:/bricks/brick2/master_brick10 Brick12: 10.70.43.156:/bricks/brick2/master_brick11 Options Reconfigured: changelog.changelog: on geo-replication.ignore-pid-check: on geo-replication.indexing: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.md-cache-timeout: 600 features.cache-invalidation: on performance.cache-invalidation: on features.quota-deem-statfs: on features.inode-quota: on features.quota: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on cluster.enable-shared-storage: enable [root@dhcp42-7 ~]# [root@dhcp43-249 ~]# gluster volume info slave Volume Name: slave Type: Distributed-Replicate Volume ID: b410de0b-9c20-4eae-a5b5-e847c5a32c98 Status: Started Snapshot Count: 0 Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: 10.70.43.249:/bricks/brick0/slave_brick0 Brick2: 10.70.43.196:/bricks/brick0/slave_brick1 Brick3: 10.70.41.187:/bricks/brick0/slave_brick2 Brick4: 10.70.43.208:/bricks/brick0/slave_brick3 Brick5: 10.70.43.249:/bricks/brick1/slave_brick4 Brick6: 10.70.43.196:/bricks/brick1/slave_brick5 Brick7: 10.70.41.187:/bricks/brick1/slave_brick6 Brick8: 10.70.43.208:/bricks/brick1/slave_brick7 Brick9: 10.70.43.249:/bricks/brick2/slave_brick8 Brick10: 10.70.43.196:/bricks/brick2/slave_brick9 Brick11: 10.70.41.187:/bricks/brick2/slave_brick10 Brick12: 10.70.43.208:/bricks/brick2/slave_brick11 Options Reconfigured: features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.md-cache-timeout: 600 features.cache-invalidation: on performance.cache-invalidation: on features.quota-deem-statfs: on features.inode-quota: on features.quota: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on cluster.enable-shared-storage: enable [root@dhcp43-249 ~]# [geo_rahul@skywalker distaf]$ time python main.py -d "geo_rep" -t "$testcases" test_1_changelog-test-create (__main__.gluster_tests) ... ok test_2_changelog-test-chmod (__main__.gluster_tests) ... ok test_3_changelog-test-chown (__main__.gluster_tests) ... ok test_4_changelog-test-chgrp (__main__.gluster_tests) ... ok test_5_changelog-test-symlink (__main__.gluster_tests) ... ok test_6_changelog-test-hardlink (__main__.gluster_tests) ... ok test_7_changelog-test-truncate (__main__.gluster_tests) ... ok test_8_changelog-test-rename (__main__.gluster_tests) ... ok test_9_changelog-test-remove (__main__.gluster_tests) ... ok test_10_xsync-test-create (__main__.gluster_tests) ... ok test_11_xsync-test-chmod (__main__.gluster_tests) ... ok test_12_xsync-test-chown (__main__.gluster_tests) ... ok test_13_xsync-test-chgrp (__main__.gluster_tests) ... ok test_14_xsync-test-symlink (__main__.gluster_tests) ... ok test_15_xsync-test-hardlink (__main__.gluster_tests) ... ok test_16_xsync-test-truncate (__main__.gluster_tests) ... ok test_17_history-test-create (__main__.gluster_tests) ... ok test_18_history-test-chmod (__main__.gluster_tests) ... ok test_19_history-test-chown (__main__.gluster_tests) ... ok test_20_history-test-chgrp (__main__.gluster_tests) ... ok test_21_history-test-symlink (__main__.gluster_tests) ... ok test_22_history-test-hardlink (__main__.gluster_tests) ... ok test_23_history-test-truncate (__main__.gluster_tests) ... ok test_24_history-test-rename (__main__.gluster_tests) ... ok test_25_history-test-remove (__main__.gluster_tests) ... ok test_26_history-dynamic-create (__main__.gluster_tests) ... ok test_27_history-dynamic-chmod (__main__.gluster_tests) ... ok test_28_history-dynamic-chown (__main__.gluster_tests) ... ok test_29_history-dynamic-chgrp (__main__.gluster_tests) ... ok test_30_history-dynamic-symlink (__main__.gluster_tests) ... ok test_31_history-dynamic-hardlink (__main__.gluster_tests) ... ok test_32_history-dynamic-truncate (__main__.gluster_tests) ... ok test_33_history-dynamic-rename (__main__.gluster_tests) ... ok test_34_history-dynamic-remove (__main__.gluster_tests) ... ok ---------------------------------------------------------------------- Ran 34 tests in 41116.410s OK real 686m23.315s user 0m14.528s sys 0m4.805s [geo_rahul@skywalker distaf]$ Moving this bug to verified state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html |
Description of problem: ======================= While trying the geo-rep sanity check with md-cache options enabled. Multiple brick crashes were seen during rename which made the complete subvolume and its data unavailable. (gdb) bt #0 0x00007fc7b5510210 in pthread_spin_lock () from /lib64/libpthread.so.0 #1 0x00007fc7a2a96189 in upcall_inode_ctx_get (inode=inode@entry=0x0, this=this@entry=0x7fc7a4017970) at upcall-internal.c:231 #2 0x00007fc7a2a8b59f in upcall_local_init (frame=frame@entry=0x7fc7b419924c, this=this@entry=0x7fc7a4017970, loc=loc@entry=0x7fc7b3b15498, fd=fd@entry=0x0, inode=0x0, xattr=xattr@entry=0x7fc7b3931cec) at upcall.c:2263 #3 0x00007fc7a2a8f377 in up_setxattr (frame=0x7fc7b419924c, this=0x7fc7a4017970, loc=0x7fc7b3b15498, dict=0x7fc7b3931cec, flags=0, xdata=0x0) at upcall.c:1688 #4 0x00007fc7b673b684 in default_setxattr_resume (frame=0x7fc7b41ee490, this=0x7fc7a4018ee0, loc=0x7fc7b3b15498, dict=0x7fc7b3931cec, flags=0, xdata=0x0) at defaults.c:1646 #5 0x00007fc7b66cd64d in call_resume (stub=0x7fc7b3b15448) at call-stub.c:2508 #6 0x00007fc7a287b957 in iot_worker (data=0x7fc7a4069450) at io-threads.c:220 #7 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 (gdb) thread apply all bt Thread 50 (Thread 0x7fc790b6e700 (LWP 13042)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a8ca5555 in janitor_get_next_fd (this=0x7fc7a4006d60) at posix-helpers.c:1341 #2 posix_janitor_thread_proc (data=0x7fc7a4006d60) at posix-helpers.c:1388 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7fc73dffb700 (LWP 14749)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b833a1f0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b833a1f0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7fc73e7fc700 (LWP 14748)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8339e30) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8339e30) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7fc73d4f7700 (LWP 14970)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7fc73effd700 (LWP 14735)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8339a70) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8339a70) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7fc7525fa700 (LWP 14688)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b83383f0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b83383f0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fc752dfb700 (LWP 14687)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8338030) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8338030) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fc753dfd700 (LWP 14685)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b83378b0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b83378b0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fc7502f3700 (LWP 14717)): #0 0x00007fc7b55121bd in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc7b550dd02 in _L_lock_791 () from /lib64/libpthread.so.0 #2 0x00007fc7b550dc08 in pthread_mutex_lock () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #3 0x00007fc7aaf61be2 in socket_submit_reply (this=0x7fc7a410b8c0, reply=0x7fc7502f26a0) at socket.c:3507 #4 0x00007fc7b646be62 in rpcsvc_transport_submit (trans=trans@entry=0x7fc7a410b8c0, rpchdr=rpchdr@entry=0x7fc7502f2770, rpchdrcount=rpchdrcount@entry=1, proghdr=proghdr@entry=0x7fc7502f2820, proghdrcount=proghdrcount@entry=1, progpayload=progpayload@entry=0x0, progpayloadcount=progpayloadcount@entry=0, iobref=iobref@entry=0x7fc71021f490, priv=0x0) at rpcsvc.c:1110 #5 0x00007fc7b646d9a0 in rpcsvc_submit_generic (req=req@entry=0x7fc7a0c6b404, proghdr=proghdr@entry=0x7fc7502f2820, hdrcount=hdrcount@entry=1, payload=payload@entry=0x0, payloadcount=payloadcount@entry=0, iobref=iobref@entry=0x7fc71021f490) at rpcsvc.c:1294 #6 0x00007fc7a1995f19 in server_submit_reply (frame=frame@entry=0x7fc7b41ca6ac, req=0x7fc7a0c6b404, arg=arg@entry=0x7fc7502f28d0, payload=payload@entry=0x0, payloadcount=payloadcount@entry=0, iobref=0x7fc71021f490, iobref@entry=0x0, xdrproc=0x7fc7b625aca0 <xdr_gf_common_rsp>) at server.c:187 #7 0x00007fc7a19a5d36 in server_entrylk_cbk (frame=0x7fc7b41ca6ac, cookie=<optimized out>, this=0x7fc7a4022fa0, op_ret=0, op_errno=0, xdata=<optimized out>) at server-rpc-fops.c:358 #8 0x00007fc7a1e05502 in io_stats_entrylk_cbk (frame=0x7fc7b41d8718, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, xdata=0x0) at io-stats.c:2480 #9 0x00007fc7b6727a4d in default_entrylk_cbk (frame=0x7fc7b41c7b9c, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, xdata=0x0) at defaults.c:1157 #10 0x00007fc7a32dd3fb in pl_common_entrylk (frame=frame@entry=0x7fc7b4212ee0, this=this@entry=0x7fc7a4012400, volume=volume@entry=0x7fc7a4201580 "master-replicate-2", inode=<optimized out>, basename=basename@entry=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=cmd@entry=ENTRYLK_UNLOCK, type=type@entry=ENTRYLK_WRLCK, loc=loc@entry=0x7fc7b3aa5708, fd=fd@entry=0x0, xdata=xdata@entry=0x7fc7b3940814) at entrylk.c:696 #11 0x00007fc7a32dd7ff in pl_entrylk (frame=frame@entry=0x7fc7b4212ee0, this=this@entry=0x7fc7a4012400, volume=volume@entry=0x7fc7a4201580 "master-replicate-2", loc=loc@entry=0x7fc7b3aa5708, basename=basename@entry=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=cmd@entry=ENTRYLK_UNLOCK, type=type@entry=ENTRYLK_WRLCK, xdata=xdata@entry=0x7fc7b3940814) at entrylk.c:716 ---Type <return> to continue, or q <return> to quit--- #12 0x00007fc7a30bb3b2 in ro_entrylk (frame=frame@entry=0x7fc7b4212ee0, this=this@entry=0x7fc7a40138a0, volume=volume@entry=0x7fc7a4201580 "master-replicate-2", loc=loc@entry=0x7fc7b3aa5708, basename=basename@entry=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=cmd@entry=ENTRYLK_UNLOCK, type=type@entry=ENTRYLK_WRLCK, xdata=xdata@entry=0x7fc7b3940814) at read-only-common.c:85 #13 0x00007fc7a2eb2d02 in ro_entrylk (frame=frame@entry=0x7fc7b4212ee0, this=this@entry=0x7fc7a4014eb0, volume=volume@entry=0x7fc7a4201580 "master-replicate-2", loc=loc@entry=0x7fc7b3aa5708, basename=basename@entry=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=cmd@entry=ENTRYLK_UNLOCK, type=type@entry=ENTRYLK_WRLCK, xdata=xdata@entry=0x7fc7b3940814) at read-only-common.c:85 #14 0x00007fc7b6723832 in default_entrylk (frame=frame@entry=0x7fc7b4212ee0, this=this@entry=0x7fc7a4016400, volume=volume@entry=0x7fc7a4201580 "master-replicate-2", loc=loc@entry=0x7fc7b3aa5708, basename=basename@entry=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=cmd@entry=ENTRYLK_UNLOCK, type=type@entry=ENTRYLK_WRLCK, xdata=xdata@entry=0x7fc7b3940814) at defaults.c:2427 #15 0x00007fc7b6723832 in default_entrylk (frame=0x7fc7b4212ee0, this=<optimized out>, volume=0x7fc7a4201580 "master-replicate-2", loc=0x7fc7b3aa5708, basename=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=ENTRYLK_UNLOCK, type=ENTRYLK_WRLCK, xdata=0x7fc7b3940814) at defaults.c:2427 #16 0x00007fc7b673c93f in default_entrylk_resume (frame=0x7fc7b41c7b9c, this=0x7fc7a4018ee0, volume=0x7fc7a4201580 "master-replicate-2", loc=0x7fc7b3aa5708, basename=0x7fc7a416f1f0 "58a29fee%%819MSNM71M", cmd=ENTRYLK_UNLOCK, type=ENTRYLK_WRLCK, xdata=0x7fc7b3940814) at defaults.c:1754 #17 0x00007fc7b66cd43c in call_resume_wind (stub=0x7fc7b3aa56b8) at call-stub.c:2143 #18 0x00007fc7b66cd64d in call_resume (stub=0x7fc7b3aa56b8) at call-stub.c:2508 #19 0x00007fc7a287b957 in iot_worker (data=0x7fc7a4069450) at io-threads.c:220 #20 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fc791bf7700 (LWP 13040)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007fc7b4e47ba3 in select () from /lib64/libc.so.6 #1 0x00007fc7a3b4214a in changelog_ev_dispatch (data=0x7fc7a40aa998) at changelog-ev-handle.c:349 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fc7933fa700 (LWP 13037)): #0 0x00007fc7b550f6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a3b41f03 in changelog_ev_connector (data=0x7fc7a40aa998) at changelog-ev-handle.c:202 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fc792bf9700 (LWP 13038)): #0 0x00007fc7b4e47ba3 in select () from /lib64/libc.so.6 #1 0x00007fc7a3b4214a in changelog_ev_dispatch (data=0x7fc7a40aa998) at changelog-ev-handle.c:349 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fc73d5f8700 (LWP 14969)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fc7501f2700 (LWP 14750)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fc73d6f9700 (LWP 14752)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fc7515f8700 (LWP 14708)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fc7819d5700 (LWP 13477)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fc7b6b81780 (LWP 13026)): #0 0x00007fc7b550cef7 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fc7b6704a38 in event_dispatch_epoll (event_pool=0x7fc7b8326e10) at event-epoll.c:758 #2 0x00007fc7b6b9cae2 in main (argc=19, argv=<optimized out>) at glusterfsd.c:2452 Thread 32 (Thread 0x7fc7535fc700 (LWP 14686)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8337c70) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8337c70) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fc7810d3700 (LWP 14675)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8337130) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8337130) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fc751df9700 (LWP 14689)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b83387b0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b83387b0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fc7514f7700 (LWP 14709)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fc7503f4700 (LWP 14716)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fc79136f700 (LWP 13827)): #0 0x00007fc7b4e1766d in nanosleep () from /lib64/libc.so.6 #1 0x00007fc7b4e17504 in sleep () from /lib64/libc.so.6 #2 0x00007fc7a8ca87fc in posix_health_check_thread_proc (data=0x7fc7a4006d60) at posix-helpers.c:1809 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fc7923f8700 (LWP 13039)): #0 0x00007fc7b4e47ba3 in select () from /lib64/libc.so.6 #1 0x00007fc7a3b4214a in changelog_ev_dispatch (data=0x7fc7a40aa998) at changelog-ev-handle.c:349 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fc7818d4700 (LWP 13837)): #0 0x00007fc7b4e47ba3 in select () from /lib64/libc.so.6 #1 0x00007fc7a3b3e372 in changelog_fsync_thread (data=0x7fc7a40aa610) at changelog-helpers.c:1427 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fc73d3f6700 (LWP 14971)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fc7822d7700 (LWP 13836)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a3b3df5e in changelog_rollover (data=0x7fc7a40aa610) at changelog-helpers.c:1317 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fc753efe700 (LWP 14684)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fc7513f6700 (LWP 14710)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8338b70) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8338b70) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fc7ab974700 (LWP 13030)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8336d70) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8336d70) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fc7823d8700 (LWP 13175)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fc73f7fe700 (LWP 14734)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b83396b0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b83396b0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fc7a0223700 (LWP 13034)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fc793fff700 (LWP 13035)): #0 0x00007fc7b550f6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a37011b3 in br_stub_signth (arg=<optimized out>) at bit-rot-stub.c:774 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fc750bf5700 (LWP 14711)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b8338f30) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b8338f30) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fc7ac976700 (LWP 13028)): #0 0x00007fc7b5513101 in sigwait () from /lib64/libpthread.so.0 #1 0x00007fc7b6b9fbfb in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2055 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc782bd9700 (LWP 13255)): #0 0x00007fc7b4e1766d in nanosleep () from /lib64/libc.so.6 #1 0x00007fc7b4e17504 in sleep () from /lib64/libc.so.6 #2 0x00007fc7a2a964bc in upcall_reaper_thread (data=0x7fc7a4017970) at upcall-internal.c:414 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 12 (Thread 0x7fc7a0122700 (LWP 13036)): #0 0x00007fc7b550f6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a36ffc4b in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc753fff700 (LWP 14683)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc73ffff700 (LWP 14718)): #0 0x00007fc7b4e41c1d in read () from /lib64/libc.so.6 #1 0x00007fc7b4dd05a0 in __GI__IO_file_underflow () from /lib64/libc.so.6 #2 0x00007fc7b4dd152e in __GI__IO_default_uflow () from /lib64/libc.so.6 #3 0x00007fc7b4db42da in __GI__IO_vfscanf () from /lib64/libc.so.6 #4 0x00007fc7b4dc0d97 in fscanf () from /lib64/libc.so.6 #5 0x00007fc7b66af4ed in gf_backtrace_fillframes (buf=buf@entry=0x7fc74812f220 "") at common-utils.c:3883 #6 0x00007fc7b66b69b5 in gf_backtrace_save (buf=buf@entry=0x7fc74812f220 "") at common-utils.c:3924 #7 0x00007fc7b66e0a9c in synctask_yield (task=0x7fc74812ed90) at syncop.c:336 #8 0x00007fc7b66f3d57 in syncop_inodelk (subvol=0x7fc7a4018ee0, volume=0x7fc7a4018570 "master-marker", loc=loc@entry=0x7fc7481582f0, cmd=cmd@entry=7, lock=lock@entry=0x7fc748157e20, xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:2992 #9 0x00007fc7a2668bcb in mq_lock (this=this@entry=0x7fc7a401a3b0, loc=loc@entry=0x7fc7481582f0, l_type=l_type@entry=1) at marker-quota.c:495 #10 0x00007fc7a266bcf5 in mq_reduce_parent_size_task (opaque=0x7fc7481f4d50) at marker-quota.c:1306 ---Type <return> to continue, or q <return> to quit--- #11 0x00007fc7b66e0b32 in synctask_wrap (old_task=<optimized out>) at syncop.c:375 #12 0x00007fc7b4d9fcf0 in ?? () from /lib64/libc.so.6 #13 0x0000000000000000 in ?? () Thread 9 (Thread 0x7fc7a96b9700 (LWP 13031)): #0 0x00007fc7b4e47590 in readv () from /lib64/libc.so.6 #1 0x00007fc7b66d1f75 in sys_readv (fd=<optimized out>, iov=<optimized out>, iovcnt=<optimized out>) at syscall.c:249 #2 0x00007fc7aaf60e1c in __socket_ssl_readv (this=this@entry=0x7fc7a410b8c0, opvector=opvector@entry=0x7fc7a96b8bb0, opcount=opcount@entry=1) at socket.c:389 #3 0x00007fc7aaf6135c in __socket_ssl_read (count=<optimized out>, buf=<optimized out>, this=0x7fc7a410b8c0) at socket.c:405 #4 __socket_cached_read (opcount=1, opvector=0x7fc7a410c438, this=0x7fc7a410b8c0) at socket.c:446 #5 __socket_rwv (this=this@entry=0x7fc7a410b8c0, vector=<optimized out>, count=count@entry=1, pending_vector=pending_vector@entry=0x7fc7a410c4c8, pending_count=pending_count@entry=0x7fc7a410c4d0, bytes=bytes@entry=0x7fc7a96b8cb0, write=write@entry=0) at socket.c:556 #6 0x00007fc7aaf62648 in __socket_readv (bytes=0x7fc7a96b8cb0, pending_count=0x7fc7a410c4d0, pending_vector=0x7fc7a410c4c8, count=1, vector=<optimized out>, this=0x7fc7a410b8c0) at socket.c:650 #7 __socket_read_frag (this=0x7fc7a410b8c0) at socket.c:1967 #8 __socket_proto_state_machine (pollin=<synthetic pointer>, this=0x7fc7a410b8c0) at socket.c:2140 #9 socket_proto_state_machine (pollin=<synthetic pointer>, this=0x7fc7a410b8c0) at socket.c:2247 #10 socket_event_poll_in (this=this@entry=0x7fc7a410b8c0) at socket.c:2263 #11 0x00007fc7aaf64785 in socket_event_handler (fd=<optimized out>, idx=29, data=0x7fc7a410b8c0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2397 #12 0x00007fc7b67045b0 in event_dispatch_epoll_handler (event=0x7fc7a96b8e80, event_pool=0x7fc7b8326e10) at event-epoll.c:571 #13 event_dispatch_epoll_worker (data=0x7fc7b837a170) at event-epoll.c:674 #14 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #15 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc783fff700 (LWP 13043)): #0 0x00007fc7b550f6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a8ca8b6b in posix_fsyncer_pick (this=this@entry=0x7fc7a4006d60, head=head@entry=0x7fc783ffee80) at posix-helpers.c:1908 #2 0x00007fc7a8ca8df5 in posix_fsyncer (d=0x7fc7a4006d60) at posix-helpers.c:2007 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc7ac175700 (LWP 13029)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b83369b0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b83369b0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc7ad177700 (LWP 13027)): #0 0x00007fc7b5512bdd in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fc7b66b7c66 in gf_timer_proc (data=0x7fc7b83363b0) at timer.c:176 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc7808d2700 (LWP 14680)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7b66e3068 in syncenv_task (proc=proc@entry=0x7fc7b83374f0) at syncop.c:603 #2 0x00007fc7b66e3eb0 in syncenv_processor (thdata=0x7fc7b83374f0) at syncop.c:695 #3 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 4 (Thread 0x7fc7a0324700 (LWP 13033)): #0 0x00007fc7b550f6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a223deab in index_worker (data=<optimized out>) at index.c:211 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc7a198c700 (LWP 13032)): #0 0x00007fc7b4e50d13 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc7b6704490 in event_dispatch_epoll_worker (data=0x7fc7a4024520) at event-epoll.c:664 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc781ad6700 (LWP 13463)): #0 0x00007fc7b550fa82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc7a287b90c in iot_worker (data=0x7fc7a4069450) at io-threads.c:180 #2 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc73d7fa700 (LWP 14751)): #0 0x00007fc7b5510210 in pthread_spin_lock () from /lib64/libpthread.so.0 #1 0x00007fc7a2a96189 in upcall_inode_ctx_get (inode=inode@entry=0x0, this=this@entry=0x7fc7a4017970) at upcall-internal.c:231 #2 0x00007fc7a2a8b59f in upcall_local_init (frame=frame@entry=0x7fc7b419924c, this=this@entry=0x7fc7a4017970, loc=loc@entry=0x7fc7b3b15498, fd=fd@entry=0x0, inode=0x0, xattr=xattr@entry=0x7fc7b3931cec) at upcall.c:2263 #3 0x00007fc7a2a8f377 in up_setxattr (frame=0x7fc7b419924c, this=0x7fc7a4017970, loc=0x7fc7b3b15498, dict=0x7fc7b3931cec, flags=0, xdata=0x0) at upcall.c:1688 #4 0x00007fc7b673b684 in default_setxattr_resume (frame=0x7fc7b41ee490, this=0x7fc7a4018ee0, loc=0x7fc7b3b15498, dict=0x7fc7b3931cec, flags=0, xdata=0x0) at defaults.c:1646 #5 0x00007fc7b66cd64d in call_resume (stub=0x7fc7b3b15448) at call-stub.c:2508 ---Type <return> to continue, or q <return> to quit--- #6 0x00007fc7a287b957 in iot_worker (data=0x7fc7a4069450) at io-threads.c:220 #7 0x00007fc7b550bdc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc7b4e5073d in clone () from /lib64/libc.so.6 (gdb) brick log suggests: ng down connection dj.lab.eng.blr.redhat.com-10304-2017/02/14-07:19:45:645412-master-client-0-0-0 pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-02-14 20:10:22 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.4 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f890ee04b92] /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f890ee0e624] /lib64/libc.so.6(+0x35250)[0x7f890d4e8250] /lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x7f890dc6a210] --------- (END) Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.8.4-13.el7rhgs.x86_64 How reproducible: ================= 2/2 during sanity check Steps carried: ============== 1. Create master and slave volume with md-cache options being enabled 2. Create geo-rep session and start it 3. Run the automated sanity test to do create,chmod,chown,chgroup,hardlink,symlink,truncate,rename Upon rename it crashes.