Description of problem: [root@localhost tcmu-runner]# ./tcmu-runner ================================================================= ==1257==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400000b6d0 at pc 0x7f71ec9f7025 bp 0x7ffd11b4e250 sp 0x7ffd11b4e240 WRITE of size 48 at 0x60400000b6d0 thread T0 #0 0x7f71ec9f7024 in gluster_cache_add /root/tcmu-runner/glfs.c:187 #1 0x7f71ec9f8686 in tcmu_create_glfs_object /root/tcmu-runner/glfs.c:376 #2 0x7f71ec9f8b72 in glfs_check_config /root/tcmu-runner/glfs.c:451 #3 0x411aa9 in on_check_config /root/tcmu-runner/main.c:198 #4 0x7f71f1667c57 in ffi_call_unix64 (/lib64/libffi.so.6+0x5c57) #5 0x7f71f16676b9 in ffi_call (/lib64/libffi.so.6+0x56b9) #6 0x7f71f3461c1d in g_cclosure_marshal_generic (/lib64/libgobject-2.0.so.0+0xfc1d) #7 0x7f71f34613e4 in g_closure_invoke (/lib64/libgobject-2.0.so.0+0xf3e4) #8 0x7f71f3473431 (/lib64/libgobject-2.0.so.0+0x21431) #9 0x7f71f347b240 in g_signal_emitv (/lib64/libgobject-2.0.so.0+0x29240) #10 0x416844 in _tcmuservice1_skeleton_handle_method_call /root/tcmu-runner/tcmuhandler-generated.c:1029 #11 0x7f71f3775e86 (/lib64/libgio-2.0.so.0+0xd1e86) #12 0x7f71f375e28b (/lib64/libgio-2.0.so.0+0xba28b) #13 0x7f71f31858e6 (/lib64/libglib-2.0.so.0+0x468e6) #14 0x7f71f3188e51 in g_main_context_dispatch (/lib64/libglib-2.0.so.0+0x49e51) #15 0x7f71f31891cf (/lib64/libglib-2.0.so.0+0x4a1cf) #16 0x7f71f31894f1 in g_main_loop_run (/lib64/libglib-2.0.so.0+0x4a4f1) #17 0x414019 in main /root/tcmu-runner/main.c:848 #18 0x7f71f233a400 in __libc_start_main (/lib64/libc.so.6+0x20400) #19 0x4079f9 in _start (/root/tcmu-runner/tcmu-runner+0x4079f9) 0x60400000b6d0 is located 0 bytes inside of 48-byte region [0x60400000b6d0,0x60400000b700) freed by thread T21 here: #0 0x7f71f3d34b00 in free (/lib64/libasan.so.3+0xc6b00) #1 0x7f71ec9f78dc in gluster_cache_refresh /root/tcmu-runner/glfs.c:256 #2 0x7f71ec9f93e9 in tcmu_glfs_close /root/tcmu-runner/glfs.c:564 #3 0x412f27 in cmdproc_thread_cleanup /root/tcmu-runner/main.c:531 #4 0x41305e in tcmur_cmdproc_thread /root/tcmu-runner/main.c:541 #5 0x7f71f2f286c9 in start_thread (/lib64/libpthread.so.0+0x76c9) previously allocated by thread T0 here: #0 0x7f71f3d35210 in realloc (/lib64/libasan.so.3+0xc7210) #1 0x7f71ec9f6f94 in gluster_cache_add /root/tcmu-runner/glfs.c:187 #2 0x7f71ec9f8686 in tcmu_create_glfs_object /root/tcmu-runner/glfs.c:376 #3 0x7f71ec9f8b72 in glfs_check_config /root/tcmu-runner/glfs.c:451 #4 0x411aa9 in on_check_config /root/tcmu-runner/main.c:198 #5 0x7f71f1667c57 in ffi_call_unix64 (/lib64/libffi.so.6+0x5c57) #6 0x7ffd11b4f66f (<unknown module>) Thread T21 created by T0 here: #0 0x7f71f3c9f488 in __interceptor_pthread_create (/lib64/libasan.so.3+0x31488) #1 0x7f71f3a4f86a in tcmulib_start_cmdproc_thread /root/tcmu-runner/libtcmu.c:732 #2 0x4134dd in dev_added /root/tcmu-runner/main.c:631 #3 0x7f71f3a4c628 in add_device /root/tcmu-runner/libtcmu.c:298 #4 0x7f71f3a4b545 in handle_netlink /root/tcmu-runner/libtcmu.c:69 #5 0x7f71f26e3634 (/lib64/libnl-genl-3.so.200+0x3634) #6 0x7f71f28f7a7b in nl_recvmsgs_report (/lib64/libnl-3.so.200+0x11a7b) SUMMARY: AddressSanitizer: heap-use-after-free /root/tcmu-runner/glfs.c:187 in gluster_cache_add Shadow bytes around the buggy address: 0x0c087fff9680: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 06 0x0c087fff9690: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd 0x0c087fff96a0: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 07 0x0c087fff96b0: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 01 0x0c087fff96c0: fa fa 00 00 00 00 00 04 fa fa 00 00 00 00 00 00 =>0x0c087fff96d0: fa fa 00 00 00 00 00 00 fa fa[fd]fd fd fd fd fd 0x0c087fff96e0: fa fa 00 00 00 00 00 00 fa fa fd fd fd fd fd fa 0x0c087fff96f0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa 0x0c087fff9700: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa 0x0c087fff9710: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa 0x0c087fff9720: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Created a replica 3 volume: Create 10 blocks for i in {1..10}; do gluster-block create r3/$i ha 3 192.168.122.61,192.168.122.123,192.168.122.113 1GiB ; done Delete them: for i in {1..10}; do gluster-block delete r3/$i ; done Create them again: It crashed with above trace. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Related patch: https://github.com/open-iscsi/tcmu-runner/pull/158
Tested and verified this on the build glusterfs-3.8.4-33 and gluster-block-0.2.1-6. Created multiple blocks in loops (>30) and deleted them, one by one as well as simultaneously when another create was taking place. The blocks in question were created and deleted successfully, with no errors/crashes seen in logs. Moving this bug to verified in 3.3.
[root@dhcp47-116 ~]# for i in {14..50}; do gluster-block create nash/nb$i ha 3 auth enable 10.70.47.115,10.70.47.116,10.70.47.117 1M; done IQN: iqn.2016-12.org.gluster-block:97bc7144-8060-4b75-8a18-5f10161d2036 USERNAME: 97bc7144-8060-4b75-8a18-5f10161d2036 PASSWORD: 052a6474-7d39-4034-b817-bc1178bf356d PORTAL(S): 10.70.47.115:3260 10.70.47.116:3260 10.70.47.117:3260 RESULT: SUCCESS IQN: iqn.2016-12.org.gluster-block:114cedc6-bdae-4974-b6a3-5d1a501be321 USERNAME: 114cedc6-bdae-4974-b6a3-5d1a501be321 PASSWORD: 99ef60e1-d6c9-4880-87c5-7bb5c96f9a15 PORTAL(S): 10.70.47.115:3260 10.70.47.116:3260 10.70.47.117:3260 RESULT: SUCCESS ... ... ... IQN: iqn.2016-12.org.gluster-block:e9b59264-9129-4ee6-a4cb-eb845cdd0231 USERNAME: e9b59264-9129-4ee6-a4cb-eb845cdd0231 PASSWORD: b2f30d1e-78de-4a28-9e44-5e224ef8d51b PORTAL(S): 10.70.47.115:3260 10.70.47.116:3260 10.70.47.117:3260 RESULT: SUCCESS IQN: iqn.2016-12.org.gluster-block:72dd48a2-98d9-47f5-b822-265fb2c05100 USERNAME: 72dd48a2-98d9-47f5-b822-265fb2c05100 PASSWORD: 877000bd-5162-4eae-9881-223e0806e2fa PORTAL(S): 10.70.47.115:3260 10.70.47.116:3260 10.70.47.117:3260 RESULT: SUCCESS [root@dhcp47-116 ~]# [root@dhcp47-115 ~]# for i in {1..20}; do gluster-block delete nash/nb$i; done SUCCESSFUL ON: 10.70.47.115 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.117 10.70.47.115 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.117 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.117 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.117 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.117 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.117 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 10.70.47.117 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.117 10.70.47.115 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.117 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.117 10.70.47.115 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.117 10.70.47.115 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 10.70.47.116 10.70.47.117 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.117 10.70.47.115 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 10.70.47.117 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.115 10.70.47.117 10.70.47.116 RESULT: SUCCESS SUCCESSFUL ON: 10.70.47.116 10.70.47.115 10.70.47.117 RESULT: SUCCESS [root@dhcp47-115 ~]# ENVIRONMENT =========== [root@dhcp47-116 ~]# [root@dhcp47-116 ~]# gluster pool list UUID Hostname State 49610061-1788-4cbc-9205-0e59fe91d842 dhcp47-121.lab.eng.blr.redhat.com Connected a0557927-4e5e-4ff7-8dce-94873f867707 dhcp47-113.lab.eng.blr.redhat.com Connected c0dac197-5a4d-4db7-b709-dbf8b8eb0896 dhcp47-114.lab.eng.blr.redhat.com Connected f828fdfa-e08f-4d12-85d8-2121cafcf9d0 dhcp47-115.lab.eng.blr.redhat.com Connected 17eb3cef-17e7-4249-954b-fc19ec608304 dhcp47-117.lab.eng.blr.redhat.com Connected a96e0244-b5ce-4518-895c-8eb453c71ded localhost Connected [root@dhcp47-116 ~]# [root@dhcp47-116 ~]# rpm -qa | grep gluster glusterfs-cli-3.8.4-33.el7rhgs.x86_64 glusterfs-rdma-3.8.4-33.el7rhgs.x86_64 python-gluster-3.8.4-33.el7rhgs.noarch vdsm-gluster-4.17.33-1.1.el7rhgs.noarch glusterfs-client-xlators-3.8.4-33.el7rhgs.x86_64 glusterfs-fuse-3.8.4-33.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-events-3.8.4-33.el7rhgs.x86_64 gluster-block-0.2.1-6.el7rhgs.x86_64 libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64 gluster-nagios-addons-0.2.9-1.el7rhgs.x86_64 samba-vfs-glusterfs-4.6.3-3.el7rhgs.x86_64 glusterfs-3.8.4-33.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-26.el7rhgs.x86_64 glusterfs-api-3.8.4-33.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-33.el7rhgs.x86_64 glusterfs-libs-3.8.4-33.el7rhgs.x86_64 glusterfs-server-3.8.4-33.el7rhgs.x86_64 [root@dhcp47-116 ~]# [root@dhcp47-116 ~]# gluster v info nash Volume Name: nash Type: Replicate Volume ID: f1ea3d3e-c536-4f36-b61f-cb9761b8a0a6 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.47.115:/bricks/brick4/nash0 Brick2: 10.70.47.116:/bricks/brick4/nash1 Brick3: 10.70.47.117:/bricks/brick4/nash2 Options Reconfigured: transport.address-family: inet nfs.disable: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off performance.open-behind: off performance.readdir-ahead: off network.remote-dio: enable cluster.eager-lock: enable cluster.quorum-type: auto cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off server.allow-insecure: on cluster.brick-multiplex: disable cluster.enable-shared-storage: enable [root@dhcp47-116 ~]# [root@dhcp47-116 ~]# gluster-block list nash nb21 nb22 nb23 nb24 nb25 nb26 nb27 nb28 nb29 nb30 nb31 nb32 nb33 nb34 nb35 nb36 nb37 nb38 nb39 nb40 nb41 nb42 nb43 nb44 nb45 nb46 nb47 nb48 nb49 nb50 [root@dhcp47-116 ~]# Eniv
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:2773