+++ This bug was initially created as a clone of Bug #1288857 +++ Description of problem: fouh->len is accessed after 'node' is freed. Also rv is int where as fouh->len is uint32 comparison needs to be changed to ssize_t variables. Asan report: ==10762== ERROR: AddressSanitizer: heap-use-after-free on address 0x602c00048700 at pc 0x7f667e468a00 bp 0x7f6675c42e20 sp 0x7f6675c42e10 READ of size 4 at 0x602c00048700 thread T9 #0 0x7f667e4689ff in notify_kernel_loop /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875 #1 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) #2 0x3cf4207ee4 in start_thread (/lib64/libpthread.so.0+0x3cf4207ee4) #3 0x3cf3ef4d1c in __clone (/lib64/libc.so.6+0x3cf3ef4d1c) 0x602c00048700 is located 64 bytes inside of 376-byte region [0x602c000486c0,0x602c00048838) freed by thread T9 here: #0 0x7f66860e00f9 (/lib64/libasan.so.0+0x160f9) #1 0x7f6685d5e6a4 in __gf_free /home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:336 #2 0x7f667e4689c4 in notify_kernel_loop /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3873 #3 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) previously allocated by thread T7 here: #0 0x7f66860e0315 (/lib64/libasan.so.0+0x16315) #1 0x7f6685d5d3be in __gf_calloc /home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:117 #2 0x7f667e4308b7 in fuse_invalidate_inode /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:295 #3 0x7f667e42f61c in fuse_invalidate /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:55 #4 0x7f6685d22071 in inode_invalidate /home/pk1/workspace/gerrit-repo/libglusterfs/src/inode.c:1158 #5 0x7f66790789ed in mdc_inode_iatt_set_validate /home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:427 #6 0x7f667907e5da in mdc_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:1040 #7 0x7f6685e3b57c in default_ftruncate_cbk /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333 #8 0x7f6685e3b57c in default_ftruncate_cbk /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333 #9 0x7f66796d52c6 in ioc_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/io-cache/src/io-cache.c:1327 #10 0x7f6679b0d33c in ra_truncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/read-ahead/src/read-ahead.c:704 #11 0x7f6679d38e90 in wb_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/write-behind/src/write-behind.c:1693 #12 0x7f667a02a74e in dht_truncate_cbk /home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-inode-write.c:283 #13 0x7f667a2ee5fd in afr_ftruncate_unwind /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:646 #14 0x7f667a2e8200 in __afr_inode_write_cbk /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:171 #15 0x7f667a2ee7a0 in afr_ftruncate_wind_cbk /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:665 #16 0x7f667a610c79 in client3_3_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-rpc-fops.c:1512 #17 0x7f6685a82e45 in rpc_clnt_handle_reply /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759 #18 0x7f6685a83674 in rpc_clnt_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900 #19 0x7f6685a7a83a in rpc_transport_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541 #20 0x7f667b5cda53 in socket_event_poll_in /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231 #21 0x7f667b5ce720 in socket_event_handler /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344 #22 0x7f6685ddaf49 in event_dispatch_epoll_handler /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571 #23 0x7f6685ddb823 in event_dispatch_epoll_worker /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674 #24 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) Thread T9 created by T8 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685d18bf9 in gf_thread_create /home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468 #2 0x7f667e4691ee in fuse_init /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3946 #3 0x7f667e46fc64 in fuse_thread_proc /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:4935 #4 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) Thread T8 created by T5 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685d18bf9 in gf_thread_create /home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468 #2 0x7f667e471205 in notify /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:5170 #3 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #4 0x7f6685e58f97 in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2879 #5 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #6 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #7 0x7f6678e5e4bb in notify /home/pk1/workspace/gerrit-repo/xlators/debug/io-stats/src/io-stats.c:3838 #8 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #9 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #10 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #11 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #12 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #13 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #14 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #15 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #16 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #17 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #18 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #19 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #20 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #21 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #22 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #23 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #24 0x7f667a024ddc in dht_notify /home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-common.c:7888 #25 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #26 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #27 0x7f667a38f3ff in afr_notify /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-common.c:4021 #28 0x7f667a3968be in notify /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr.c:34 #29 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #30 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #31 0x7f667a5dc91a in client_notify_dispatch /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:83 #32 0x7f667a5dc761 in client_notify_dispatch_uniq /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:61 #33 0x7f667a64f7d2 in client_notify_parents_child_up /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:133 #34 0x7f667a65551a in client_post_handshake /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1053 #35 0x7f667a65637b in client_setvolume_cbk /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1210 #36 0x7f6685a82e45 in rpc_clnt_handle_reply /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759 #37 0x7f6685a83674 in rpc_clnt_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900 #38 0x7f6685a7a83a in rpc_transport_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541 #39 0x7f667b5cda53 in socket_event_poll_in /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231 #40 0x7f667b5ce720 in socket_event_handler /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344 #41 0x7f6685ddaf49 in event_dispatch_epoll_handler /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571 #42 0x7f6685ddb823 in event_dispatch_epoll_worker /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674 #43 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) Thread T5 created by T0 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685ddba89 in event_dispatch_epoll /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:726 #2 0x7f6685d5b92f in event_dispatch /home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:124 #3 0x40eeb6 in main /home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2345 #4 0x3cf3e21d64 in __libc_start_main (/lib64/libc.so.6+0x3cf3e21d64) Thread T7 created by T5 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685ddbfac in event_reconfigure_threads_epoll /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:834 #2 0x7f6685d5ba8b in event_reconfigure_threads /home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:140 #3 0x7f667a5f5f6c in client_check_event_threads /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2332 #4 0x7f667a5f69ec in init /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2448 #5 0x7f6685cf665d in __xlator_init /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:399 #6 0x7f6685cf68b7 in xlator_init /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:424 #7 0x7f6685d83a14 in glusterfs_graph_init /home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:320 #8 0x7f6685d84dec in glusterfs_graph_activate /home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:667 #9 0x40e4f4 in glusterfs_process_volfp /home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2186 #10 0x417168 in mgmt_getspec_cbk /home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd-mgmt.c:1640 #11 0x7f6685a82e45 in rpc_clnt_handle_reply /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759 #12 0x7f6685a83674 in rpc_clnt_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900 #13 0x7f6685a7a83a in rpc_transport_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541 #14 0x7f667b5cda53 in socket_event_poll_in /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231 #15 0x7f667b5ce720 in socket_event_handler /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344 #16 0x7f6685ddaf49 in event_dispatch_epoll_handler /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571 #17 0x7f6685ddb823 in event_dispatch_epoll_worker /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674 #18 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) SUMMARY: AddressSanitizer: heap-use-after-free /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875 notify_kernel_loop Shadow bytes around the buggy address: 0x0c0600001090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa 0x0c06000010a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd =>0x0c06000010e0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c0600001100: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa 0x0c0600001110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c0600001120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c0600001130: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap righ redzone: fb Freed Heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 ASan internal: fe ==10762== ABORTING fsync: Software caused connection abort Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. run iozone -a on a mount with address sanitizer enabled build and it crashes. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Vijay Bellur on 2015-12-06 12:21:03 EST --- REVIEW: http://review.gluster.org/12886 (mount/fuse: Fix use-after-free crash) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
Pranith, Could you please provide steps to re-create the problem and tests to validate the fix
hi Rajesh Reddy, I don't think it is easy to recreate this bug without some kind of memory sanitizer loaded. If you run 'iozone -a' and nothing bad happens on the fuse mount (Any volume type is fine, I took plain replicate), you can close the bug. Pranith
Tested with glusterfs-server-3.7.5-11, on FUSE mount ran iozone -a and iozone test completed without any error so marking this bug as verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html