+++ This bug was initially created as a clone of Bug #1288857 +++ Description of problem: fouh->len is accessed after 'node' is freed. Also rv is int where as fouh->len is uint32 comparison needs to be changed to ssize_t variables. Asan report: ==10762== ERROR: AddressSanitizer: heap-use-after-free on address 0x602c00048700 at pc 0x7f667e468a00 bp 0x7f6675c42e20 sp 0x7f6675c42e10 READ of size 4 at 0x602c00048700 thread T9 #0 0x7f667e4689ff in notify_kernel_loop /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875 #1 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) #2 0x3cf4207ee4 in start_thread (/lib64/libpthread.so.0+0x3cf4207ee4) #3 0x3cf3ef4d1c in __clone (/lib64/libc.so.6+0x3cf3ef4d1c) 0x602c00048700 is located 64 bytes inside of 376-byte region [0x602c000486c0,0x602c00048838) freed by thread T9 here: #0 0x7f66860e00f9 (/lib64/libasan.so.0+0x160f9) #1 0x7f6685d5e6a4 in __gf_free /home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:336 #2 0x7f667e4689c4 in notify_kernel_loop /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3873 #3 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) previously allocated by thread T7 here: #0 0x7f66860e0315 (/lib64/libasan.so.0+0x16315) #1 0x7f6685d5d3be in __gf_calloc /home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:117 #2 0x7f667e4308b7 in fuse_invalidate_inode /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:295 #3 0x7f667e42f61c in fuse_invalidate /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:55 #4 0x7f6685d22071 in inode_invalidate /home/pk1/workspace/gerrit-repo/libglusterfs/src/inode.c:1158 #5 0x7f66790789ed in mdc_inode_iatt_set_validate /home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:427 #6 0x7f667907e5da in mdc_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:1040 #7 0x7f6685e3b57c in default_ftruncate_cbk /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333 #8 0x7f6685e3b57c in default_ftruncate_cbk /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333 #9 0x7f66796d52c6 in ioc_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/io-cache/src/io-cache.c:1327 #10 0x7f6679b0d33c in ra_truncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/read-ahead/src/read-ahead.c:704 #11 0x7f6679d38e90 in wb_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/performance/write-behind/src/write-behind.c:1693 #12 0x7f667a02a74e in dht_truncate_cbk /home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-inode-write.c:283 #13 0x7f667a2ee5fd in afr_ftruncate_unwind /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:646 #14 0x7f667a2e8200 in __afr_inode_write_cbk /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:171 #15 0x7f667a2ee7a0 in afr_ftruncate_wind_cbk /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:665 #16 0x7f667a610c79 in client3_3_ftruncate_cbk /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-rpc-fops.c:1512 #17 0x7f6685a82e45 in rpc_clnt_handle_reply /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759 #18 0x7f6685a83674 in rpc_clnt_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900 #19 0x7f6685a7a83a in rpc_transport_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541 #20 0x7f667b5cda53 in socket_event_poll_in /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231 #21 0x7f667b5ce720 in socket_event_handler /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344 #22 0x7f6685ddaf49 in event_dispatch_epoll_handler /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571 #23 0x7f6685ddb823 in event_dispatch_epoll_worker /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674 #24 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) Thread T9 created by T8 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685d18bf9 in gf_thread_create /home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468 #2 0x7f667e4691ee in fuse_init /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3946 #3 0x7f667e46fc64 in fuse_thread_proc /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:4935 #4 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) Thread T8 created by T5 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685d18bf9 in gf_thread_create /home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468 #2 0x7f667e471205 in notify /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:5170 #3 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #4 0x7f6685e58f97 in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2879 #5 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #6 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #7 0x7f6678e5e4bb in notify /home/pk1/workspace/gerrit-repo/xlators/debug/io-stats/src/io-stats.c:3838 #8 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #9 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #10 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #11 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #12 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #13 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #14 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #15 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #16 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #17 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #18 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #19 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #20 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #21 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #22 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #23 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #24 0x7f667a024ddc in dht_notify /home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-common.c:7888 #25 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #26 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #27 0x7f667a38f3ff in afr_notify /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-common.c:4021 #28 0x7f667a3968be in notify /home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr.c:34 #29 0x7f6685cf6faa in xlator_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492 #30 0x7f6685e5903b in default_notify /home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885 #31 0x7f667a5dc91a in client_notify_dispatch /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:83 #32 0x7f667a5dc761 in client_notify_dispatch_uniq /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:61 #33 0x7f667a64f7d2 in client_notify_parents_child_up /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:133 #34 0x7f667a65551a in client_post_handshake /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1053 #35 0x7f667a65637b in client_setvolume_cbk /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1210 #36 0x7f6685a82e45 in rpc_clnt_handle_reply /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759 #37 0x7f6685a83674 in rpc_clnt_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900 #38 0x7f6685a7a83a in rpc_transport_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541 #39 0x7f667b5cda53 in socket_event_poll_in /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231 #40 0x7f667b5ce720 in socket_event_handler /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344 #41 0x7f6685ddaf49 in event_dispatch_epoll_handler /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571 #42 0x7f6685ddb823 in event_dispatch_epoll_worker /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674 #43 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) Thread T5 created by T0 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685ddba89 in event_dispatch_epoll /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:726 #2 0x7f6685d5b92f in event_dispatch /home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:124 #3 0x40eeb6 in main /home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2345 #4 0x3cf3e21d64 in __libc_start_main (/lib64/libc.so.6+0x3cf3e21d64) Thread T7 created by T5 here: #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a) #1 0x7f6685ddbfac in event_reconfigure_threads_epoll /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:834 #2 0x7f6685d5ba8b in event_reconfigure_threads /home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:140 #3 0x7f667a5f5f6c in client_check_event_threads /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2332 #4 0x7f667a5f69ec in init /home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2448 #5 0x7f6685cf665d in __xlator_init /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:399 #6 0x7f6685cf68b7 in xlator_init /home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:424 #7 0x7f6685d83a14 in glusterfs_graph_init /home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:320 #8 0x7f6685d84dec in glusterfs_graph_activate /home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:667 #9 0x40e4f4 in glusterfs_process_volfp /home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2186 #10 0x417168 in mgmt_getspec_cbk /home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd-mgmt.c:1640 #11 0x7f6685a82e45 in rpc_clnt_handle_reply /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759 #12 0x7f6685a83674 in rpc_clnt_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900 #13 0x7f6685a7a83a in rpc_transport_notify /home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541 #14 0x7f667b5cda53 in socket_event_poll_in /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231 #15 0x7f667b5ce720 in socket_event_handler /home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344 #16 0x7f6685ddaf49 in event_dispatch_epoll_handler /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571 #17 0x7f6685ddb823 in event_dispatch_epoll_worker /home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674 #18 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7) SUMMARY: AddressSanitizer: heap-use-after-free /home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875 notify_kernel_loop Shadow bytes around the buggy address: 0x0c0600001090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa 0x0c06000010a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd =>0x0c06000010e0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c06000010f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c0600001100: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa 0x0c0600001110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c0600001120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c0600001130: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap righ redzone: fb Freed Heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 ASan internal: fe ==10762== ABORTING fsync: Software caused connection abort Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. run iozone -a on a mount with address sanitizer enabled build and it crashes. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Vijay Bellur on 2015-12-06 12:21:03 EST --- REVIEW: http://review.gluster.org/12886 (mount/fuse: Fix use-after-free crash) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu) --- Additional comment from Vijay Bellur on 2015-12-06 23:57:41 EST --- COMMIT: http://review.gluster.org/12886 committed in master by Raghavendra G (rgowdapp) ------ commit 05b510bb893761864d3830eb781210445056a6f9 Author: Pranith Kumar K <pkarampu> Date: Sun Dec 6 22:05:54 2015 +0530 mount/fuse: Fix use-after-free crash fouh->len is accessed after 'node' is freed. Also 'rv' is int where as fouh->len is uint32, changed comparison to ssize_t variables. BUG: 1288857 Change-Id: Ied43d29e1e52719f9b52fe839cee31ce65711eea Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/12886 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> --- Additional comment from Vijay Bellur on 2016-01-21 17:02:20 EST --- REVIEW: http://review.gluster.org/13274 (fuse: use-after-free fix in fuse-bridge, revisited) posted (#1) for review on master by Kaleb KEITHLEY (kkeithle) --- Additional comment from Vijay Bellur on 2016-02-02 02:23:29 EST --- REVIEW: http://review.gluster.org/13274 (fuse: use-after-free fix in fuse-bridge, revisited) posted (#2) for review on master by Kaleb KEITHLEY (kkeithle) --- Additional comment from Vijay Bellur on 2016-02-02 05:10:14 EST --- REVIEW: http://review.gluster.org/13274 (fuse: use-after-free fix in fuse-bridge, revisited) posted (#3) for review on master by Kaleb KEITHLEY (kkeithle) --- Additional comment from Vijay Bellur on 2016-02-03 00:13:14 EST --- COMMIT: http://review.gluster.org/13274 committed in master by Raghavendra G (rgowdapp) ------ commit 29bd2316b6d4f522e1bd00e3c9a1c97dcc7d80ea Author: Kaleb S KEITHLEY <kkeithle> Date: Thu Jan 21 15:03:38 2016 -0500 fuse: use-after-free fix in fuse-bridge, revisited Prompted by the email exchange in gluster-devel between Oleksandr Natalenko, xavi, and soumyak, I looked at this because the fuse client on the longevity cluster has also been suffering from a serious memory leak for some time. (longevity cluster is currently running 3.7.6) The longevity cluster manifests the same kernel notifier loop terminated log message the Oleksandr sees, and some sample runs suggest that the length passed to the (sys_)write call is unexpectedly and abnormally large. Basically this fix a) uses correct types for len and rv, b) copies the len from potentially incorrectly aligned memory (in a way that should minimize potential performance issues related to accessing unaligned memory.) c) changes log level of the kernel notifier loop terminated message d) fixes a potential mutex lock/unlock issue Change-Id: Icedb3525706f59803878bb37ef6b4ffe4a986880 BUG: 1288857 Signed-off-by: Kaleb S KEITHLEY <kkeithle> Reviewed-on: http://review.gluster.org/13274 Smoke: Gluster Build System <jenkins.com> Reviewed-by: Xavier Hernandez <xhernandez> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra Bhat <raghavendra> Reviewed-by: Raghavendra G <rgowdapp>
Discussed with Dev. and found that there is no clear way of validating this bug as this is a code change with no direct functional impact. Hence moving it to verified However, following was what i did test: tried to check the lookups using profiling and didn't find difference b/w fuse client versions of 3.7.9-1(before fix) and 3.7.9-3(after fix) (the server versions for both were common 3.7.9-3) However dev. mentioned that this may not be right test Did a lot of IO and checked statedump before clearing client cache and after clearing client cache( searched for IO the hot-count in inode table ) didnt see much difference b/w the different client version
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240