Hide Forgot
Created attachment 635
(In reply to comment #1) > Created an attachment (id=635) [details] > valgrind logs lakshmi, the functions in backtrace don't exist on master, is it master or some other branch?.
While running valgrind with master (12895212d64e2f209190f389a92be7f5e67ec84e) valgraind log from last nights run - ----------- ==14595== Invalid read of size 1 ==14595== at 0x4A08258: memcpy (mc_replace_strmem.c:402) ==14595== by 0x714C9A3: afr_sh_has_metadata_pending (afr-self-heal-common.c:663) ==14595== by 0x716CF8B: afr_lookup_collect_xattr (afr-common.c:571) ==14595== by 0x716E345: afr_revalidate_lookup_cbk (afr-common.c:995) ==14595== by 0x6EFFFD9: client3_1_lookup_cbk (client3_1-fops.c:1948) ==14595== by 0x4E8CB5B: rpc_clnt_handle_reply (rpc-clnt.c:757) ==14595== by 0x4E8CEBA: rpc_clnt_notify (rpc-clnt.c:870) ==14595== by 0x4E8A45B: rpc_transport_notify (rpc-transport.c:1043) ==14595== by 0x840F876: socket_event_poll_in (socket.c:1623) ==14595== by 0x840FC29: socket_event_handler (socket.c:1737) ==14595== by 0x4C5419C: event_dispatch_epoll_handler (event.c:812) ==14595== by 0x4C543AC: event_dispatch_epoll (event.c:876) ==14595== Address 0x83cb383 is 3 bytes after a block of size 8 alloc'd ==14595== at 0x4A0763E: malloc (vg_replace_malloc.c:207) ==14595== by 0x4C54C7C: __gf_malloc (mem-pool.c:160) ==14595== by 0x4C1A874: memdup (common-utils.h:292) ==14595== by 0x4C204BD: dict_unserialize (dict.c:2601) ==14595== by 0x6EFFD3A: client3_1_lookup_cbk (client3_1-fops.c:1920) ==14595== by 0x4E8CB5B: rpc_clnt_handle_reply (rpc-clnt.c:757) ==14595== by 0x4E8CEBA: rpc_clnt_notify (rpc-clnt.c:870) ==14595== by 0x4E8A45B: rpc_transport_notify (rpc-transport.c:1043) ==14595== by 0x840F876: socket_event_poll_in (socket.c:1623) ==14595== by 0x840FC29: socket_event_handler (socket.c:1737) ==14595== by 0x4C5419C: event_dispatch_epoll_handler (event.c:812) ==14595== by 0x4C543AC: event_dispatch_epoll (event.c:876) ==14595==
(In reply to comment #2) > (In reply to comment #1) > > Created an attachment (id=635) [details] [details] > > valgrind logs > > lakshmi, the functions in backtrace don't exist on master, is it master or some > other branch?. seems like it used some old binaries from 3.1.6 - I'm running test again with master,will update its results. (will also check with 3.1.6 after master)
running with master it shows invalid read at Invalid read of size 8 ==25908== at 0x71678D0: afr_update_read_child (afr-transaction.c:428) ==25908== by 0x7167AF9: afr_changelog_post_op (afr-transaction.c:481) ==25908== by 0x716BD3E: afr_transaction_resume (afr-transaction.c:1193) ==25908== by 0x715CE73: afr_writev_wind_cbk (afr-inode-write.c:123) ==25908== by 0x6F1A98D: client3_1_writev_cbk (client3_1-fops.c:685) ==25908== by 0x4E9D904: rpc_clnt_submit (rpc-clnt.c:1463) ==25908== by 0x6F18154: client_submit_vec_request (client3_1-fops.c:95) ==25908== by 0x6F26315: client3_1_writev (client3_1-fops.c:3613) ==25908== by 0x6F0FEC8: client_writev (client.c:817) ==25908== by 0x715D1D1: afr_writev_wind (afr-inode-write.c:151) ==25908== by 0x7169862: afr_changelog_pre_op_cbk (afr-transaction.c:731) ==25908== by 0x6F1DFB1: client3_1_fxattrop_cbk (client3_1-fops.c:1500) ==25908== Address 0xf1fc940 is 8 bytes before a block of size 16 alloc'd ==25908== at 0x4A05414: calloc (vg_replace_malloc.c:397) ==25908== by 0x4C5A206: __gf_default_calloc (mem-pool.h:83) ==25908== by 0x4C5A6B2: __gf_calloc (mem-pool.c:135) ==25908== by 0x71A2A8B: afr_transaction_local_init (afr-common.c:3489) ==25908== by 0x716BE68: afr_transaction (afr-transaction.c:1234) ==25908== by 0x715D441: afr_do_writev (afr-inode-write.c:225) ==25908== by 0x715D9F9: afr_writev (afr-inode-write.c:287) ==25908== by 0x73BD18D: wb_sync (write-behind.c:548) ==25908== by 0x73C375E: wb_do_ops (write-behind.c:1859) ==25908== by 0x73C3FD1: wb_process_queue (write-behind.c:2048) ==25908== by 0x73BC8BE: wb_sync_cbk (write-behind.c:405) ==25908== by 0x715CA75: afr_writev_unwind (afr-inode-write.c:69) ==25908== ==25908== Invalid read of size 4 ==25908== at 0x71678E0: afr_update_read_child (afr-transaction.c:428) ==25908== by 0x7167AF9: afr_changelog_post_op (afr-transaction.c:481) ==25908== by 0x716BD3E: afr_transaction_resume (afr-transaction.c:1193) ==25908== by 0x715CE73: afr_writev_wind_cbk (afr-inode-write.c:123) ==25908== by 0x6F1A98D: client3_1_writev_cbk (client3_1-fops.c:685) ==25908== by 0x4E9D904: rpc_clnt_submit (rpc-clnt.c:1463) ==25908== by 0x6F18154: client_submit_vec_request (client3_1-fops.c:95) ==25908== by 0x6F26315: client3_1_writev (client3_1-fops.c:3613) ==25908== by 0x6F0FEC8: client_writev (client.c:817) ==25908== by 0x715D1D1: afr_writev_wind (afr-inode-write.c:151) ==25908== by 0x7169862: afr_changelog_pre_op_cbk (afr-transaction.c:731) ==25908== by 0x6F1DFB1: client3_1_fxattrop_cbk (client3_1-fops.c:1500) ==25908== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==25908==
Created attachment 639
(In reply to comment #5) > Created an attachment (id=639) [details] > fuse valgrind log BUG 3443, afr crash is also because of the same memory access failure. I have just sent a patch for it. We shall re-run this post the fix is merged and verify that this does not happen. Then I will close the bug.
ran the same test with 3.2.3qa3 - it didn't show the invalid reads there. Seems like this happens only with master but not 3.2.3qa3?
(In reply to comment #7) > ran the same test with 3.2.3qa3 - it didn't show the invalid reads there. Seems > like this happens only with master but not 3.2.3qa3? Could you test with 3.3.0qa4 and update.
checking against commit-id "d9ead57226faf82f3f1375a29e06d348625ba905" - invalid read error doesn't appear.