+++ This bug was initially created as a clone of Bug #1340361 +++
Description of problem:
This was observed on one of user's production setup.
[raghu@unused 01610290]$ grep "cannot lookup the saved" <client-log>
[2016-02-15 22:40:02.575925] C [rpc-clnt.c:452:rpc_clnt_fill_request_info] <client-log>: cannot lookup the saved frame corresponding to xid (14161323)
Above is the log-message indicating that a reply from server couldn't lookup a saved-frame to unwind the response. The xid (0xd815ab) matches to one of the unaccounted call-bails. Similar msg was seen from other log.
As for the RCA for not able to lookup a saved frame for response, I saw in rpc-clnt:
1. Submit the request to transport for transmission to brick.
2. Save the frame for future reference while processing reply.
Now, if we get a response between 1 and 2 (before we were able to save the frame), we would be saving the frame of a request whose reply is already received. This can result in call-bail.
Version-Release number of selected component (if applicable):
Zero day bug, present in all releases
Racy. Not consistent
Steps to Reproduce:
A reply didn't have an associated saved-frame in rpc-clnt layer.
A reply should always have a saved-frame in rpc-clnt layer.
--- Additional comment from Vijay Bellur on 2016-05-27 09:36:29 CEST ---
REVIEW: http://review.gluster.org/14547 (rpc-clnt: save the frame before submitting request to transport) posted (#1) for review on master by Raghavendra G (email@example.com)
master patch : http://review.gluster.org/14547
This bug is being closed as GlusterFS-3.6 is nearing its End-Of-Life and only important security bugs will be fixed. This bug has been fixed in more recent GlusterFS releases. If you still face this bug with the newer GlusterFS versions, please open a new bug