Description of problem: The kernel panics in | > CIFS VFS: Send error in SessSetup = -88 | > CIFS VFS: Unexpected lookup error -88 Version-Release number of selected component (if applicable): kernel-2.6.18-194 How reproducible: Rarely Steps to Reproduce: 1. Use cifsd 2. Wait ~280 days.. Additional info: I've attempted to trace the code crash-5.1.2> bt PID: 11263 TASK: e68ad000 CPU: 1 COMMAND: "cifsd" #0 [d5c7bd80] crash_kexec at c04426da #1 [d5c7bdc4] die at c040649f #2 [d5c7bdf4] do_page_fault at c061e611 #3 [d5c7be44] error_code (via page_fault) at c0405a87 EAX: 00000000 EBX: c069c9a0 ECX: 00000004 EDX: 00000000 EBP: 00000004 DS: 007b ESI: 00000000 ES: 007b EDI: 00000000 CS: 0060 EIP: c05b5efe ERR: ffffffff EFLAGS: 00010246 #4 [d5c7be78] sock_recvmsg at c05b5efe #5 [d5c7bf54] kernel_recvmsg at c05b7d6d #6 [d5c7bf64] cifs_demultiplex_thread at e8cf99de [cifs] #7 [d5c7bfcc] kthread at c0436339 #8 [d5c7bfe4] kernel_thread_helper at c0405c51 This is odd, as the ctcpStatus for this struct is passing a null socket, but the tcpStatus of this server_info is null (which I guess corresponds to why there is an error 88 ( ENOTSOCK 88 /* Socket operation on non-socket */ ). crash-5.1.2> struct TCP_Server_Info.srv_count,hostname,ssocket,tcpStatus,lstrp 0xdc5b2000 srv_count = 4, hostname = 0xd62b2ec0 "localhost", ssocket = 0x0, <====== here tcpStatus = CifsGood, lstrp = 2768191999 crash-5.1.2> Jlayton asked me to create this bug and let him know.
I've looked over the code but simply don't see it. This code is really too complicated for anyone's good, but basically the recreation and reconnection of the socket is supposed to be done by cifsd. When a reconnect event occurs, then cifsd will close down the socket and set ssocket to NULL, and then try to create a new socket and connect it. It shouldn't return until that has successfully occurred. The above stack trace though makes it look like it did happen. There is one possibility -- it could be that there was a flurry of reconnect/disconnect activity, cifs_setup_session raced in and reset the tcpStatus to CifsGood while cifsd was trying (and failing) to reconnect the socket. That would probably explain what happened... The fundamental problem here though is that the tcpStatus has no clear locking rules around it. This will probably require a fairly fundamental overhaul to fix it correctly.
Note that there is a discussion about a very similar problem going on upstream. I think I have a patch that may fix this there, but it will need to be backported for RHEL5: http://article.gmane.org/gmane.linux.kernel.cifs/3402
I've posted a set of test kernels that contain patches for this issue on my people.redhat.com page: http://people.redhat.com/jlayton/ ...could you test them and let me know if they resolve the issue.
The customer in case sadly can't reproduce the issue, and is unwilling to run a test kernel on their environment.
That's a pity, but understandable. Based on the analysis, I think this patch will probably fix the issue. I'll plan to go ahead with it for 5.8.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Patch(es) available in kernel-2.6.18-283.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0150.html