Description of problem: After a CIFS mount, it is impossible to read/write any file on it, the application that tries the IO hangs, and syslog messages appear in the console in a loop : -- kernel: CIFS VFS: server not responding kernel: CIFS VFS: No response for cmd 162 mid 380 kernel: CIFS VFS: RFC1001 size 135 bigger than SMB for Mid=384 -- The kernel version used is : 2.6.18-8.1.6.el5 686 The CIFS server used is : EMC NAS model is EMC NS502G with firmware 5.5.19. This issue has not been reproduced so far in other CIFS servers such as RHEL4 samba 3.0.10. Additional info: https://bugzilla. novell.com/show_bug.cgi?id=247090, a similar problem reported on OpenSUSE. It seems to be a problem in EMC.
Created attachment 158133 [details] patch -- request extended info on CIFS open operations This patch makes CIFS request extended information in open operations. Windows apparently always requests this info with create operations. The EMC NAS appliance assumes that all clients and never actually checks to see. So when we get the response, there's too much data in it and the client complains about possible buffer overrun. The workaround here is to mimic what windows does and request this extra info, even though we don't actually do anything with it. This means that these packets will be slightly larger on the wire, but other than that there shouldn't be any negative effects.
Note that the above patch is unconfirmed, AFAIK. We need to have someone having this problem test it. I'm building a set of test kernels now that contain this patch.
I've posted a set of RHEL5 test kernels with this patch on my people page: http://people.redhat.com/jlayton/ ...could you have the customer test these someplace non-critical and see if they resolve the issue? I'd feel better about proposing this for 5.1 if we have some confirmation that it fixes the issue.
Got an email from customer that the peoplepage kernels didn't help. We'll be working via the ticketing system from here on out though.
This patch also doesn't seem to be quite correct. It does this: pSMB->OpenFlags |= REQ_EXTENDED_INFO; ...but maybe it should be something like this? pSMB->OpenFlags |= cpu_to_le32(REQ_EXTENDED_INFO); ...also I'm guessing that makes the client request extra info from the server, but I'm guessing that we need to do something else to make the client actually expect this extra info in the reply. Though maybe this should be doing that and I'm missing something.
note that the endianness change above shouldn't matter if the customer's using x86 or x86_64
Created attachment 158340 [details] patch -- on a NTCreateX call, zero out the bcc Steve French clarified that his intent with that patch was to try to trick the server into filling out the BCC field correctly. Apparently, it still doesn't. I can only assume that Windows machines don't even look at the BCC in a NTCreateX reply. Here's a patch I've proposed in a private email to Steve. If I've gotten right, it should force the bcc to be 0 in a NTCreateX call and that should work around this bug. I'm not sure if I've gotten this right and whether we want to make this conditional on something, however, so awaiting his response.
Created attachment 158341 [details] patch -- on a NTCreateX call, zero out the bcc Steve French clarified that his intent with that patch was to try to trick the server into filling out the BCC field correctly. Apparently, it still doesn't. I can only assume that Windows machines don't even look at the BCC in a NTCreateX reply. Here's a patch I've proposed in a private email to Steve. If I've gotten right, it should force the bcc to be 0 in a NTCreateX call and that should work around this bug. I'm not sure if I've gotten this right and whether we want to make this conditional on something, however, so awaiting his response.
Created attachment 158342 [details] patch -- on a NTCreateX call, zero out the bcc Steve French clarified that his intent with that patch was to try to trick the server into filling out the BCC field correctly. Apparently, it still doesn't. I can only assume that Windows machines don't even look at the BCC in a NTCreateX reply. Here's a patch I've proposed in a private email to Steve. If I've gotten right, it should force the bcc to be 0 in a NTCreateX call and that should work around this bug. I'm not sure if I've gotten this right and whether we want to make this conditional on something, however, so awaiting his response.
Created attachment 158343 [details] patch -- on a NTCreateX call, zero out the bcc Steve French clarified that his intent with that patch was to try to trick the server into filling out the BCC field correctly. Apparently, it still doesn't. I can only assume that Windows machines don't even look at the BCC in a NTCreateX reply. Here's a patch I've proposed in a private email to Steve. If I've gotten right, it should force the bcc to be 0 in a NTCreateX call and that should work around this bug. I'm not sure if I've gotten this right and whether we want to make this conditional on something, however, so awaiting his response.
Created attachment 158344 [details] patch -- on a NTCreateX call, zero out the bcc Steve French clarified that his intent with that patch was to try to trick the server into filling out the BCC field correctly. Apparently, it still doesn't. I can only assume that Windows machines don't even look at the BCC in a NTCreateX reply. Here's a patch I've proposed in a private email to Steve. If I've gotten right, it should force the bcc to be 0 in a NTCreateX call and that should work around this bug. I'm not sure if I've gotten this right and whether we want to make this conditional on something, however, so awaiting his response.
I've built a new series of test kernels that contain the patch in comment #12. Please ask the customer to test these on a non-critical machine and let us know if the problem goes away. Note that this is truly just a test patch. A final patch (if any) will likely look different.
Adding EMC... Wayne, can you add the appropriate people in EMC for this issue?
Some things EMC could help us with here: 1) clarify what hw/sw revs have this bug (for our support folks) 2) clarify what's being done to fix the bug (an ETA would be great) 3) test the patch in comment #12 (or the kernels on my people page), and let us know if it's works around the issue ...any help is appreciated!
Added Xiangping and Li...
The following fix was added to the EMC Celerra DART code 5.5.27.5 which should resolve this issue. The fix is to take into account the extended response bit from the NTCreateX request in Create Flag field to return or not an extended response. Before the fix, the extended response was returned based on the client type we had determined during the negotiation.
Excellent. Does the patch also correct the BCC value in the response? I believe it's always supposed to be 0, but we were often seeing random values in that field, leading us to believe that it might be uninitialized.
This event sent from IssueTracker by rrajaram issue 125277
The Bcc was seeing 'random' because the client was expecting a short response (it seems the client was not taking into account the WordCount provided in the response). The Bcc was intrerpreted as placed in the extended info. As for in the fix, the client will receive a 'short' response, the WordCount is now as expected by the client and the Bcc is correctly set to 0.
Given that there is now a server-side patch, I'm going to propose that we close this as NOTABUG and recomment that we consider a release note so that people hitting this bug are directed to EMC to have their Celerra patched.
First pass at release note text: Some versions of EMC's Celerra product have a known bug in their handling of CIFS NTCreateX calls. This bug is characterized by kernel messages similar to the following, kernel: CIFS VFS: server not responding kernel: CIFS VFS: No response for cmd 162 mid 380 kernel: CIFS VFS: RFC1001 size 135 bigger than SMB for Mid=384 Programs doing I/O to the mountpoint will hang. Users experiencing this issue should contact EMC support, and reference EMC case number [need this info].
Thanks Jeff. will add this note to the 5.1 release notes when EMC case number has been supplied.
Andrius, could you track down the EMC case ID for this problem for the release note?
Wayne, do you happen to have the EMC case ID for this problem?
Hi Andrius. I have an internal Remedy case number at this time. I'm working on getting a PRIMUS case number assigned so CS will have something to reference. Regards, Wayne.
setting NEEDINFO=berthiaume_wayne release note will be added when we have the case number. thanks!
Hi! just thought I'd mention that I've just tested the i686 on Jeff's peoplepage, and it works well so far - all CIFS problems seem to have gone. I'm unable to upgrade our Celerra's Dart code, and this is a good stop-gap until we get a new NAS/SAN in two months. Thanks! Rob
Hi Wayne, do we have the case number yet? please advise that the deadline for release notes is August 1, 2007. thanks!
Hi Don. Truely a tough question for you. In order for the PRIMUS case to be accurate, is it possible to disclose what versions of RHEL this issue will be seen? This BZ only references RHEL 5.0 but my understanding is it is seen in Fedora 6 as well. Thanks and regards, Wayne.
Hi Wayne, the release notes i'm doing right now are specific to RHEL5.1 (i am not part of the Fedora documentation team). anyhow, won't you be able to provide a case number for the RHEL5.1 release notes with the understanding / assumption that only RHEL is affected? if you need a complete list of affected systems, perhaps Jeff Layton can help us out? as of now, setting NEEDINFO=wayne again. thanks!
Andrius has provided the EMC case number via email. below is the RHEL5.1 release note for this issue, added under "Known Issues": <quote> Some versions of the EMC Celerra are unable to properly handle CIFS NTCreateX calls. Programs performing an I/O to the mountpoint will hang, following these kernel messages: kernel: CIFS VFS: server not responding kernel: CIFS VFS: No response for cmd 162 mid 380 kernel: CIFS VFS: RFC1001 size 135 bigger than SMB for Mid=384 If you encounter this issue, please contact EMC and reference case number 19189788. </quote> please advise if any revisions are in order. thanks!
The EMC Primus case number is emc165978. ID: emc165978 Domain: EMC1 Solution Class: 3.X Compatibility Goal After recent Linux upgrade, accessing CIFS shares on EMC NAS causes Linux CIFS client to hang or possibly panic Fact Product: Celerra Fact Protocol: Server Message Block (SMB) Fact Protocol: Common Internet File System (CIFS) Fact OS: SuSE Linux 10.2 (2.6.18 Kernel, 1.45 CIFS) Fact OS: Fedora Core 6 (2.6.20 Kernel, 1.47 CIFS) Fact EMC SW: NAS Code 5.5.26.x and below Symptom Accessing CIFS shares on EMC NAS causes Linux CIFS client to hang or possibly panic Symptom Celerra server log contains the following or similar error messages when a client attempts to read a file: 2007-03-30 15:25:19: SMB: 3: Client=10.0.0.1 OS='Linux version 2.6.20- 1.2925.fc6', LM='CIFS VFS Client for Linux' not registered capa=0xd0dc (R=8/8) 2007-03-30 15:25:19: SMB: 3: Client=10.0.0.1 OS=Linux version 2.6.20- 1.2925.fc6 LM=CIFS VFS Client for Linux Extra=- type=- (1) 2007-03-30 15:25:19: SMB: 3: Client=10.0.0.1 OS='Linux version 2.6.20- 1.2925.fc6', LM='CIFS VFS Client for Linux' not registered capa=0xd0dc (R=8/8) 2007-03-30 15:25:19: SMB: 3: Client=10.0.0.1 OS=Linux version 2.6.20- 1.2925.fc6 LM=CIFS VFS Client for Linux Extra=- type=- (1) Change Recent upgrade to Linux client Cause EMC DART OS was modified to return extended response to NTcreateX only if the bit 0x10 is set in the Create Flag, instead of assume the extended info are required for W2K client (see below). With the code modification, the extended response is only returned when the Extended Response Create Flag is set. Affected versions include: Fedora Core 6 (32-bit) Kernel: 2.6.20-1.2925 CIFS Module Version: 1.47 SuSe Linux 10.2 Kernel: 2.6.18.2-34-default CIFS Module Version: 1.45 SMB Header NT Create AndX Request Create Flags: 0x00000010 .... .... .... .... .... .... ...1 .... = Extended Response: Extended responses required .... .... .... .... .... .... .... 0... = Create Directory: Target of open can be a file .... .... .... .... .... .... .... .0.. = Batch Oplock: Does NOT request batch oplock .... .... .... .... .... .... .... ..0. = Exclusive Oplock: Does NOT request oplock Fix Upgrade to NAS Code 5.5.27.5 or later
As best I can tell, this will be an issue on all versions of RHEL4 and RHEL5, though with versions before 4.5, the error will look a bit different. It'll probably also be seen on any relatively recent version of fedora too (>=FC3). Not sure about anything before that...
I'm not exactly sure what this bug is asking for now. Appears this issue has been identified as a server-side fix and we've prepared a release note pointing affected users to EMC for a patch. Sounds like there's nothing left for Red Hat to do, save making sure the release note lands??
Yes, the bug is array side and fixed as noted in comment #37 with an upgrade to the Celerra DART code. All that is left for Red Hat is docuemtation in the event a customer looks to Red Hat for the solution. EMC has the bug documented in its PRIMUS case for use by both customersand Cs to dtermine bug and fix. Should be able to close this once Red Hat has documentation in place.
This issue will become a KBASE entry close to as follows: =============================== Title: Why does the CIFS client on Red Hat Enterprise Linux versions 4 or 5 hang when accessing shares on EMC NAS? Some versions of EMC's Celerra product (NAS Code 5.5.26.x and below), have a known bug in their handling of CIFS NTCreateX calls. This issue is characterized by kernel messages similar to the following: kernel: CIFS VFS: server not responding kernel: CIFS VFS: No response for cmd 162 mid 380 kernel: CIFS VFS: RFC1001 size 135 bigger than SMB for Mid=384 After a CIFS mount, it is impossible to read/write any file on it and the application that tries the I/O hangs. To solve this, upgrade to NAS Code 5.5.27.5 or later. The EMC Primus case number is emc165978. ===============================
Don - I'll leave it to you if you think this needs a release note in addition to the KBASE article Evan is creating. Please set to CLOSED/NOTABUG after your decision to close this issue completely.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Andrius, i think it's best to have the note present in both Kbase and Release Note. as such, adding the following under "Other Updates" (integrating it with a note on CIFS update to 1.48aRH: <quote> Note that for users of the EMC Celerra product (NAS Code 5.5.26.x and below) the CIFS client hangs when accessing shares on EMC NAS. This issue is characterized by the following kernel messages: kernel: CIFS VFS: server not responding kernel: CIFS VFS: No response for cmd 162 mid 380 kernel: CIFS VFS: RFC1001 size 135 bigger than SMB for Mid=384 After a CIFS mount, it becomes impossible to read/write any file on it and any application that attempts an I/O on the mountpoint will hang. To resolve this issue, upgrade to NAS Code 5.5.27.5 or later. The EMC Primus case number is emc165978. </quote> closing this bug. thanks!
The kbase submission has been accepted and is accessible at: http://kbase.redhat.com/faq/FAQ_85_11046 (RHEL4 category) http://kbase.redhat.com/faq/FAQ_103_11046 (RHEL5 category)
minor edit to release note: <quote> After a CIFS mount, it becomes impossible to read/write any file on it and any application that attempts an I/O on the mountpoint will hang. To resolve this issue, upgrade to NAS Code 5.5.27.5 or later (use EMC Primus case number emc165978). </quote>