We're carrying the CVE fixes Jeff opened bugs for on top of 3.13.  Not sure if those play into things.

Yep, it's due to one of those patches, but the problem was likely there before. We're just explicitly rejecting calls where the buffer length doesn't match what we think it should be.

I think I see the problem and will look at the best way to fix it. Stay tuned...

Klaus, what sort of server are you mounting here?

Steve, I see this in ip_rfc1001_connect():

                /* sizeof RFC1002_SESSION_REQUEST with no scope */
                smb_buf->smb_buf_length = cpu_to_be32(0x81000044);

Where does this value come from? This frame is clearly not >2G in length...

0x81 is a session request frame sent during NetBIOS over TCP session initialization, see section 4.3.2 of RFC 1002

0x81 is a session request frame sent during NetBIOS over TCP session initialization, see section 4.3.2 of RFC 1002

0x81 is a session request frame sent during NetBIOS over TCP session initialization, see section 4.3.2 of RFC 1002

The rfc 1001 length field was 17 bits long, the next 7 bits were unused and the high byte is the type field.  The rfc1001 length field was later extended to the full 24 bits which samba and cifs.ko and others support, but for all the top byte is the NetBIOS packet type.

Created attachment 866509 [details]
patch -- mask off top byte in get_rfc1002_length

Ahh right, thanks!

So I guess the real bug is that we ought to be masking off that top byte in get_rfc1002_length()? Something like this patch should fix it (though I haven't tested it other than to see if it'll compile).

Yes

Yeah. In fact, that bug looks like it's probably also been causing this if condition to return true in smb_send_rqst even on a successful send:

        if ((total_len > 0) && (total_len != smb_buf_length + 4)) {
                cifs_dbg(FYI, "partial send (wanted=%u sent=%zu): terminating session\n",
                         smb_buf_length + 4, total_len);
                /*
                 * If we have only sent part of an SMB then the next SMB could
                 * be taken as the remainder of this one. We need to kill the
                 * socket so the server throws away the partial SMB
                 */
                server->tcpStatus = CifsNeedReconnect;
        }

...but that tcpStatus gets clobbered soon afterward so it doesn't seem to actually be problematic. Still, seems worth fixing that too and the patch in comment #9 ought to take care of it.

Klaus, would you be able to test it?

(In reply to Jeff Layton from comment #3)
> Klaus, what sort of server are you mounting here?

TeraStation from Buffalo - an older NAS thingie

|<

(In reply to Jeff Layton from comment #11)
> Yeah. In fact, that bug looks like it's probably also been causing this if
> condition to return true in smb_send_rqst even on a successful send:
> 
>         if ((total_len > 0) && (total_len != smb_buf_length + 4)) {
>                 cifs_dbg(FYI, "partial send (wanted=%u sent=%zu):
> terminating session\n",
>                          smb_buf_length + 4, total_len);
>                 /*
>                  * If we have only sent part of an SMB then the next SMB
> could
>                  * be taken as the remainder of this one. We need to kill the
>                  * socket so the server throws away the partial SMB
>                  */
>                 server->tcpStatus = CifsNeedReconnect;
>         }
> 
> ...but that tcpStatus gets clobbered soon afterward so it doesn't seem to
> actually be problematic. Still, seems worth fixing that too and the patch in
> comment #9 ought to take care of it.
> 
> Klaus, would you be able to test it?

I would have no problem testing a new kernel, I can't compile it, though. My relatively small SSD is almost full.

|<

Ok, building a scratch kernel now:

    http://koji.fedoraproject.org/koji/taskinfo?taskID=6561298

...if you can test it and report back then I'll plan to send the patch to the mailing list next week and we'll go ahead and incorporate it in Fedora kernels.

This should only be a problem for people who are connecting to the server via port 139. Anyone connecting via port 445 (or other ports) shouldn't hit this.

I'm also seeing this.  I mount a OS/2 share using port 139

*** Bug 1069411 has been marked as a duplicate of this bug. ***

I see this when trying to talk to an appliance which runs Win95 internally.

I'm running F19, if there is a somewhat easy way to test your patch I'm happy to. I would start with getting the kernel.src.rpm?

Hans, yeah you can build a f19 kernel from the f20 sources, or alternately just try the f20 kernel.

I posted the patch this morning:

    http://article.gmane.org/gmane.linux.kernel.cifs/9419

Josh, can you pull that into f19 and f20? I guess f18 is EOL'ed and rawhide will get it eventually once Steve merges it up to Linus.

Jeff, the kernel you posted on Koji works fine for me. Thank you for all your work!

Applied the patch from comment #19 to f19-rawhide.  Thanks Jeff!

I have tested the kernel provided in 
https://bugzilla.redhat.com/show_bug.cgi?id=1068862#c14
and can confirm that it fixes the issue.

The patch was also backported to RHEL 7 which contains the patches which introduced this regression. The patch fixes the problem there too.

Sachin Prabhu

kernel-3.13.5-101.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/FEDORA-2014-2887/kernel-3.13.5-101.fc19

Package kernel-3.13.5-101.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.13.5-101.fc19'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-2887/kernel-3.13.5-101.fc19
then log in and leave karma (feedback).

i've updated to 3.13.4-200.fc20.x86_64 kernel. But i still have problem with mount.
When i try mount i see the following errors:
mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)


And in messages log i see the following:

Feb 27 08:12:07 localhost kernel: [ 1035.881053] ------------[ cut here ]------------
Feb 27 08:12:07 localhost kernel: [ 1035.881074] WARNING: CPU: 0 PID: 10003 at fs/cifs/transport.c:313 smb_send_rqst+0x215/0x270 [cifs]()
Feb 27 08:12:07 localhost kernel: [ 1035.881076] Send length mismatch(send_length=72 smb_buf_length=2164260932)
Feb 27 08:12:07 localhost kernel: [ 1035.881078] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables usb_storage nls_utf8 cifs dns_resolver fscache bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek iTCO_wdt snd_hda_codec_hdmi gpio_ich iTCO_vendor_support r8169 coretemp snd_hda_intel snd_hda_codec snd_hwdep mii i2c_i801 lpc_ich mfd_core snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer snd shpchp soundcore serio_raw microcode acpi_cpufreq binfmt_misc i915 video i2c_algo_bit drm_kms_helper firewire_ohci firewire_core ata_generic pata_acpi crc_itu_t drm i2c_core
Feb 27 08:12:07 localhost kernel: [ 1035.881124] CPU: 0 PID: 10003 Comm: mount.cifs Tainted: G        W    3.13.4-200.fc20.x86_64 #1
Feb 27 08:12:07 localhost kernel: [ 1035.881127] Hardware name: Gigabyte Technology Co., Ltd. EG41MF-S2H/EG41MF-S2H, BIOS F3 11/11/2008
Feb 27 08:12:07 localhost kernel: [ 1035.881129]  0000000000000009 ffff88006bc2bbb8 ffffffff81686b68 ffff88006bc2bc00
Feb 27 08:12:07 localhost kernel: [ 1035.881133]  ffff88006bc2bbf0 ffffffff8106d47d ffff88006bc2bd08 ffff880137505400
Feb 27 08:12:07 localhost kernel: [ 1035.881137]  ffff88006bc2bcd0 0000000000000001 0000000000000000 ffff88006bc2bc50
Feb 27 08:12:07 localhost kernel: [ 1035.881141] Call Trace:
Feb 27 08:12:07 localhost kernel: [ 1035.881149]  [<ffffffff81686b68>] dump_stack+0x45/0x56
Feb 27 08:12:07 localhost kernel: [ 1035.881155]  [<ffffffff8106d47d>] warn_slowpath_common+0x7d/0xa0
Feb 27 08:12:07 localhost kernel: [ 1035.881159]  [<ffffffff8106d4ec>] warn_slowpath_fmt+0x4c/0x50
Feb 27 08:12:07 localhost kernel: [ 1035.881170]  [<ffffffffa03b9d55>] smb_send_rqst+0x215/0x270 [cifs]
Feb 27 08:12:07 localhost kernel: [ 1035.881181]  [<ffffffffa03b9ded>] smb_sendv+0x3d/0x40 [cifs]
Feb 27 08:12:07 localhost kernel: [ 1035.881191]  [<ffffffffa03b9e18>] smb_send+0x28/0x30 [cifs]
Feb 27 08:12:07 localhost kernel: [ 1035.881200]  [<ffffffffa03a401d>] generic_ip_connect+0x30d/0x3e0 [cifs]
Feb 27 08:12:07 localhost kernel: [ 1035.881206]  [<ffffffff8119e785>] ? __kmalloc+0x55/0x240
Feb 27 08:12:07 localhost kernel: [ 1035.881215]  [<ffffffffa03a84fc>] cifs_mount+0x84c/0xae0 [cifs]
Feb 27 08:12:07 localhost kernel: [ 1035.881222]  [<ffffffffa0396fa0>] cifs_do_mount+0xa0/0x4d0 [cifs]
Feb 27 08:12:07 localhost kernel: [ 1035.881227]  [<ffffffff811bc8f9>] mount_fs+0x39/0x1b0
Feb 27 08:12:07 localhost kernel: [ 1035.881231]  [<ffffffff811d6da3>] vfs_kern_mount+0x63/0xf0
Feb 27 08:12:07 localhost kernel: [ 1035.881235]  [<ffffffff811d93be>] do_mount+0x23e/0xac0
Feb 27 08:12:07 localhost kernel: [ 1035.881239]  [<ffffffff8114ef74>] ? __get_free_pages+0x14/0x50
Feb 27 08:12:07 localhost kernel: [ 1035.881242]  [<ffffffff811d9006>] ? copy_mount_options+0x36/0x170
Feb 27 08:12:07 localhost kernel: [ 1035.881246]  [<ffffffff811d9f63>] SyS_mount+0x83/0xc0
Feb 27 08:12:07 localhost kernel: [ 1035.881251]  [<ffffffff81695b69>] system_call_fastpath+0x16/0x1b
Feb 27 08:12:07 localhost kernel: [ 1035.881253] ---[ end trace 2d5b06ad4e1a6da1 ]---
Feb 27 08:12:07 localhost kernel: [ 1035.883020] CIFS VFS: Error connecting to socket. Aborting operation.
Feb 27 08:12:07 localhost kernel: [ 1035.886327] CIFS VFS: cifs_mount failed w/return code = -5


it seems like we still have problem with smb_send_rqst.

Anton,

The fix will be included in kernel-3.13.5-101.fc19
You are running 3.13.4-200.fc20.x86_64. This doesn't contain the fix.

*** Bug 1070893 has been marked as a duplicate of this bug. ***

Another duplicate https://bugzilla.redhat.com/show_bug.cgi?id=1066499

*** Bug 1066499 has been marked as a duplicate of this bug. ***

kernel-3.13.5-101.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

I am still seeing this problem with Fedora 20 / kernel-3.13.5-200.fc20.x86_64, which I would have assumed would have contained this fix based on Comment #26, but there's no mention of it in the changelog...

Any ETA on a fix for Fedora 20, because having to choose between an insecure kernel and being able to backup data isn't easy... :-/

Looks like f20 hasn't had a kernel built with this patch yet, where f19 has. I do see that the patch has been merged though so the next one ought to get it. Josh, any idea when f20 will get the kernel with this patch?

It looks like this got closed when the f19 package got pushed, so I'll move it back to MODIFIED until the f20 package is.

F20 will get a build today for unrelated reasons.  We'll push that out.  Apologies on missing this.

kernel-3.13.5-202.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.13.5-202.fc20

Package kernel-3.13.5-202.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.13.5-202.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-3442/kernel-3.13.5-202.fc20
then log in and leave karma (feedback).

kernel-3.13.5-202.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment #22 indicates the fix was backported to RHEL 7. How about RHEL 6 that also introduced the same regression in the latest update (2.6.32-431.11.2.el6)?

Akemi,

An internal bugzilla has been open for that case and the proposed patch posted. We should see a kernel with the fix soon.

Sachin Prabhu

That is great. Thanks for the note, Sachin.

Any ETA for the RHEL 6 fix?  Specifically 6.5.

Could anyone tell me if it is possible to patch the kernel myself for a RHEL 6/Centos 6? I have done a small amount of kernel work but cannot find the file cifsglob.h in the patch.  Its been a few years since I did any kernel compiling.  Thanks in advance.

(In reply to Alex Kiresuk from comment #42)
> Could anyone tell me if it is possible to patch the kernel myself for a RHEL
> 6/Centos 6? I have done a small amount of kernel work but cannot find the
> file cifsglob.h in the patch.  Its been a few years since I did any kernel
> compiling.  Thanks in advance.

If you cannot wait for the kernel update that has the fix and really need to build your own, follow the instructions on this CentOS wiki:

http://wiki.centos.org/HowTos/Custom_Kernel

You may be able to build just the cifs kernel module. Please see:

http://wiki.centos.org/HowTos/BuildingKernelModules

Tried to build with the patch today but got this error-
(Centos 6.5 kernel-2.6.32-431.11.2.el6)

+ make -s ARCH=x86_64 V=1 -j16 modules
In file included from fs/cifs/cifsfs.c:43:
fs/cifs/cifsglob.h: In function 'get_rfc1002_length':
fs/cifs/cifsglob.h:237: error: lvalue required as unary '&' operand
make[2]: *** [fs/cifs/cifsfs.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from fs/cifs/cifssmb.c:39:
fs/cifs/cifsglob.h: In function 'get_rfc1002_length':
fs/cifs/cifsglob.h:237: error: lvalue required as unary '&' operand
make[2]: *** [fs/cifs/cifssmb.o] Error 1
make[1]: *** [fs/cifs] Error 2
make[1]: *** Waiting for unfinished jobs....

clearly its in the change- likely a simple thing but it is beyond me to fix.

Thanks

We've seen the same kernel trace mounting shares from Windows NT4 (tcp/139) server to Centos 6.5 (on 2.6.32-431.11.2.el6.x86_64) for backup. Will 6.5 get the patch? 

------------[ cut here ]------------
WARNING: at fs/cifs/transport.c:312 smb_send_rqst+0x236/0x280 [cifs]() (Tainted: G        W  ---------------   )
Hardware name: (cut)
Send length mismatch(send_length=72 smb_buf_length=2164260932)
Modules linked in: nls_utf8 cifs vboxpci(U) vboxnetadp(U) vboxnetflt(U) vboxdrv(U) 8021q garp stp llc tun ipt_REDIRECT iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev iTCO_wdt iTCO_vendor_support serio_raw parport_pc parport snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc tg3 ptp pps_core sg lpc_ich mfd_core shpchp e752x_edac edac_core ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic ata_piix usb_storage nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi video output wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 32608, comm: mount.cifs Tainted: G        W  ---------------    2.6.32-431.11.2.el6.x86_64 #1
Call Trace:
 [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
 [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
 [<ffffffffa04eb816>] ? smb_send_rqst+0x236/0x280 [cifs]
 [<ffffffffa04eb8b0>] ? smb_send+0x50/0x60 [cifs]
 [<ffffffffa04d70b6>] ? generic_ip_connect+0x376/0x4d0 [cifs]
 [<ffffffffa04d79b9>] ? cifs_get_tcp_session+0x7a9/0x900 [cifs]
 [<ffffffffa04dafdd>] ? cifs_mount+0x4d/0x690 [cifs]
 [<ffffffffa04c836f>] ? cifs_get_sb+0xaf/0x2a0 [cifs]
 [<ffffffff8118b8cb>] ? vfs_kern_mount+0x7b/0x1b0
 [<ffffffff8118ba72>] ? do_kern_mount+0x52/0x130
 [<ffffffff811aca1b>] ? do_mount+0x2fb/0x930
 [<ffffffff811ad0e0>] ? sys_mount+0x90/0xe0
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
---[ end trace 280bfac4fd4b5e43 ]---
CIFS VFS: Error connecting to socket. Aborting operation
CIFS VFS: cifs_mount failed w/return code = -5

Alex, Chris, 

The fix is scheduled to be released on the next async errata release for RHEL 6.5.

You just need the change to a single line in the module.
 get_rfc1002_length(void *buf)
 {
-	return be32_to_cpu(*((__be32 *)buf));
+	return be32_to_cpu(*((__be32 *)buf)) & 0xffffff;
 }

If you have a support subscription for RHEL, you can contact support for a supported hotfix package. 

Sachin Prabhu

ref: RHEL 6 bz - 1085358 (This is a private bz)

Hi - I wanted to confirm kernel 2.6.32-431.17.1.el6.x86_64 on 6.5 fixed this for us. Thanks!

Hi Cris- Does not look like it - I am waiting too.

Reviewing the BZ it looks like the fix was not slated for inclusion within kernel-2.6.32-431.17.1.el6.x86_64. I'll continue to monitor the BZ for updates and will relay any changes in status accordingly.

Regards,

Red Hat Support

Alex,

I can confirm that the fix _was_ included in kernel-2.6.32-431.17.1.el6.

https://rhn.redhat.com/errata/RHSA-2014-0475.html

I have checked the sources and the patch is definitely there.

Sachin Prabhu

HI Sachin-

Interesting- why would redhat support tell me it was not?  I do now see it in the references section.

Alex Kiresuk