Bug 498755

Summary: b43 wireless connection much slower on 2.6.29+ vs. 2.6.27.
Product: [Fedora] Fedora Reporter: Gideon Mayhak <gnafu_the_great>
Component: kernelAssignee: John W. Linville <linville>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 12CC: caiqian, itamar, jchadima, johannes, kernel-maint, mgrepl, ssorce, tmraz, yaneti
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-03-01 13:48:08 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
Output of lspci -nn none

Description Gideon Mayhak 2009-05-02 18:29:04 EDT
Description of problem:
Syncing local files from my laptop to my desktop over the network is much, much slower in Fedora 11 than Fedora 10.  It goes at ~100KB/s, rather than the ~2MB/s I get with Fedora 10.  I am syncing the same types of files from the same partition to the same destination over the same network.

Version-Release number of selected component (if applicable):
rsync-3.0.5-1.fc11.x86_64

How reproducible:
Every time, with various file types being rsync'd.

Steps to Reproduce:
1. Do an rsync like this:
rsync -aH --progress --delete-after --delete-excluded --include=/.grip* \
--include=/.cddb --include=/.VirtualBox --exclude=/.* ./ \
gideon@gidux.dyndns.org:~/Download/laptop-backup/home/gideon/

Actual results:
~100KB/s

Expected results:
~2MB/s

Additional info:
I'm using a script I created a while back, so the command is identical each time.  I've been using the same script since at least Fedora 9, if not 8.  Did rsync change the default transfer method?  If it's just something I need to add to the command to switch it to the old, faster method, let me know
Comment 1 Simo Sorce 2009-05-03 10:50:09 EDT
No, there has been no change like that, I suggest you check your hardware and NIC configuration before opening a bug against a software component.
Unless you have some sort of indication that rsync and only rsync is the key factor.

Can you install F10 rsync on the F11 box and see if there is any difference?
Comment 2 Gideon Mayhak 2009-05-03 13:53:08 EDT
Sorry to jump to conclusions.  rsync is the only place where I've noticed a difference in network performance, so that's where my mind went.  I'll try downgrading rsync and see what happens.
Comment 3 Gideon Mayhak 2009-05-04 21:14:56 EDT
I apologize for pointing the finger at rsync.  I see that the versions in F10 and F11 are essentially identical.  I tried an scp of a file both from my desktop and to my desktop under both F10 and F11.  Here's a quick rundown of the results:

From desktop to laptop in F10: ~2.2MB/s
To desktop from laptop in F10: ~2.2MB/s
Summary in F10: ~2.2MB/s both ways

From desktop to laptop in F11: ~600-700KB/s
To desktop from laptop in F11: ~100KB/s
Summary in F11: Much less

It looks like openssh has been updated from 5.1 to 5.2 in rawhide.  I'm guessing this is much more likely the culprit.  I'd rather not try downgrading that, as I tried and there were a lot of dependencies.  Let me know what information is needed to diagnose: config files, logs, etc.

Can someone please reassign this to openssh?
Comment 4 Gideon Mayhak 2009-05-04 21:16:19 EDT
Nevermind, I got it.

Reassigning to openssh.
Comment 5 Tomas Mraz 2009-05-05 02:49:38 EDT
Could you also try scp from F11 client to F10 server and vice versa?
Comment 6 Gideon Mayhak 2009-05-05 22:49:45 EDT
I'm sorry; I guess I never specified that my desktop is running Fedora 8 (I know, I need to upgrade; I plan on moving it to CentOS 5.3 soon).  I don't have another system available right now that I can put F10 or F11 on.  When I've said desktop, I've been referring to an install of F8.  The destination has been the same the whole time, at least since my laptop's been running F9.
Comment 7 Tomas Mraz 2009-05-06 02:55:39 EDT
So the ssh client was F10 or F11 and the ssh server was always F8 version?
Comment 8 Gideon Mayhak 2009-05-06 21:47:11 EDT
I'm sorry; reading back over my comments, I realize I haven't been very clear :-P.  Yes, the server has always been F8 and the client has been F10 and F11 (I should have used that terminology instead of desktop/laptop).  I've run all commands on my laptop.
Comment 9 Jan F. Chadima 2009-05-12 10:04:23 EDT
Can you set in the file /etc/ssh/ssh_config in both clients set the same ciphers and compare it again?
Comment 10 Yanko Kaneti 2009-05-12 10:54:23 EDT
2MB/s , this sounds like wireless ... is the laptop a wireless client ?
Is anything else going faster on that connection with F11? like wget(http) from somewhere...
Comment 11 Gideon Mayhak 2009-05-12 22:04:12 EDT
[gideon@gidux-laptop-rawhide ~]$ diff /mnt/fedora-10/etc/ssh/ssh_config /etc/ssh/ssh_config
1c1
< #	$OpenBSD: ssh_config,v 1.23 2007/06/08 04:40:40 pvalchev Exp $
---
> #	$OpenBSD: ssh_config,v 1.25 2009/02/17 01:28:32 djm Exp $
40c40
< #   Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
---
> #   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
45a46
> #   VisualHostKey no
Comment 12 Gideon Mayhak 2009-05-12 22:09:45 EDT
Looks like there is a difference.  I have made no changes to the configuration of either, so these should be the defaults.  As for the network, I'm using the same wireless connection under F10 and F11.  I'll check a local wget to be sure, though.  I might end up reassigning this to the kernel (I'm using the in-kernel b43 driver for my wireless).
Comment 13 Gideon Mayhak 2009-05-12 22:29:45 EDT
Well, this sucks.  I thought I had done other transfers without a noticeable difference, so I only blamed scp.  I just did a wget from my F8 box's HTTP server under F10 and F11.  F10 is 2+MB/s, whereas F11 averaged around 600-800KB/s.

Is there anything else in the system that would have an effect on scp and wget performance, other than the wireless driver?  Is there any other component I should switch this to other than the kernel?

Also, I'll confirm whether a hardwired connection is affected by this, to make sure whether it's my wireless or not.
Comment 14 Gideon Mayhak 2009-05-12 22:45:34 EDT
Okay, here are the quick results from my wired test:

wget from local HTTP server
    F10 & F11 - 2.5-2.8MB/s, peaking at 3.0MB/s

scp from local SSH server
    F10 - 2.7-2.8MB/s, peaking at 2.9MB/s
    F11 - 2.6-2.7MB/s, peaking at 2.8MB/s

So, scp is still just a smidgen slower, but not noticeably.  Do you think that could be due to the difference in the cipher lines?  I'd rather leave it default with such a small (0.1MB/s) difference, especially if there are other benefits (you must have had a good reason for changing it, right?).

Anyway, shall I reassign to kernel as a b43 wireless bug, or is there something else I should point the finger at?  I'm sorry for pointing it at rsync and scp!
Comment 15 Tomas Mraz 2009-05-13 02:54:29 EDT
Reassigning to kernel as this is obviously caused by the wireless driver.

Please leave the ciphers list as the default. The very small 0.1MB/s slowdown might even be caused by other means than the ciphers used. (If you really want faster, but slightly less secure ciphers you can put blowfish or arcfour first.)
Comment 16 Gideon Mayhak 2009-05-13 21:08:40 EDT
Also of note: every once in a while, the NetworkManager applet won't connect to my wireless network with the first try.  It will be connecting for a while and may eventually ask me to confirm the WEP key (which is correct, because I just have to click OK to connect almost instantly).  This never happened in F10.
Comment 17 Gideon Mayhak 2009-05-16 13:15:48 EDT
Any initial thoughts?  What kind of information would help in identifying the performance regression from 2.6.27 to 2.6.29?
Comment 18 Gideon Mayhak 2009-06-03 20:56:42 EDT
Ping.
Comment 19 Chuck Ebbert 2009-06-07 05:31:27 EDT
Can you post the output of the 'lspci -nn' command so we can see which wireless card you have? Also, if you still have the F10 install can you try the 2.6.29.4-75 kernel from the updates-testing repository?
Comment 20 Gideon Mayhak 2009-06-08 22:09:46 EDT
Created attachment 346955 [details]
Output of lspci -nn

Here is the output of lspci -nn.  I am about to install the testing kernel and reboot.  I'll let you know how it goes!
Comment 21 Gideon Mayhak 2009-06-08 22:28:33 EDT
2.6.29 in F10 exhibits exactly the same behavior as 2.6.29 in F11, so I guess we're definitely looking at a regression between 2.6.27 and 2.6.29.  Is there any other information I can provide to help track this down?  Thank you so much for looking at this.
Comment 22 Bug Zapper 2009-06-09 11:02:08 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 23 John W. Linville 2009-06-09 11:54:34 EDT
The biggest "likely suspect" is the change from the PID rate control algorithm
in 2.6.27 to the Minstrel algorithm used in 2.6.29.

I am building test F11 kernels w/ PID re-enabled (and made the default) here:

   http://koji.fedoraproject.org/koji/taskinfo?taskID=1401682

Please give them a try when the builds are completed and re-run your
performances tests (e.g. from comment 14).  Do they perform differently with
those kernels?
Comment 24 John W. Linville 2009-06-09 14:04:40 EDT
Build is complete -- get it while it's hot!
Comment 25 Gideon Mayhak 2009-06-10 21:26:08 EDT
Download seems to be slightly improved, coming in at 1.0-1.3MB/s using HTTP or SSH, but upload is still a very pitiful 100KB/s.  I guess that's not it (at least not entirely).
Comment 26 Gideon Mayhak 2009-07-08 19:48:34 EDT
Ping.  Anything else I can do to help this get moving?
Comment 27 John W. Linville 2009-07-09 11:18:12 EDT
Could you try setting a fixed bitrate?

   iwconfig wlan0 rate 54M fixed

You might try some other rates as well.  Standard 802.11g supports rates of 6, 9, 12, 18, 24, 36, 48, 54 in addition to the 802.11b rates of 1, 2, 5.5 and 11Mbps.

Do you get consistently better throughput with any of those?
Comment 28 Gideon Mayhak 2009-07-09 22:36:27 EDT
Downloading a file over HTTP after fixing to the following speeds:

 54M: ~150KB/s
 48M: ~150KB/s
 36M: ~150KB/s
 24M: ~150KB/s
 18M: ~150KB/s
 12M: ~150KB/s
 11M: ~2MB/s
  9M: ~150KB/s
  6M: ~150KB/s
5.5M: ~2MB/s
  2M: ~1.5MB/s
  1M: ~1MB/s

So basically, what you've revealed by having me do this, is that things only work well if they're running in 802.11b modes (11M, 5.5M, 2M, and 1M).  What's really odd to me is that none of those modes should be fast enough to accommodate 2MB/s, as that would require a 16Mbps or better connection.  Does this give you some good information to go on?

And thanks!

/me runs 'iwconfig wlan0 rate 11M fixed' and giggles.
Comment 29 Johannes Berg 2009-07-10 17:10:19 EDT
Well, you've fixed your _upload_ bitrate, and your http _download_ should not be affected by that all that much -- except for the TCP ack packets etc. of course... It's odd that it gives you such a huge difference, there must be something seriously wrong with the ofdm bitrates.
Comment 30 Gideon Mayhak 2009-07-10 21:25:05 EDT
Another thing I noticed, though it might've just been me, was that the HTTP connection seemed to be made a lot quicker at 11M, and web pages seem to load snappier.  Again, it might just be me.
Comment 31 Johannes Berg 2009-07-11 04:10:22 EDT
Well that too makes sense -- you need to do DNS queries etc. which is most of the time that you spend waiting for web pages since the actual download speed does't usually matter _that_ much (unless you're using a modem from the last millennium)...
Comment 32 Gideon Mayhak 2009-08-11 20:40:01 EDT
I just wanted to post a quick update.  I wasn't running a fully updated F11 install until last night (I'm a procrastinator).  The situation is the same.  Also, I no longer have an F10 install to test with (and I'm guessing that's not useful at this point).  I will be installing F12 Alpha on this laptop (alongside F11) once that's out, so I can see if there's been an improvement with the newer kernel.

Anything else I can do in the meantime as far as giving you information to work with?
Comment 33 Gideon Mayhak 2009-08-25 23:45:16 EDT
I don't know what you did, but it's working great now!  I believe it was as of the most recent kernel update in Fedora 11.  My download and upload are both now 2.5-2.9MB/s, without running any iwconfig commands first.

Thanks!
Comment 34 Gideon Mayhak 2009-08-26 00:02:32 EDT
Holy crap, I'm not too bright at the moment...

I had my laptop plugged into a wired network connection, because I was doing a network install of F12 Alpha, hence the increased performance.  I will try again with my wireless connection under both F11 and F12 Alpha in a short bit and update once more.

Sorry!
Comment 35 Gideon Mayhak 2009-08-26 01:05:25 EDT
Yep, still very slow under F12 Alpha.  Any ideas?
Comment 36 CAI Qian 2009-10-11 07:06:31 EDT
Not sure if it is the same problem here. The used working wireless in F11 is very slow in F12/rawhide at the moment. The ping does not show anything obvious, but it takes a long time seems forever even to connect to local wireless router's web interface via firefox.

02:02.0 Network controller: Broadcom Corporation BCM4318 [AirForce One 54g] 802.11g Wireless LAN Controller (rev 02)

Tried both 2.6.29 and 2.6.31 kernels, and 4.174.64.19 and 4.150.10.5 firmware without luck.

All firewalls are disabled.
Comment 37 Bug Zapper 2009-11-16 04:57:46 EST
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 38 John W. Linville 2010-02-01 12:38:32 EST
http://marc.info/?l=linux-wireless&m=126488652005514&w=2
Comment 39 John W. Linville 2010-02-02 14:48:16 EST
http://koji.fedoraproject.org/koji/taskinfo?taskID=1959523

Give those kernels a try when the build is complete?  Do they improve b43 throughput for you?
Comment 40 John W. Linville 2010-02-02 15:54:41 EST
Not sure why that one failed, but I think Koji burped.  Hopefully this one will succeed...

http://koji.fedoraproject.org/koji/taskinfo?taskID=1959563
Comment 41 Gideon Mayhak 2010-02-05 00:43:48 EST
While I don't currently have a local HTTP server set up to test local file transfers, I have done a bit of web surfing with the new kernel.  Before, surfing was a little sluggish, with pages taking a while to load.  Things seem much snappier now.  Of course, that is just speculation, and I can't give you any solid numbers at the moment, but it really does feel like a (big) improvement.
Comment 43 John W. Linville 2010-03-01 13:48:08 EST
The patch in question is part of 2.6.32.9.  I believe that current Koji F-12 kernels will resolve this issue:

http://koji.fedoraproject.org/koji/buildinfo?buildID=158902