Description of problem: Syncing local files from my laptop to my desktop over the network is much, much slower in Fedora 11 than Fedora 10. It goes at ~100KB/s, rather than the ~2MB/s I get with Fedora 10. I am syncing the same types of files from the same partition to the same destination over the same network. Version-Release number of selected component (if applicable): rsync-3.0.5-1.fc11.x86_64 How reproducible: Every time, with various file types being rsync'd. Steps to Reproduce: 1. Do an rsync like this: rsync -aH --progress --delete-after --delete-excluded --include=/.grip* \ --include=/.cddb --include=/.VirtualBox --exclude=/.* ./ \ gideon.org:~/Download/laptop-backup/home/gideon/ Actual results: ~100KB/s Expected results: ~2MB/s Additional info: I'm using a script I created a while back, so the command is identical each time. I've been using the same script since at least Fedora 9, if not 8. Did rsync change the default transfer method? If it's just something I need to add to the command to switch it to the old, faster method, let me know
No, there has been no change like that, I suggest you check your hardware and NIC configuration before opening a bug against a software component. Unless you have some sort of indication that rsync and only rsync is the key factor. Can you install F10 rsync on the F11 box and see if there is any difference?
Sorry to jump to conclusions. rsync is the only place where I've noticed a difference in network performance, so that's where my mind went. I'll try downgrading rsync and see what happens.
I apologize for pointing the finger at rsync. I see that the versions in F10 and F11 are essentially identical. I tried an scp of a file both from my desktop and to my desktop under both F10 and F11. Here's a quick rundown of the results: From desktop to laptop in F10: ~2.2MB/s To desktop from laptop in F10: ~2.2MB/s Summary in F10: ~2.2MB/s both ways From desktop to laptop in F11: ~600-700KB/s To desktop from laptop in F11: ~100KB/s Summary in F11: Much less It looks like openssh has been updated from 5.1 to 5.2 in rawhide. I'm guessing this is much more likely the culprit. I'd rather not try downgrading that, as I tried and there were a lot of dependencies. Let me know what information is needed to diagnose: config files, logs, etc. Can someone please reassign this to openssh?
Nevermind, I got it. Reassigning to openssh.
Could you also try scp from F11 client to F10 server and vice versa?
I'm sorry; I guess I never specified that my desktop is running Fedora 8 (I know, I need to upgrade; I plan on moving it to CentOS 5.3 soon). I don't have another system available right now that I can put F10 or F11 on. When I've said desktop, I've been referring to an install of F8. The destination has been the same the whole time, at least since my laptop's been running F9.
So the ssh client was F10 or F11 and the ssh server was always F8 version?
I'm sorry; reading back over my comments, I realize I haven't been very clear :-P. Yes, the server has always been F8 and the client has been F10 and F11 (I should have used that terminology instead of desktop/laptop). I've run all commands on my laptop.
Can you set in the file /etc/ssh/ssh_config in both clients set the same ciphers and compare it again?
2MB/s , this sounds like wireless ... is the laptop a wireless client ? Is anything else going faster on that connection with F11? like wget(http) from somewhere...
[gideon@gidux-laptop-rawhide ~]$ diff /mnt/fedora-10/etc/ssh/ssh_config /etc/ssh/ssh_config 1c1 < # $OpenBSD: ssh_config,v 1.23 2007/06/08 04:40:40 pvalchev Exp $ --- > # $OpenBSD: ssh_config,v 1.25 2009/02/17 01:28:32 djm Exp $ 40c40 < # Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc --- > # Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc 45a46 > # VisualHostKey no
Looks like there is a difference. I have made no changes to the configuration of either, so these should be the defaults. As for the network, I'm using the same wireless connection under F10 and F11. I'll check a local wget to be sure, though. I might end up reassigning this to the kernel (I'm using the in-kernel b43 driver for my wireless).
Well, this sucks. I thought I had done other transfers without a noticeable difference, so I only blamed scp. I just did a wget from my F8 box's HTTP server under F10 and F11. F10 is 2+MB/s, whereas F11 averaged around 600-800KB/s. Is there anything else in the system that would have an effect on scp and wget performance, other than the wireless driver? Is there any other component I should switch this to other than the kernel? Also, I'll confirm whether a hardwired connection is affected by this, to make sure whether it's my wireless or not.
Okay, here are the quick results from my wired test: wget from local HTTP server F10 & F11 - 2.5-2.8MB/s, peaking at 3.0MB/s scp from local SSH server F10 - 2.7-2.8MB/s, peaking at 2.9MB/s F11 - 2.6-2.7MB/s, peaking at 2.8MB/s So, scp is still just a smidgen slower, but not noticeably. Do you think that could be due to the difference in the cipher lines? I'd rather leave it default with such a small (0.1MB/s) difference, especially if there are other benefits (you must have had a good reason for changing it, right?). Anyway, shall I reassign to kernel as a b43 wireless bug, or is there something else I should point the finger at? I'm sorry for pointing it at rsync and scp!
Reassigning to kernel as this is obviously caused by the wireless driver. Please leave the ciphers list as the default. The very small 0.1MB/s slowdown might even be caused by other means than the ciphers used. (If you really want faster, but slightly less secure ciphers you can put blowfish or arcfour first.)
Also of note: every once in a while, the NetworkManager applet won't connect to my wireless network with the first try. It will be connecting for a while and may eventually ask me to confirm the WEP key (which is correct, because I just have to click OK to connect almost instantly). This never happened in F10.
Any initial thoughts? What kind of information would help in identifying the performance regression from 2.6.27 to 2.6.29?
Ping.
Can you post the output of the 'lspci -nn' command so we can see which wireless card you have? Also, if you still have the F10 install can you try the 2.6.29.4-75 kernel from the updates-testing repository?
Created attachment 346955 [details] Output of lspci -nn Here is the output of lspci -nn. I am about to install the testing kernel and reboot. I'll let you know how it goes!
2.6.29 in F10 exhibits exactly the same behavior as 2.6.29 in F11, so I guess we're definitely looking at a regression between 2.6.27 and 2.6.29. Is there any other information I can provide to help track this down? Thank you so much for looking at this.
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle. Changing version to '11'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
The biggest "likely suspect" is the change from the PID rate control algorithm in 2.6.27 to the Minstrel algorithm used in 2.6.29. I am building test F11 kernels w/ PID re-enabled (and made the default) here: http://koji.fedoraproject.org/koji/taskinfo?taskID=1401682 Please give them a try when the builds are completed and re-run your performances tests (e.g. from comment 14). Do they perform differently with those kernels?
Build is complete -- get it while it's hot!
Download seems to be slightly improved, coming in at 1.0-1.3MB/s using HTTP or SSH, but upload is still a very pitiful 100KB/s. I guess that's not it (at least not entirely).
Ping. Anything else I can do to help this get moving?
Could you try setting a fixed bitrate? iwconfig wlan0 rate 54M fixed You might try some other rates as well. Standard 802.11g supports rates of 6, 9, 12, 18, 24, 36, 48, 54 in addition to the 802.11b rates of 1, 2, 5.5 and 11Mbps. Do you get consistently better throughput with any of those?
Downloading a file over HTTP after fixing to the following speeds: 54M: ~150KB/s 48M: ~150KB/s 36M: ~150KB/s 24M: ~150KB/s 18M: ~150KB/s 12M: ~150KB/s 11M: ~2MB/s 9M: ~150KB/s 6M: ~150KB/s 5.5M: ~2MB/s 2M: ~1.5MB/s 1M: ~1MB/s So basically, what you've revealed by having me do this, is that things only work well if they're running in 802.11b modes (11M, 5.5M, 2M, and 1M). What's really odd to me is that none of those modes should be fast enough to accommodate 2MB/s, as that would require a 16Mbps or better connection. Does this give you some good information to go on? And thanks! /me runs 'iwconfig wlan0 rate 11M fixed' and giggles.
Well, you've fixed your _upload_ bitrate, and your http _download_ should not be affected by that all that much -- except for the TCP ack packets etc. of course... It's odd that it gives you such a huge difference, there must be something seriously wrong with the ofdm bitrates.
Another thing I noticed, though it might've just been me, was that the HTTP connection seemed to be made a lot quicker at 11M, and web pages seem to load snappier. Again, it might just be me.
Well that too makes sense -- you need to do DNS queries etc. which is most of the time that you spend waiting for web pages since the actual download speed does't usually matter _that_ much (unless you're using a modem from the last millennium)...
I just wanted to post a quick update. I wasn't running a fully updated F11 install until last night (I'm a procrastinator). The situation is the same. Also, I no longer have an F10 install to test with (and I'm guessing that's not useful at this point). I will be installing F12 Alpha on this laptop (alongside F11) once that's out, so I can see if there's been an improvement with the newer kernel. Anything else I can do in the meantime as far as giving you information to work with?
I don't know what you did, but it's working great now! I believe it was as of the most recent kernel update in Fedora 11. My download and upload are both now 2.5-2.9MB/s, without running any iwconfig commands first. Thanks!
Holy crap, I'm not too bright at the moment... I had my laptop plugged into a wired network connection, because I was doing a network install of F12 Alpha, hence the increased performance. I will try again with my wireless connection under both F11 and F12 Alpha in a short bit and update once more. Sorry!
Yep, still very slow under F12 Alpha. Any ideas?
Not sure if it is the same problem here. The used working wireless in F11 is very slow in F12/rawhide at the moment. The ping does not show anything obvious, but it takes a long time seems forever even to connect to local wireless router's web interface via firefox. 02:02.0 Network controller: Broadcom Corporation BCM4318 [AirForce One 54g] 802.11g Wireless LAN Controller (rev 02) Tried both 2.6.29 and 2.6.31 kernels, and 4.174.64.19 and 4.150.10.5 firmware without luck. All firewalls are disabled.
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle. Changing version to '12'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
http://marc.info/?l=linux-wireless&m=126488652005514&w=2
http://koji.fedoraproject.org/koji/taskinfo?taskID=1959523 Give those kernels a try when the build is complete? Do they improve b43 throughput for you?
Not sure why that one failed, but I think Koji burped. Hopefully this one will succeed... http://koji.fedoraproject.org/koji/taskinfo?taskID=1959563
While I don't currently have a local HTTP server set up to test local file transfers, I have done a bit of web surfing with the new kernel. Before, surfing was a little sluggish, with pages taking a while to load. Things seem much snappier now. Of course, that is just speculation, and I can't give you any solid numbers at the moment, but it really does feel like a (big) improvement.
The patch in question is part of 2.6.32.9. I believe that current Koji F-12 kernels will resolve this issue: http://koji.fedoraproject.org/koji/buildinfo?buildID=158902