Bug 562662

Summary:	Very slow file copying to a usb flash drive
Product:	[Fedora] Fedora	Reporter:	Hedayat Vatankhah <hedayatv>
Component:	kernel	Assignee:	Kernel Maintainer List <kernel-maint>
Status:	CLOSED WONTFIX	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	low
Version:	17	CC:	adundovi, anton, avm-xandry, dougsland, gansalmon, itamar, jonathan, kernel-maint, marco
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-08-01 18:27:30 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hedayat Vatankhah 2010-02-07 21:54:50 UTC

Description of problem:
I was trying to copy a very large file (3.3 Gb iso of Fedora x86_64) to a flash drive, and after copying a few hundred megabytes (e.g. 300MB) the copying speed slowed down to around 500KB/s and it was going even lower gradually. It was really unacceptable (the remaining time was climbing up to over 1 hour and 30 minutes using nautilus to copy the file). I've tried different flash drives and different USB ports of my laptop, and there was no significant difference. I encountered this slow copying in past, but usually I've ignored it as the files was not that big. But this time it was really disappointing and I've decided to see why it is so slow.

iotop showed slow disk writes (it was often 0, jumping to some low values (usually less than 500KB/s) and lowering to 0 again) and lots of IO waiting time for pdflush. After searching a little in the Internet, I found that playing with dirty_ratio and dirty_background_ratio kernel parameters in /proc/sys/vm/ might help.

First, I lowered the dirty_background_ratio to 1 (default value is 10), and the disk write speed jumped to around 2.5 MB/s for awhile (around 1 or a few minutes) and then the speed dropped again to around 0.

Finally, I lowered the dirty_ratio parameter to 10 (default value is 20), and it resulted in a constant 2.5MB/s to 3MB/s disk read (from hard disk) and write (to usb disk) speed to the end of the copy operation (which took a few minutes rather than more than an hour!).

The problem is so weird and unacceptable.

As it might help, I have 1.5GBs of RAM and at the time of copy around 44% of it was used by running programs.

Version-Release number of selected component (if applicable):
kernel-2.6.31.12-174.2.3.fc12.x86_64
but the problem was observed in previous F12 kernels too.

How reproducible:
100% in my few tests.

Steps to Reproduce:
1. Start copying a large file (e.g. over 2GB) to a regular USB flash drive. Use a single big file rather than many small files.
2. Observe the copying speed

Actual results:
Very slow copying operation (around 450KB/s in my case)

Expected results:
Regular copy speed (2.5MB/s seems to be possible with my hardware and the mentioned settings)

Additional info:
I don't know if it helps but: the source file was on an NTFS mounted partition, and both of the partitions were mounted by Gnome.
The problem might be related to the mount options used by gnome, but doesn't seem to be related to nautilus. Since I get slow speeds even using dd to copy the file.

Comment 1 Bug Zapper 2010-11-03 22:47:06 UTC

This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 2 Bug Zapper 2010-12-03 23:10:16 UTC

Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 3 Andrej 2011-03-22 06:50:05 UTC

This bug is still present (in version F14):
2.6.35.6-48.fc14.x86_64

Copying big files is at beginning fast but gradually becoming slower and then stops at the end of the file, after a few moments (minutes or so) it continue and finally finish.

What should I do to gather more details?

Comment 4 Hedayat Vatankhah 2011-12-17 13:01:32 UTC

This problem still persists in Fedora 16. Again, lowering dirty_ratio and dirty_background_ratio to 2 and 1 respectively (instead of 20 and 10) resulted in constant 4.5 MB/s speed while copying with the default settings the speed was going down... (I stopped it when it was around 1.5MB/s).

Comment 5 Dave Jones 2011-12-19 18:07:55 UTC

This is probably going to get fixed for real in 3.3, but there's a hack that might make things at least slightly better until then.  I'll throw into the next 16 build.

Comment 6 Dave Jones 2011-12-19 18:14:58 UTC

oh, actually we have that hack in f16 since 3.1.2-0.rc1.1

Comment 7 Hedayat Vatankhah 2011-12-19 18:27:08 UTC

IIRC, I experienced the problem on kernel-3.1.2-1.fc16.x86_64 :(
I hope that it'll be at least really fixed in 3.3.

Comment 8 Josh Boyer 2012-02-28 18:25:10 UTC

There were further fixes for this issue in 3.2.  Is this problem still there on 3.2.7 or newer?

If anyone is willing to test 3.3-rc5, this build should also contain the fixes Dave mentioned:

http://koji.fedoraproject.org/koji/buildinfo?buildID=301620

Comment 9 Dave Jones 2012-03-22 16:37:04 UTC

[mass update]
kernel-3.3.0-4.fc16 has been pushed to the Fedora 16 stable repository.
Please retest with this update.

Comment 10 Dave Jones 2012-03-22 16:42:10 UTC

[mass update]
kernel-3.3.0-4.fc16 has been pushed to the Fedora 16 stable repository.
Please retest with this update.

Comment 11 Dave Jones 2012-03-22 16:50:25 UTC

[mass update]
kernel-3.3.0-4.fc16 has been pushed to the Fedora 16 stable repository.
Please retest with this update.

Comment 12 Hedayat Vatankhah 2012-07-13 09:21:42 UTC

I have experienced very slow copying with 3.4.4-5.fc17.x86_64 when the number of files/the volume of the files to copy become large. Unfortunately, it didn't improve with the trick I mentioned in the report.

Comment 13 Dave Jones 2012-10-23 15:24:36 UTC

# Mass update to all open bugs.

Kernel 3.6.2-1.fc16 has just been pushed to updates.
This update is a significant rebase from the previous version.

Please retest with this kernel, and let us know if your problem has been fixed.

In the event that you have upgraded to a newer release and the bug you reported
is still present, please change the version field to the newest release you have
encountered the issue with.  Before doing so, please ensure you are testing the
latest kernel update in that release and attach any new and relevant information
you may have gathered.

If you are not the original bug reporter and you still experience this bug,
please file a new report, as it is possible that you may be seeing a
different problem. 
(Please don't clone this bug, a fresh bug referencing this bug in the comment is sufficient).

Comment 14 Hedayat Vatankhah 2012-10-23 20:12:45 UTC

I should try to find a new problematic device and test (Under Fedora 17).

Comment 15 Hedayat Vatankhah 2012-11-26 20:22:08 UTC

Experienced similar issue as comment #12. However I witnessed the following:
1. a normal copy by nautilus was going. The speed was around 2.9M/s and in iotop I saw this: in one update, a write operation around 3M/s happened, in the next 2 updates no read/write operation happened. This pattern was repeatedly happen in iotop. 

2. I did the trick I mentioned above to force Linux to flush its caches. I saw a constant write operation in iotop from 3 to 6 M/s. nautilus stalled for a while until buffers were flushing. Unfortunately, I didn't see the final speed shown by nautilus. But considering iotop results, the speed should be better than normal case (2.9M/s)

3. I did another copy of another directory, but it was even slower than 2.9M/s. I undone my changes to kernel parameters and nautilus suddenly finished the copying (which actually copied into memory). I don't know if the actual write was faster or slower.

Unfortunately, I forgot to test with one big file rather than a number of small files. But considering what happened in number 2 above, I'd assume that Linux still needs to better manage its in-RAM buffer for slow USB devices.

Comment 16 Hedayat Vatankhah 2013-01-16 00:31:44 UTC

Well, I just discovered something about the new behavior. I found that sometimes, when I insert a flash disk, it is registered in a USB 1 bus rather than a USB 2 bus, and this is why it is slow. If I re-insert the disk (even in the same port), it might be recognized as a USB 2 device and so it'll be much faster.

Therefore, I think the original bug is already solved. I just wonder why sometimes the disks are registered as USB 1?!

Thanks

Comment 17 Fedora End Of Life 2013-07-04 06:48:58 UTC

This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 18 Fedora End Of Life 2013-08-01 18:27:37 UTC

Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.