Red Hat Bugzilla – Bug 502026
dd allocates bs memory chunk even with count=0
Last modified: 2013-04-15 08:44:35 EDT
Description of problem:
I've noticed that this command creates error message:
LANG=en_US dd if=/dev/zero of=/tmp/x bs=10G seek=1 count=0
dd: memory exhausted
Obviously I could use bs=1G seek=10 and I'll achieve error-free result on my 4GB machine - but it is still imho a bug to trying to mmap 10GB bs when count=0 - because no memory chunk will be actually used - I assume it's cheap to add extra test for this case into dd.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
Thanks for report, apparently check for count=0 is done after xmalloc() call. It's easy to fix cosmetic thing, I'll propose patch to upstream and we'll see :) ...
(In reply to comment #0)
> Obviously I could use bs=1G seek=10 and I'll achieve error-free result on my
> 4GB machine - but it is still imho a bug to trying to mmap 10GB bs when count=0
Anyway both variants sound like a big waste to me. With the current version of coreutils I suggest using 'bs=1 seek=10G' to create 10 GiB sparse file more efficiently.
I think we could agree that bs=1 seek=10G & bs=10G seek=1 are equivalent when count==0 as both should not allocate a single byte in memory ;)
I see your point. It was only my suggestion for using the current coreutils. It will take some before the build will be ready (if ever).
Ha I noticed that issue when doing my truncate script 4 years ago:
Then I promptly forgot about it.
Note coreutils now has the truncate utility that has none of these tricky issues.
Patch looks good BTW.
p.s. if=/dev/zero is redundant in the original command
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.
More information and reason for this action is here:
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '11'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 11's end of life.
Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 11 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
The process we are following is described here:
Well still interesting how long it takes to fix cosmetic bug :)
There is no reason to allocate any memory if the count is 0 no matter how big the block size was set - that's the whole point of this bugzilla. It's wasting system resources.
Also I've not really checked the code recently - but allocating any memory seems to be not always good idea in the first place - there should be used slicing for copy to avoid pointless memory transfers to userspace between 2 descriptors if the kernel could handle this all itself (of course no filters, pages must be properly aligned....) but obviously this is not a cosmetic fix.
opps - slice -> splice/sendfile
Zdenek - it turned out that it is not cosmetic bug ... neither only cosmetic fix ... it looked so at first sight but it was not - this is quite common in such cases...
It might be required to allocate the memory for seek even if the count is 0 - if you have not-seekable device...
See http://lists.gnu.org/archive/html/bug-coreutils/2009-05/msg00215.html for related discussion on upstream mailing list. I had a patch which covers the possible situations in dd without regressions - to allow your seek=1 count=0 without memory allocation - but I haven't sent it to the list as it was more obfuscating the code for very little benefit. After that it got completely off my radar to the bottom of my todo list. It might be possible to achieve the result without regressions and in upstream acceptable way, but tbh very low priority for me...
Therefore I moved it to Rawhide to have it still tracked ...
Well it's been obvious it will not be 'a one line fix' even in the simple case - but it didn't looked that hard to implement delayed allocation when the memory is in fact really needed.
What should give real speed boost is the usage of 'new' kernel syscalls.
That would not only safe memory but also lower CPU usage considerably.
Obviously wrong design of 'dd' will require some thinking how to use them...
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.
More information and reason for this action is here:
A patch to resolve this issue has now been accepted upstream: