Bug 502026

Summary: dd allocates bs memory chunk even with count=0
Product: [Fedora] Fedora Reporter: Zdenek Kabelac <zkabelac>
Component: coreutilsAssignee: Ondrej Oprala <ooprala>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: low    
Version: rawhideCC: kdudka, ooprala, ovasik, p, twaugh
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-15 08:44:35 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Zdenek Kabelac 2009-05-21 10:53:51 EDT
Description of problem:

I've noticed that this command creates error message:

LANG=en_US dd if=/dev/zero of=/tmp/x bs=10G seek=1 count=0
dd: memory exhausted

Obviously I could use  bs=1G seek=10 and I'll achieve error-free result on my 4GB machine - but it is still imho a bug to trying to mmap 10GB bs when count=0 - because no memory chunk will be actually used - I assume it's cheap to add extra test for this case into dd.


Version-Release number of selected component (if applicable):
coreutils-7.2-1.fc11.x86_64

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Ondrej Vasik 2009-05-21 11:20:27 EDT
Thanks for report, apparently check for count=0 is done after xmalloc() call. It's easy to fix cosmetic thing, I'll propose patch to upstream and we'll see :) ...
Comment 2 Kamil Dudka 2009-05-21 11:31:49 EDT
(In reply to comment #0)
> Obviously I could use  bs=1G seek=10 and I'll achieve error-free result on my
> 4GB machine - but it is still imho a bug to trying to mmap 10GB bs when count=0

Anyway both variants sound like a big waste to me. With the current version of coreutils I suggest using 'bs=1 seek=10G' to create 10 GiB sparse file more efficiently.
Comment 3 Zdenek Kabelac 2009-05-21 11:46:01 EDT
I think we could agree that bs=1 seek=10G  &  bs=10G  seek=1 are equivalent when  count==0   as both should not allocate a single byte in memory ;)
Comment 4 Kamil Dudka 2009-05-21 11:57:09 EDT
I see your point. It was only my suggestion for using the current coreutils. It will take some before the build will be ready (if ever).
Comment 5 Pádraig Brady 2009-05-22 04:48:34 EDT
Ha I noticed that issue when doing my truncate script 4 years ago:
http://www.pixelbeat.org/scripts/truncate
Then I promptly forgot about it.

Note coreutils now has the truncate utility that has none of these tricky issues.

Patch looks good BTW.

p.s. if=/dev/zero is redundant in the original command
Comment 6 Bug Zapper 2009-06-09 12:15:56 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 7 Bug Zapper 2010-04-27 10:26:29 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 9 Zdenek Kabelac 2010-06-15 05:30:59 EDT
Well still interesting how long it takes to fix cosmetic bug :) 

There is no reason to allocate any memory if the count is 0 no matter how big the block size was set - that's the whole point of this bugzilla.  It's wasting system resources.  

Also I've not really checked the code recently - but allocating any memory seems to be not always good idea in the first place - there should be used slicing for copy to avoid pointless memory transfers to userspace between 2 descriptors if the kernel could handle this all itself (of course no filters, pages must be properly aligned....)  but obviously this is not a cosmetic fix.
Comment 10 Zdenek Kabelac 2010-06-15 05:33:39 EDT
opps - slice -> splice/sendfile
Comment 11 Ondrej Vasik 2010-06-15 06:14:45 EDT
Zdenek - it turned out that it is not cosmetic bug ... neither only cosmetic fix ... it looked so at first sight but it was not - this is quite common in such cases... 

It might be required to allocate the memory for seek even if the count is 0 - if you have not-seekable device...

See http://lists.gnu.org/archive/html/bug-coreutils/2009-05/msg00215.html for related discussion on upstream mailing list. I had a patch which covers the possible situations in dd without regressions - to allow your seek=1 count=0 without memory allocation - but I haven't sent it to the list as it was more obfuscating the code for very little benefit. After that it got completely off my radar to the bottom of my todo list. It might be possible to achieve the result without regressions and in upstream acceptable way, but tbh very low priority for me...
Therefore I moved it to Rawhide to have it still tracked ...
Comment 12 Zdenek Kabelac 2010-06-15 09:13:45 EDT
Well it's been obvious it will not be 'a one line fix' even in the simple case - but it didn't looked that hard to implement delayed allocation when the memory is in fact really needed.

What should give real speed boost is the usage of 'new' kernel syscalls.
That would not only safe memory but also lower CPU usage considerably.
Obviously wrong design of 'dd' will require some thinking how to use them...
Comment 13 Bug Zapper 2010-07-30 06:40:05 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 14 Ondrej Oprala 2013-04-15 08:44:35 EDT
A patch to resolve this issue has now been accepted upstream:
http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=8901e010fa24ba375d53e3e37fc7819621adcf19