Bug 449263 - dd program apparently failing with undocumented 32 bit limits on x86_64 OS
Summary: dd program apparently failing with undocumented 32 bit limits on x86_64 OS
Alias: None
Product: Fedora
Classification: Fedora
Component: coreutils
Version: 9
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: Ondrej Vasik
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2008-06-01 09:08 UTC by c.h.
Modified: 2008-07-30 20:12 UTC (History)
2 users (show)

Fixed In Version: 6.10-28.fc9
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2008-07-30 20:12:30 UTC

Attachments (Terms of Use)

Description c.h. 2008-06-01 09:08:56 UTC
Description of problem:

System: Fedora 9, X86_64, 8GB of mostly unused RAM, userid = root.

dd truncates certain copy operations at < 2GB sizes for no apparent reason.

info dd, man dd do not suggest there should be 32 bit size limits intrinsic
to any of the given parameters / functions.  

IDK what the POSIX / OSF specifications may say, but I'd be shocked if dd
shouldn't be expected to work to full 64 bit+ capabilities given the nature of
the utility.

Version-Release number of selected component (if applicable):

# rpm -qf /bin/dd

How reproducible:

Steps to Reproduce:
1.  # dd if=FC8_X64.ddread.iso of=iso.iso bs=3876407296 count=1
2.  0+1 records in
    0+1 records out
    2147479552 bytes (2.1 GB) copied, 44.587 s, 48.2 MB/s

N.B. 2147479552 = 0x7FFFF000 or 4KB less than 2^32, as if maybe it tried to
allocate 2^32 and got 4KB less than that (malloc heap header?) as a buffer
or something.  I haven't looked at the source, but thought I ought to document
the conditions promptly since it may be a while before I can investigate any

3. # ls -l iso.iso
-rw-r--r-- 1 root root 2147479552 2008-06-01 01:35 iso.iso

# ulimit 

# vmstat -S M
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0   1312    992   4641    0    0   384   425   72  262  2  1 91  6  0

Actual results:

See above.

Expected results:

Copying the full size of requested data, not stopping at 2^32 - 4KB bytes.

Since I'm root it shouldn't have been a memory rlimit issue (see above ulimit).

Since there's free disk space, there should be no reason to truncate the output

Since there's 8GB of RAM with not much used it shouldn't have been a RAM limit.

Additional info:

Comment 1 c.h. 2008-06-01 09:12:27 UTC
Correction: Ok it's late; 0x7FFFF000 isn't 2^32 - 4KB, it's 2^31 - 4KB, so
something like a signed 32 bit int's maximum positive value - 4KB is the
observed limit.

Comment 2 Ondrej Vasik 2008-07-07 12:17:53 UTC
Thanks for report. Just want to say a bit about the issue. I investigated the
issue meanwhile a bit although I haven't time to find the complete
solution(other issues in todo list). It is not exactly inability of dd and it's
handling of large portion of data, it is more issue of glibc read() function.
Although 64 bit dd supports/requests larger blocksize, read() returns only
limited portion of it (the fix is obvious - dd should call another read() until
EOF or blocksize reached). Manual pages of read() say that the returned number
of bytes could be less than requested number of bytes, so the behaviour is
correct and dd should handle it better way. Although it is more issue of glibc
read(), it is documented and should be workarounded.

Comment 3 Kamil Dudka 2008-07-18 07:11:29 UTC
As solution to this problem (and similar #431997) I proposed a patch to 
upstream developers. You can watch discussion in the thread http://

Comment 4 Fedora Update System 2008-07-28 09:45:44 UTC
coreutils-6.10-28.fc9 has been submitted as an update for Fedora 9

Comment 5 Kamil Dudka 2008-07-28 10:39:22 UTC
I thing this is not a bug. "0+1 records in" means 0 full records (full = 
3876407296B in this case), 1 partial records (partial = less than 3876407296B 
in this case).

This is documented dd behavior (standardized by POSIX) and can not be changed. 
To work around this, you can use new dd option iflag=fullblock, which turns on 
reading full blocks where possible.

The option iflag=fullblock is available since coreutils-6.10-28.fc9.

Comment 6 Fedora Update System 2008-07-30 20:12:28 UTC
coreutils-6.10-28.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.