Bug 449263

Summary: dd program apparently failing with undocumented 32 bit limits on x86_64 OS
Product: [Fedora] Fedora Reporter: c.h. <fc6_req>
Component: coreutilsAssignee: Ondrej Vasik <ovasik>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 9CC: kdudka, twaugh
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: 6.10-28.fc9 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-07-30 16:12:30 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description c.h. 2008-06-01 05:08:56 EDT
Description of problem:

System: Fedora 9, X86_64, 8GB of mostly unused RAM, userid = root.

dd truncates certain copy operations at < 2GB sizes for no apparent reason.

info dd, man dd do not suggest there should be 32 bit size limits intrinsic
to any of the given parameters / functions.  

IDK what the POSIX / OSF specifications may say, but I'd be shocked if dd
shouldn't be expected to work to full 64 bit+ capabilities given the nature of
the utility.

Version-Release number of selected component (if applicable):

# rpm -qf /bin/dd
coreutils-6.10-23.fc9.x86_64

How reproducible:


Steps to Reproduce:
1.  # dd if=FC8_X64.ddread.iso of=iso.iso bs=3876407296 count=1
2.  0+1 records in
    0+1 records out
    2147479552 bytes (2.1 GB) copied, 44.587 s, 48.2 MB/s

N.B. 2147479552 = 0x7FFFF000 or 4KB less than 2^32, as if maybe it tried to
allocate 2^32 and got 4KB less than that (malloc heap header?) as a buffer
or something.  I haven't looked at the source, but thought I ought to document
the conditions promptly since it may be a while before I can investigate any
furtner.


3. # ls -l iso.iso
-rw-r--r-- 1 root root 2147479552 2008-06-01 01:35 iso.iso

# ulimit 
unlimited

# vmstat -S M
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0   1312    992   4641    0    0   384   425   72  262  2  1 91  6  0

  
Actual results:

See above.

Expected results:

Copying the full size of requested data, not stopping at 2^32 - 4KB bytes.

Since I'm root it shouldn't have been a memory rlimit issue (see above ulimit).

Since there's free disk space, there should be no reason to truncate the output
file.

Since there's 8GB of RAM with not much used it shouldn't have been a RAM limit.

Additional info:
Comment 1 c.h. 2008-06-01 05:12:27 EDT
Correction: Ok it's late; 0x7FFFF000 isn't 2^32 - 4KB, it's 2^31 - 4KB, so
something like a signed 32 bit int's maximum positive value - 4KB is the
observed limit.
Comment 2 Ondrej Vasik 2008-07-07 08:17:53 EDT
Thanks for report. Just want to say a bit about the issue. I investigated the
issue meanwhile a bit although I haven't time to find the complete
solution(other issues in todo list). It is not exactly inability of dd and it's
handling of large portion of data, it is more issue of glibc read() function.
Although 64 bit dd supports/requests larger blocksize, read() returns only
limited portion of it (the fix is obvious - dd should call another read() until
EOF or blocksize reached). Manual pages of read() say that the returned number
of bytes could be less than requested number of bytes, so the behaviour is
correct and dd should handle it better way. Although it is more issue of glibc
read(), it is documented and should be workarounded.
Comment 3 Kamil Dudka 2008-07-18 03:11:29 EDT
As solution to this problem (and similar #431997) I proposed a patch to 
upstream developers. You can watch discussion in the thread http://
lists.gnu.org/archive/html/bug-coreutils/2008-07/msg00118.html
Comment 4 Fedora Update System 2008-07-28 05:45:44 EDT
coreutils-6.10-28.fc9 has been submitted as an update for Fedora 9
Comment 5 Kamil Dudka 2008-07-28 06:39:22 EDT
I thing this is not a bug. "0+1 records in" means 0 full records (full = 
3876407296B in this case), 1 partial records (partial = less than 3876407296B 
in this case).

This is documented dd behavior (standardized by POSIX) and can not be changed. 
To work around this, you can use new dd option iflag=fullblock, which turns on 
reading full blocks where possible.

The option iflag=fullblock is available since coreutils-6.10-28.fc9.
Comment 6 Fedora Update System 2008-07-30 16:12:28 EDT
coreutils-6.10-28.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.