Bug 449263
| Summary: | dd program apparently failing with undocumented 32 bit limits on x86_64 OS | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | c.h. <fc6_req> |
| Component: | coreutils | Assignee: | Ondrej Vasik <ovasik> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | low | ||
| Version: | 9 | CC: | kdudka, twaugh |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | 6.10-28.fc9 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2008-07-30 20:12:30 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Correction: Ok it's late; 0x7FFFF000 isn't 2^32 - 4KB, it's 2^31 - 4KB, so something like a signed 32 bit int's maximum positive value - 4KB is the observed limit. Thanks for report. Just want to say a bit about the issue. I investigated the issue meanwhile a bit although I haven't time to find the complete solution(other issues in todo list). It is not exactly inability of dd and it's handling of large portion of data, it is more issue of glibc read() function. Although 64 bit dd supports/requests larger blocksize, read() returns only limited portion of it (the fix is obvious - dd should call another read() until EOF or blocksize reached). Manual pages of read() say that the returned number of bytes could be less than requested number of bytes, so the behaviour is correct and dd should handle it better way. Although it is more issue of glibc read(), it is documented and should be workarounded. As solution to this problem (and similar #431997) I proposed a patch to upstream developers. You can watch discussion in the thread http:// lists.gnu.org/archive/html/bug-coreutils/2008-07/msg00118.html coreutils-6.10-28.fc9 has been submitted as an update for Fedora 9 I thing this is not a bug. "0+1 records in" means 0 full records (full = 3876407296B in this case), 1 partial records (partial = less than 3876407296B in this case). This is documented dd behavior (standardized by POSIX) and can not be changed. To work around this, you can use new dd option iflag=fullblock, which turns on reading full blocks where possible. The option iflag=fullblock is available since coreutils-6.10-28.fc9. coreutils-6.10-28.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report. |
Description of problem: System: Fedora 9, X86_64, 8GB of mostly unused RAM, userid = root. dd truncates certain copy operations at < 2GB sizes for no apparent reason. info dd, man dd do not suggest there should be 32 bit size limits intrinsic to any of the given parameters / functions. IDK what the POSIX / OSF specifications may say, but I'd be shocked if dd shouldn't be expected to work to full 64 bit+ capabilities given the nature of the utility. Version-Release number of selected component (if applicable): # rpm -qf /bin/dd coreutils-6.10-23.fc9.x86_64 How reproducible: Steps to Reproduce: 1. # dd if=FC8_X64.ddread.iso of=iso.iso bs=3876407296 count=1 2. 0+1 records in 0+1 records out 2147479552 bytes (2.1 GB) copied, 44.587 s, 48.2 MB/s N.B. 2147479552 = 0x7FFFF000 or 4KB less than 2^32, as if maybe it tried to allocate 2^32 and got 4KB less than that (malloc heap header?) as a buffer or something. I haven't looked at the source, but thought I ought to document the conditions promptly since it may be a while before I can investigate any furtner. 3. # ls -l iso.iso -rw-r--r-- 1 root root 2147479552 2008-06-01 01:35 iso.iso # ulimit unlimited # vmstat -S M procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 1312 992 4641 0 0 384 425 72 262 2 1 91 6 0 Actual results: See above. Expected results: Copying the full size of requested data, not stopping at 2^32 - 4KB bytes. Since I'm root it shouldn't have been a memory rlimit issue (see above ulimit). Since there's free disk space, there should be no reason to truncate the output file. Since there's 8GB of RAM with not much used it shouldn't have been a RAM limit. Additional info: