Bug 668247 - dd reads random sizes of data
dd reads random sizes of data
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: coreutils (Show other bugs)
5.5.z
All Linux
urgent Severity high
: rc
: ---
Assigned To: Ondrej Vasik
qe-baseos-daemons
: ZStream
Depends On: 431997
Blocks: 664882 668465
  Show dependency treegraph
 
Reported: 2011-01-09 07:21 EST by Saggi Mizrahi
Modified: 2015-03-02 00:22 EST (History)
15 users (show)

See Also:
Fixed In Version: coreutils-5.97-26.el5
Doc Type: Bug Fix
Doc Text:
Previously, when the dd utility read data from a pipe and received a signal such as SIGPIPE, it stopped reading the current block and started with the new one immediately. This caused random output values when the dd command was used to measure size of an input file. With this update, the new "iflag=fullblock" option is available. When the option is used, the dd utility always continues to read incomplete blocks after receiving a signal.
Story Points: ---
Clone Of: 431997
Environment:
Last Closed: 2011-07-21 06:35:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Saggi Mizrahi 2011-01-09 07:21:00 EST
+++ This bug was initially created as a clone of Bug #431997 +++

Description of problem:
When dd reads from a pipe it produces random results

Version-Release number of selected component (if applicable):
coreutils-6.9-9

How reproducible:
Always

Steps to Reproduce:
1. Find a large file, say 10MB
2. cat largefile | dd ibs=100000 count=2 >output1
3. repeat 2) several times recording size of output1
4. dd if=largefile | dd ibs=100000 count=2 >output2
5. repeat 4) several times recording size of output2
  
Actual results:
The size of output will vary.
2. Various sizes such as 65536,139264,143360.
5. Various sizes such as 6144,7168,6656

Expected results:
size should always be 200000, and never anything else

Additional info:
The apparent block size of a pipe "device" should be transparent.
But in any case, when using ibs we are instructing dd what the input block size
is!  So dd should obey our instructions, and not use other block size.
Interesting to note that a conceptually simple program like dd has been so
hacked around since v5.2 that it is a wonder it still works at all.

--- Additional comment from ovasik@redhat.com on 2008-02-08 18:31:28 EST ---

I think this is similar report to the : 
http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg447197.html

Jim Meyering(upstream) wrote there:
"Thanks for the report, but that behavior is required by POSIX.
dd must handle SIGINT the way you want, but dd may not handle
SIGPIPE that way:

    ASYNCHRONOUS EVENTS

        For SIGINT, the dd utility shall interrupt its current processing,
        write status information to standard error, and exit as though
        terminated by SIGINT. It shall take the standard action for all
        other signals; see the ASYNCHRONOUS EVENTS section in Section 1.4
        (on page 280)."

Recommendation was to not use unnecessary pipes with dd command (both 2 and 4
could be written in one command without pipe), closing NOTABUG (see this mailing
list thread for better explanation).

A bit more explanation is also there:
http://lists.gnu.org/archive/html/bug-coreutils/2008-02/msg00000.html

Feel free to add some comments if you are not satisfied with that explanation.
You could also discuss about that behaviour on upstream mailing list
bug-coreutils@gnu.org .

--- Additional comment from redhat3@zacglen.com on 2008-02-08 18:42:55 EST ---

It should be handled properly because any SIGPIPE is synchronous with actual
read/write and is totally predictable. I have been patching dd for the last 5 or
more years and using stat/S_ISFIFO() to modify dd's behavior.

But I am getting sick of reworking the patch every time the hackers totally
reinvent dd.

It is still a bug because dd used to run properly.


--- Additional comment from dvlasenk@redhat.com on 2008-07-14 08:32:06 EDT ---

Single User Specification v3 says at
http://www.opengroup.org/onlinepubs/009695399/ :

The processing order shall be as follows:

1. An input block is read.
2. If the input block is shorter than the specified input block size and the
sync conversion is specified, null bytes shall be appended to the input data up
to the specified size. (If either block or unblock is also specified, <space>s
shall be appended instead of null bytes.) The remaining conversions and output
shall include the pad characters as if they had been read from the input.
...

paragraph (2) clearly acknowledges the fact that read can be a short one, and it
says nowhere that additional reads should accumulate the full input block. The
fact that such additional reads might block for indeterminate amount of time may
break scripts which do not expect this.

Your assumption stated here: "But in any case, when using ibs we are instructing
dd what the input block size is! So dd should obey our instructions" does not
seem to match what standard says. Standard only says that reads shall be
_attempted_ with that size.

As I see it, changing dd behavior might make you happy, but at the same time
_other_ users will experience breakage ("hung" dd waiting for additional input
which will never come).

> It is still a bug because dd used to run properly.

Which version of dd worked for you?

--- Additional comment from dvlasenk@redhat.com on 2008-07-14 08:35:13 EDT ---

Sorry, wrong link. The correct one is:

http://www.opengroup.org/onlinepubs/009695399/utilities/dd.html

--- Additional comment from redhat3@zacglen.com on 2008-07-14 09:37:31 EDT ---

If dd is waiting for "additional input that will never come" then it is either
at the end of file (which would be treated normally and cause termination) or
there is more data to come (eventually).  There is no other possibility.

In any case I am not concerned with what sync behavior does (pad), I am
concerned when sync is NOT specified.  What does your standard say about that? 

paragraph (2) is clearly talking about a condition where sync is specified -
nothing else.

The version that used to work for me was the version that existed 5 years ago.


--- Additional comment from kdudka@redhat.com on 2008-07-18 03:10:55 EDT ---

As solution to this problem (and similar #449263) I proposed a patch to 
upstream developers. You can watch discussion in the thread http://
lists.gnu.org/archive/html/bug-coreutils/2008-07/msg00118.html

--- Additional comment from kdudka@redhat.com on 2008-07-28 06:52:03 EDT ---

I thing this is not a bug. Option count= specifies sum of full and partial 
records (full = 100000B; partial = less than 100000B in this case).

This is documented dd behavior (standardized by POSIX) and can not be changed.
But there is already solution in Fedora 9 and rawhide - new dd option 
iflag=fullblock, which turns on reading full blocks where possible.

The option iflag=fullblock is available since coreutils-6.10-28.fc9 and 
coreutils-6.12-7.fc10.

--- Additional comment from updates@fedoraproject.org on 2008-08-07 10:09:32 EDT ---

coreutils-6.9-18.fc8 has been submitted as an update for Fedora 8

--- Additional comment from updates@fedoraproject.org on 2008-08-12 14:23:27 EDT ---

coreutils-6.9-18.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

--- Additional comment from ovasik@redhat.com on 2008-08-19 07:54:02 EDT ---

Closing CURRENTRELEASE ... it looks like automatic closing bot is too lazy to do that. New option iflag=fullblock added to address the issue in coreutils-6.9-18.fc8.
Comment 1 Saggi Mizrahi 2011-01-09 07:23:52 EST
This blocks VDSM BZ#664882 because we can't be sure that dd read all the data unless we use very small iops.
Comment 2 Ondrej Vasik 2011-01-09 08:18:20 EST
So ...to clarify this clone purpose. Is this the request for iflag=fullblock feature backport to coreutils-5.97 ?
Comment 3 Ayal Baron 2011-01-09 09:28:00 EST
(In reply to comment #2)
> So ...to clarify this clone purpose. Is this the request for iflag=fullblock
> feature backport to coreutils-5.97 ?

if 5.97 is RHEL 5.6.z then yes, it is.
Comment 4 Ondrej Vasik 2011-01-09 10:52:47 EST
Yes, RHEL-5 contains coreutils-5.97 (coreutils-5.97-23.el5 atm.). Backport should be easy, affected parts were not signficantly changed between coreutils-5.97 and upstream acceptance in coreutils-7.0 .

Patch will be merge of http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=f674a10f7ebc55de6181007a5216a297e3dc3365 and http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=e535754fdbe701e70f3ac9834b643a9272dc1d98 ... possibly with some minor tweaks.
Comment 13 Tomas Capek 2011-07-13 08:09:28 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, when the dd utility read data from a pipe and received a signal such as SIGPIPE, it stopped reading the current block and started with the new one immediately. This caused random output values when the dd command was used to measure size of an input file. With this update, the new "iflag=fullblock" option is available. When the option is used, the dd utility always continues to read incomplete blocks after receiving a signal.
Comment 14 JW 2011-07-13 08:21:03 EDT
This is totally wrong and not the correct fix.  You cannot fix defective bug-ridden behaviors by adding a command-line option.  Nobody is going to figure that to fix erratic and faulty program behavior one must specify some arcane additional command-line option.

When one specifies an input block size then that ibs must be honored regardless of the output destination.

Whether there is a SIGPIPE involved or not, the input block size MUST be adhered to without any additional command-line options.

The input is not exhausted; the output is not in error; therefore the input block size must continue to be properly satisfied.

For goodness sake - dd used to work perfectly until certain people started meddling with it and subsequently breaking it.  If it isn't broken then why have certain people been trying so hard to fix it.  Got back to the last properly working version and there you have your proper repair.
Comment 15 errata-xmlrpc 2011-07-21 06:35:20 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1074.html
Comment 16 errata-xmlrpc 2011-07-21 08:18:52 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1074.html

Note You need to log in before you can comment on or make changes to this bug.