Bug 1276664 - RFE: implement --ignore-missing for sha*sum -c
RFE: implement --ignore-missing for sha*sum -c
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: coreutils (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Ondrej Vasik
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-30 08:32 EDT by Kamil Páral
Modified: 2015-11-23 08:23 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-30 11:14:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kamil Páral 2015-10-30 08:32:24 EDT
Description of problem:
My usual use case is that I have several of the files mentioned in one or many CHECKSUM files, but not all of them. That is not a problem for me, I don't need all of them, but I want to verify that all present files are correct. Currently that's hard to do both manually and automatically. The output looks like this:

$ sha256sum -c *SUM
sha256sum: Fedora-Live-Cinnamon-x86_64-23-10.iso: No such file or directory
Fedora-Live-Cinnamon-x86_64-23-10.iso: FAILED open or read
Fedora-Live-KDE-x86_64-23-10.iso: OK
sha256sum: Fedora-Live-LXDE-x86_64-23-10.iso: No such file or directory
Fedora-Live-LXDE-x86_64-23-10.iso: FAILED open or read
sha256sum: Fedora-Live-MATE_Compiz-x86_64-23-10.iso: No such file or directory
Fedora-Live-MATE_Compiz-x86_64-23-10.iso: FAILED open or read
sha256sum: Fedora-Live-SoaS-x86_64-23-10.iso: No such file or directory
Fedora-Live-SoaS-x86_64-23-10.iso: FAILED open or read
sha256sum: Fedora-Live-Xfce-x86_64-23-10.iso: No such file or directory
Fedora-Live-Xfce-x86_64-23-10.iso: FAILED open or read
sha256sum: WARNING: 5 listed files could not be read
Fedora-Server-DVD-x86_64-23.iso: OK
Fedora-Server-netinst-x86_64-23.iso: OK
Fedora-Workstation-netinst-x86_64-23.iso: OK
Fedora-Live-Workstation-x86_64-23-10.iso: OK

It's very hard to see in a quick glance whether some of the files are corrupted, because the output is full of errors about missing files. Yes, I could use grep to filter them out, but that's equally inconvenient as examining the output carefully.

In term of automation, the exit status is non-zero when some files are missing. I haven't found an easy way to ignore missing files and check the rest. So even automating this is hard.

Please implement something like --ignore-missing option, that could be used together with --check. It would not print any warnings about missing files, and they would not affect the exit status of the process.

Thank you.

Version-Release number of selected component (if applicable):
coreutils-8.24-4.fc23.x86_64
Comment 1 Kamil Dudka 2015-10-30 09:02:18 EDT
A similar request (including a patch) was posted to the upstream mailing-list in 2010:

https://lists.gnu.org/archive/html/coreutils/2010-12/msg00032.html

... and this was the solution suggested by upstream:

https://lists.gnu.org/archive/html/coreutils/2010-12/msg00036.html
Comment 2 Kamil Páral 2015-10-30 09:26:46 EDT
Ugh. That makes me very sad.

Of course that does not solve neither extraneous output (missing is not FAILED):

$ sha256sum -c *SUM 2>/dev/null
Fedora-Live-Cinnamon-x86_64-23-10.iso: FAILED open or read
Fedora-Live-KDE-x86_64-23-10.iso: OK
Fedora-Live-LXDE-x86_64-23-10.iso: FAILED open or read
Fedora-Live-MATE_Compiz-x86_64-23-10.iso: FAILED open or read
Fedora-Live-SoaS-x86_64-23-10.iso: FAILED open or read
Fedora-Live-Xfce-x86_64-23-10.iso: FAILED open or read
Fedora-Server-DVD-x86_64-23.iso: OK
Fedora-Server-netinst-x86_64-23.iso: OK
Fedora-Workstation-netinst-x86_64-23.iso: OK
Fedora-Live-Workstation-x86_64-23-10.iso: OK

nor the exit code.

Is there any chance some developer watching RH Bugzilla could pick this up, or is the only reasonable approach here to contact coreutils upstream mailing list?
Comment 3 Kamil Dudka 2015-10-30 10:04:20 EDT
(In reply to Kamil Páral from comment #2)
> nor the exit code.

You can get the exit code from grep, too, if this is intended for scripting.

> Is there any chance some developer watching RH Bugzilla could pick this up,
> or is the only reasonable approach here to contact coreutils upstream
> mailing list?

Upstream ML is definitely a better place for requesting new features unless they are related to downstream patches (which this one is certainly not).
Comment 4 Kamil Páral 2015-10-30 11:14:23 EDT
Thanks for info. I wanted to avoid mailing lists, that's why I put it here as my first option. Closing, will consider reporting upstream.
Comment 5 Pádraig Brady 2015-11-01 11:46:25 EST
One reason we _might_ consider this is because currently open() and read() errors are not distinguished. You would want to treat a read() error like a checksum error, and not like an open()==ENOENT. Also other errors from open()
should not be treated like ENOENT.

Another reason is because grep -E '(OK|FAILED)$' is locale dependent,
so one would need LC_ALL=C md5sum ... | grep ...

I was 50:50 before, but with the above arguments I'm 60:40 for adding the option.

I'll propose upstream.
Comment 6 Kamil Páral 2015-11-03 06:58:51 EST
Thanks Pádraig. While working around the problem with grep is possible, it is a) complicated, especially when considering all the corner cases b) not reliable, unless you know exactly what strings *sum tools generate to the output (i.e. you would need to study the source code). Moreover stdout/stderr is not like an API which is guaranteed not to change, your workaround script can break any time.

Grep is useful for many things, but modifying program behavior considerably based on stdout/stderr and adjusting exit codes is, I believe, not one of them. Having a specific option to distinguish missing files from unreadable files and invalid checksum files would make everything much easier here.
Comment 7 Pádraig Brady 2015-11-23 08:23:19 EST
Upstream commit: http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=9fd0662f

Note You need to log in before you can comment on or make changes to this bug.