Bug 1276664

Summary: RFE: implement --ignore-missing for sha*sum -c
Product: [Fedora] Fedora Reporter: Kamil Páral <kparal>
Component: coreutilsAssignee: Ondrej Vasik <ovasik>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: admiller, kdudka, kparal, kzak, ooprala, ovasik, pbrady, p, twaugh
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-30 15:14:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kamil Páral 2015-10-30 12:32:24 UTC
Description of problem:
My usual use case is that I have several of the files mentioned in one or many CHECKSUM files, but not all of them. That is not a problem for me, I don't need all of them, but I want to verify that all present files are correct. Currently that's hard to do both manually and automatically. The output looks like this:

$ sha256sum -c *SUM
sha256sum: Fedora-Live-Cinnamon-x86_64-23-10.iso: No such file or directory
Fedora-Live-Cinnamon-x86_64-23-10.iso: FAILED open or read
Fedora-Live-KDE-x86_64-23-10.iso: OK
sha256sum: Fedora-Live-LXDE-x86_64-23-10.iso: No such file or directory
Fedora-Live-LXDE-x86_64-23-10.iso: FAILED open or read
sha256sum: Fedora-Live-MATE_Compiz-x86_64-23-10.iso: No such file or directory
Fedora-Live-MATE_Compiz-x86_64-23-10.iso: FAILED open or read
sha256sum: Fedora-Live-SoaS-x86_64-23-10.iso: No such file or directory
Fedora-Live-SoaS-x86_64-23-10.iso: FAILED open or read
sha256sum: Fedora-Live-Xfce-x86_64-23-10.iso: No such file or directory
Fedora-Live-Xfce-x86_64-23-10.iso: FAILED open or read
sha256sum: WARNING: 5 listed files could not be read
Fedora-Server-DVD-x86_64-23.iso: OK
Fedora-Server-netinst-x86_64-23.iso: OK
Fedora-Workstation-netinst-x86_64-23.iso: OK
Fedora-Live-Workstation-x86_64-23-10.iso: OK

It's very hard to see in a quick glance whether some of the files are corrupted, because the output is full of errors about missing files. Yes, I could use grep to filter them out, but that's equally inconvenient as examining the output carefully.

In term of automation, the exit status is non-zero when some files are missing. I haven't found an easy way to ignore missing files and check the rest. So even automating this is hard.

Please implement something like --ignore-missing option, that could be used together with --check. It would not print any warnings about missing files, and they would not affect the exit status of the process.

Thank you.

Version-Release number of selected component (if applicable):
coreutils-8.24-4.fc23.x86_64

Comment 1 Kamil Dudka 2015-10-30 13:02:18 UTC
A similar request (including a patch) was posted to the upstream mailing-list in 2010:

https://lists.gnu.org/archive/html/coreutils/2010-12/msg00032.html

... and this was the solution suggested by upstream:

https://lists.gnu.org/archive/html/coreutils/2010-12/msg00036.html

Comment 2 Kamil Páral 2015-10-30 13:26:46 UTC
Ugh. That makes me very sad.

Of course that does not solve neither extraneous output (missing is not FAILED):

$ sha256sum -c *SUM 2>/dev/null
Fedora-Live-Cinnamon-x86_64-23-10.iso: FAILED open or read
Fedora-Live-KDE-x86_64-23-10.iso: OK
Fedora-Live-LXDE-x86_64-23-10.iso: FAILED open or read
Fedora-Live-MATE_Compiz-x86_64-23-10.iso: FAILED open or read
Fedora-Live-SoaS-x86_64-23-10.iso: FAILED open or read
Fedora-Live-Xfce-x86_64-23-10.iso: FAILED open or read
Fedora-Server-DVD-x86_64-23.iso: OK
Fedora-Server-netinst-x86_64-23.iso: OK
Fedora-Workstation-netinst-x86_64-23.iso: OK
Fedora-Live-Workstation-x86_64-23-10.iso: OK

nor the exit code.

Is there any chance some developer watching RH Bugzilla could pick this up, or is the only reasonable approach here to contact coreutils upstream mailing list?

Comment 3 Kamil Dudka 2015-10-30 14:04:20 UTC
(In reply to Kamil Páral from comment #2)
> nor the exit code.

You can get the exit code from grep, too, if this is intended for scripting.

> Is there any chance some developer watching RH Bugzilla could pick this up,
> or is the only reasonable approach here to contact coreutils upstream
> mailing list?

Upstream ML is definitely a better place for requesting new features unless they are related to downstream patches (which this one is certainly not).

Comment 4 Kamil Páral 2015-10-30 15:14:23 UTC
Thanks for info. I wanted to avoid mailing lists, that's why I put it here as my first option. Closing, will consider reporting upstream.

Comment 5 Pádraig Brady 2015-11-01 16:46:25 UTC
One reason we _might_ consider this is because currently open() and read() errors are not distinguished. You would want to treat a read() error like a checksum error, and not like an open()==ENOENT. Also other errors from open()
should not be treated like ENOENT.

Another reason is because grep -E '(OK|FAILED)$' is locale dependent,
so one would need LC_ALL=C md5sum ... | grep ...

I was 50:50 before, but with the above arguments I'm 60:40 for adding the option.

I'll propose upstream.

Comment 6 Kamil Páral 2015-11-03 11:58:51 UTC
Thanks Pádraig. While working around the problem with grep is possible, it is a) complicated, especially when considering all the corner cases b) not reliable, unless you know exactly what strings *sum tools generate to the output (i.e. you would need to study the source code). Moreover stdout/stderr is not like an API which is guaranteed not to change, your workaround script can break any time.

Grep is useful for many things, but modifying program behavior considerably based on stdout/stderr and adjusting exit codes is, I believe, not one of them. Having a specific option to distinguish missing files from unreadable files and invalid checksum files would make everything much easier here.

Comment 7 Pádraig Brady 2015-11-23 13:23:19 UTC
Upstream commit: http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=9fd0662f