Bug 1175759 - Checksum validation in PowerShell is wrong
Summary: Checksum validation in PowerShell is wrong
Status: ASSIGNED
Alias: None
Product: Fedora Documentation
Classification: Fedora
Component: install-guide
Version: devel
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Pete Travis
QA Contact: Fedora Docs QA
URL:
Whiteboard:
Keywords: Reopened
: 1226101 1266693 1282226 1299138 1303424 1303428 1303989 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-18 14:11 UTC by gene
Modified: 2018-11-05 10:21 UTC (History)
14 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2014-12-21 05:48:53 UTC


Attachments (Terms of Use)

Description gene 2014-12-18 14:11:51 UTC
Description of problem:
I tried to use the info at http://docs.fedoraproject.org/en-US/Fedora/21/html/Installation_Guide/sect-verifying-images.html and found that the commands did not work due to missing quotes and case sensitivity.

Version-Release number of selected component (if applicable):
Fedora 21

How reproducible:
Every time for me.

Steps to Reproduce:
1. Download ISO
2. Download checksum file
3. Use code on http://docs.fedoraproject.org/en-US/Fedora/21/html/Installation_Guide/sect-verifying-images.html

Actual results:
PS D:\Downloads> $image = "Fedora-Live-Workstation-x86_64-21-5.iso"
PS D:\Downloads> $checksum_file = "Fedora-Workstation-21-x86_64-CHECKSUM.md5"
PS D:\Downloads> $sha256 = New-Object -TypeName System.Security.Cryptography.sha256CryptoSer
viceProvider
PS D:\Downloads> $exepected_checksum = ((Get-Content $checksum_file | Select-String -Pattern
 $image) -split " ")[0]
PS D:\Downloads> $download_checksum = [System.BitConverter]::ToString($sha256.ComputeHash([S
ystem.IO.File]::ReadAllBytes($PWD\$image)))
At line:1 char:109
+ ... adAllBytes($PWD\$image)))
+                    ~
Missing ')' in method call.
At line:1 char:109
+ ... adAllBytes($PWD\$image)))
+                    ~~~~~~~
Unexpected token '\$image' in expression or statement.
At line:1 char:116
+ ... tes($PWD\$image)))
+                    ~
Unexpected token ')' in expression or statement.
At line:1 char:117
+ ... es($PWD\$image)))
+                    ~
Unexpected token ')' in expression or statement.
At line:1 char:118
+ ... s($PWD\$image)))
+                    ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInMethodCall


Expected results:
No errors in console

Additional info:
I fixed the code and made the output more readable. Below is what I think should be posted on that page of the docs:

$image = "Fedora-Live-Workstation-x86_64-21-5.iso"
$checksum_file = "Fedora-Workstation-21-x86_64-CHECKSUM"
$sha256 = New-Object -TypeName System.Security.Cryptography.sha256CryptoServiceProvider
$expected_checksum = ((Get-Content $checksum_file | Select-String -Pattern $image) -split " ")[0].ToLower()
$download_checksum = [System.BitConverter]::ToString($sha256.ComputeHash([System.IO.File]::ReadAllBytes("$PWD\$image"))).ToLower() -replace '-', ''
if ( "$download_checksum" -eq "$expected_checksum" ) {
  echo "Checksum test passed!"
} else {
  echo "Checksum test failed."
}
echo "Download Checksum: $download_checksum"
echo "Expected Checksum: $expected_checksum"

Comment 1 Zach Oglesby 2014-12-18 19:07:49 UTC
Trying this on a Windows 7 Pro Service Pack 1 provides the same results. gliverma's version provides the expected result.

Comment 2 Zach Oglesby 2014-12-21 03:14:46 UTC
After more testing the only issue with the original code was missing quotes in ReadAllBytes("$PWD\$image"). That change was made and is awaiting QA and release.

Comment 3 gene 2014-12-21 03:36:52 UTC
For what it's worth, I think the two echo lines at the end make the process much more transparent and would like to see them added. Assuming this is not the place to request enhancements beyond the missing quotes, how should I go about submitting my version of the script?

Comment 4 Pete Travis 2014-12-21 05:48:53 UTC
This is an appropriate place to request enhancements :)

I do agree that the two echo lines are more demonstrative.  After confirming that Zach's changes fixed the script, I also added those lines.

The updated guide has been published, coming soon to a mirror near you.

Comment 5 gene 2014-12-22 01:31:44 UTC
Awesome, thanks! For what it's worth, in the original there is text manipulation happening inside the if statement for the downloaded file's checksum. In my version, that has been moved to where the variable is defined so that the if just does the comparison and nothing more. If the echos are to be used this will need to also be accounted for.

Comment 6 Pete Travis 2014-12-22 14:41:47 UTC
d'oh!  I did not take that into account. reopening.

Comment 7 Pete Travis 2016-02-02 14:30:10 UTC
*** Bug 1226101 has been marked as a duplicate of this bug. ***

Comment 8 Pete Travis 2016-02-02 14:30:32 UTC
*** Bug 1299138 has been marked as a duplicate of this bug. ***

Comment 9 Pete Travis 2016-02-02 14:30:41 UTC
*** Bug 1303428 has been marked as a duplicate of this bug. ***

Comment 10 Pete Travis 2016-02-02 14:30:53 UTC
*** Bug 1282226 has been marked as a duplicate of this bug. ***

Comment 11 Pete Travis 2016-02-02 14:31:00 UTC
*** Bug 1266693 has been marked as a duplicate of this bug. ***

Comment 12 Pete Travis 2016-02-02 14:34:51 UTC
There have been a ton of bugs filed on this; it was an interesting proof of concept at first, but not especially user-friendly, and behavior seems to vary depending on the version of PowerShell invoked.

I've consolidated all the reports, and will use this one to track writing up a more approachable and maintainable verification procedure for Windows systems. Thanks to all who reported problems with the script, it truly helps improve the docs.

Comment 13 tlhackque 2016-02-02 15:01:09 UTC
(In reply to Pete Travis from comment #12)
> There have been a ton of bugs filed on this; it was an interesting proof of
> concept at first, but not especially user-friendly, and behavior seems to
> vary depending on the version of PowerShell invoked.

As I noted in bug 1299138, this isn't only dependent on the PowerShell version.
It also depends on the size of the ISO.  PowerShell can't deal with files larger than 2GB.

For a pure Microsoft solution, consider the M$ FCIV, which computes either or both MD5 and SHA1 checksums.

The download link is https://www.microsoft.com/en-us/download/details.aspx?id=11533

fciv -both foo.iso
fciv -sha1 foo.iso

(FCIV is "file check integrity verifier" :-()

Comment 14 Pete Travis 2016-02-03 01:04:37 UTC
*** Bug 1303989 has been marked as a duplicate of this bug. ***

Comment 15 Matthew Miller 2016-02-03 06:17:37 UTC
Thanks Pete. It looks like for F24, we're going to be emphasizing a new version of LiveUSB Creator as our primary download for at least Fedora Workstation. I believe it has checksum verification already integrated. I suppose that means the documentation should focus on that. See https://fedoraproject.org/wiki/Changes/LUCasPrimaryDownloadable

Comment 16 Pete Travis 2016-02-03 13:00:24 UTC
We did notice that LiveUSB Creator grew direct-write capability on Windows; with checksum verification too, it would nicely solve this problem.  I'll check it out, thanks.

Comment 17 Matthew Miller 2016-02-03 13:09:43 UTC
*** Bug 1303424 has been marked as a duplicate of this bug. ***

Comment 18 Ansgar Wiechers 2016-02-04 23:14:05 UTC
The 2 GB limit mentioned by tlhackque@yahoo.com is a limitation of the [IO.File]::ReadAllBytes() method, not of PowerShell. And that method shouldn't be used in the first place, because it would read the entire file into memeory. Read the file as a stream to prevent memory exhaustion.

The PowerShell code below should work on any Windows version that meets the following requirements:

- Windows XP SP3 or newer
- PowerShell v2 or newer
- .NET Framework 3.5 or newer

Tested on Windows XP SP3, Windows 7 SP1, Windows Server 2008 R2 and Windows Server 2012.

> $image = 'Fedora-Server-DVD-x86_64-21.iso'
> $checksum_file = 'Fedora-Server-21-x86_64-CHECKSUM'
>
> $sha256 = New-Object Security.Cryptography.SHA256Managed
> $stream = (Get-Item "$PWD\$image").OpenRead()
> $download_checksum = [BitConverter]::ToString($sha256.ComputeHash($stream)).ToLower() -replace '-'
> $stream.Close()
> 
> $expected_checksum = ((Get-Content $checksum_file) -match [regex]::Escape($image) -split ' += +')[1].ToLower()
>
> echo "Download Checksum: $download_checksum"
> echo "Expected Checksum: $expected_checksum"
> if ( $download_checksum -eq $expected_checksum ) {
>   echo "Checksum test passed!"
> } else {
>   echo "Checksum test failed."
> }

If you must you can squeeze the calculation of $download_checksum in a single line like this:

> $download_checksum = [BitConverter]::ToString((New-Object Security.Cryptography.SHA256Managed).ComputeHash((Get-Item "$PWD\$image").OpenRead())).ToLower() -replace '-'

Comment 19 tlhackque 2016-02-05 00:19:27 UTC
RE #18: Sure.  But why bother with PowerShell at all?  Just use FCIV.  Easier and much less error-prone than typing in a script.

Still to be resolved (as I also noted previously): verifying that the posted list of checksums was indeed signed by the Fedora distribution key and that that key is trusted.  All easy enough when you have a running system, not so easy to bootstrap.  Especially in a way that normal people can do with OS-bundled tools -- and simple enough that they'll bother.

At some point one needs a trust anchor.  Perhaps fetching the public PGP key from a DNSSEC secured zone, assuming Windows has a reasonable command line DNSSEC verifier.  (Bundled, or you have to verifiably download it.)  But, you still need to securely fetch the PGP verification software, as that isn't bundled with Windows either.

After you're done with all that - why not just serve the checksums and the data files on an https:// connection - on a DNSSEC-secured domain?

Get the download sites to be certificate/public key pinned (new browsers support this).  And it's as good as or better than any other software repo.

At that point, you don't need to sign the checksum list; you only need it to catch transmission or storage corruption.  Assuming the distribution servers are trustworthy.  If they're compromised, all the signatures have to be elsewhere, or the bad guys just put up an imitation website with their infected OS and corresponding checksums, signed by their fake key - the public version of which is posted on their imitation website.

All this is to say that you really need to step back and decide which problem(s) you are trying to solve, and exactly what assurances you are giving the downloader.

Are the bits the same as what left the server?

Are the bits on that server the ones released?
  Both the distribution data and the verification bits (checksums,keys)

Do you assume trustworthy servers?  Or (real world more likely) random re-distribution where, at best, some verification material can be found on trusted (fedora?) servers.

Is the solution just for servers under Fedora control (at least by policy)?

Getting the necessary tools onto the Windows system (verifiably)

Getting more than 1% of the customers to bother with going thru what you come up with... Which argues for an automated process - that has to be verifiable.

And so the wheel turns.

Comment 20 Ansgar Wiechers 2016-02-05 04:41:37 UTC
> Sure.  But why bother with PowerShell at all?  Just use FCIV.  Easier
> and much less error-prone than typing in a script.

- Doesn't support SHA-256.
- Needs to be downloaded first.

> Still to be resolved (as I also noted previously): verifying that the
> posted list of checksums was indeed signed by the Fedora distribution
> key and that that key is trusted.

Different problem. My response proposes a fix for the script in the installation guide. Nothing else.

Comment 21 Pete Travis 2016-02-05 20:40:27 UTC
You folks seem sharp with PowerShell; how much of a concern is the invoking version of powershell with this script?  Can you help build a checksum script that's agnostic about the version of Windows or Powershell in play, or a set of scripts to cover most use cases?  I really do like the idea of using native tools for this job, but I don't play in PowerShell these days and dropping it for a trusted downloadable utility is increasingly attractive.

Comment 22 tlhackque 2016-02-05 21:00:11 UTC
I'm not a PowerShell expert.

PowerShell is bundled with many recent releases of Windows; I think it was still optional as recently as Windows Server 2008.  You can probably find the history somewhere in MSDN.

As I noted in #19, it's not clear which of several problems you're actually trying to solve.  Fixing the script addresses only a small part of the 'do I have a good CD' question.

I would be inclined to create a down-loadable utility that:

- was signed with a fedora code-signing key (so we know the utility is good)
- can checksum the ISO (so we have a signature for the image)
- can compare the checksum to file containing the image checksum (is the image what the file says it is)
- can verify the checksum file against a built-in public key. (So we know the checksum file is authentic)

Whether that utility is coded in C or Powershell is an implementation question.

I'm not volunteering.  If I were responsible for this, that's the tack I'd take.

Comment 23 Ansgar Wiechers 2016-02-28 02:17:17 UTC
Sorry, took me a while to get back to this. As tlhackque mentioned, there are two different problems to solve if you want to guarantee the integrity of the downloaded image:

- verifying the downloaded image against the checksum file, and
- verifying that the checksum file hasn't been tampered with.

The first problem is addressed by the code I posted before. The code should run on every Windows version with PowerShell v2 and .NET Framework 2.0. I tested it on Windows XP SP3, Windows 7 SP1, Windows Server 2008 R2 and Windows Server 2012 R2. I don't have access to Windows Vista and Server 2008, but since they don't come with PowerShell pre-installed it doesn't matter anyway. All currently supported Windows versions can run at least PowerShell v2 and all versions that ship with PowerShell have v2 already installed, so I'd consider that a reasonable baseline.

The second problem is normally addressed by the PGP signature of the checksum file. However, PGP (or GnuPG) isn't commonly installed on Windows computers, and I'm also not aware of PowerShell modules or .NET classes providing this functionality. The user would need to install something like gpg4win, e.g. via chocolatey (run as administrator):

> Set-ExecutionPolicy RemoteSigned -Force
> iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
> # restart PowerShell (on one system I had to reboot)
> choco install gpg4win

import the Fedora key and verify its fingerprint (as user):

> & "$env:ProgramFiles\GNU\GnuPG\gpg2.exe" --keyserver pgp.mit.edu --search-key 34EC9CBA
> & "$env:ProgramFiles\GNU\GnuPG\gpg2.exe" --fingerprint 34EC9CBA

On 64-bit systems the path is "${env:ProgramFiles(x86)}\GNU\GnuPG\gpg2.exe".

Comment 24 Herb 2016-07-05 21:30:32 UTC
I had a problem with the Fedora procedure for checksum & used your solution of "Ansgar Wiechers 2016-02-04 18:14:05 EST" on the Windows 10 platform & it worked as advertised.  Thanks for the great assistance!!

Comment 25 Aaron 2016-09-13 04:41:27 UTC
Hello everyone!

Thank you for all the hard work! @Ansgar Wiechers thank you for helping with this, I wouldn't have ever figured it out. I am less than a novice when it comes to scripts and only know enough to get myself into serious trouble. 

To my point: as I followed the thread a thought comes to mind - and I ask please forgive if I insult intelligence or really expose my ignorance - would it be possible to: 
1)list all of the hashes on a web page, 
2)then build a special verifier program similar to "MD5 and SHA Checksum Utility.exe" 
3)and for security purposes only downloadable from your website and can only check fedora .iso, etc?

Would that solve the problem? 

Just a thought.

Thank you again!

Comment 26 Aaron 2016-09-13 04:55:10 UTC
Sorry for posting twice in a row: Just to clarify all of this would be on just one page on the web dedicated to verification. If someone didn't simply want to heck against the published list using a third party verifier, then the special verifier is an extra level of protection and the program would be a stand alone needing no installation and already have the full list of hashes internally, automatically comparing, offering only a "pass/fail". The only thing that would need to be verified then is the download of the verifier program. On that same web page have a web based pass/fail simply to check the download of the verifier. 

I hope that was clear. Let me know if you have questions.

Comment 27 dave.keber 2017-07-14 15:24:43 UTC
Did someone forget to post the corrected script to the documentation page? As of Fedora 26, this is still broken:

https://docs.fedoraproject.org/en-US/Fedora/26/html/Installation_Guide/sect-verifying-images.html

Comment 29 Richard Hrkach Jr 2018-11-05 10:21:44 UTC
PS C:\Users\Richard\Downloads\Fedora> $image = 'Fedora-Server-dvd-x86_64-29-1.2.iso'
PS C:\Users\Richard\Downloads\Fedora> $checksum_file = 'Fedora-Server-29-Checksum'
PS C:\Users\Richard\Downloads\Fedora> $sha256 = New-Object Security.Cryptography.SHA256Managed
PS C:\Users\Richard\Downloads\Fedora> $stream = (Get-Item "$PWD\$image").OpenRead()
PS C:\Users\Richard\Downloads\Fedora> $download_checksum = [BitConverter]::ToString($sha256.ComputeHash($stream)).ToLowe
r() -replace '-'
PS C:\Users\Richard\Downloads\Fedora> $stream.Close()
PS C:\Users\Richard\Downloads\Fedora> $expected_checksum = ((Get-Content $checksum_file) -match [regex]::Escape($image)
-split ' += +')[1].ToLower()
PS C:\Users\Richard\Downloads\Fedora> echo "Download Checksum: $download_checksum"
Download Checksum: 129d131a55e5bd518f593f0eacdce095f7c795fe7ccbef1f3f6aeb2ff9f99f35
PS C:\Users\Richard\Downloads\Fedora> echo "Expected Checksum: $expected_checksum"
Expected Checksum: sha256 (fedora-server-dvd-x86_64-29-1.2.iso)
PS C:\Users\Richard\Downloads\Fedora>

Problem is with the second last line there is no checksum to compare, ran on windows 8.1 for Fedora 29 Server


Note You need to log in before you can comment on or make changes to this bug.