Bug 877623 - fedup does not verify online update sources
Summary: fedup does not verify online update sources
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker
Depends On:
Blocks: F18-accepted, F18FinalFreezeExcept
TreeView+ depends on / blocked
 
Reported: 2012-11-17 11:24 UTC by Till Maas
Modified: 2015-10-30 16:51 UTC (History)
25 users (show)

Fixed In Version: fedup-0.8.0-3.fc19
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-19 07:21:07 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Till Maas 2012-11-17 11:24:01 UTC
Description of problem:
According to https://fedoraproject.org/wiki/QA%3AFedora_18_Upgrade_Testing fedup uses a plain http connection to fetch updates from http://tflink.fedorapeople.org/fedup/f18-upgrade/.
Also the code does not show any sign of proper cryptographic verification of updates. Therefore it seems that it is prone to man-in-the-middle attacks or maybe allows to install packages from compromised mirrors without any verification.

Version-Release number of selected component (if applicable):
fedup-0.7-1.fc17

Comment 1 Tim Flink 2012-11-19 15:57:17 UTC
The current process on that wiki page is, AFAIK, not what the final process is going to look like. Eventually, the initrd is going to be generated as part of the releng processess and '--instrepo' isn't going to be required.

I'm pretty sure that the downloaded initramfs/vmlinuz pair is verified using the sha256sum in .treeinfo but this is in the background, though.

Comment 2 Till Maas 2012-11-19 16:48:49 UTC
(In reply to comment #1)
> The current process on that wiki page is, AFAIK, not what the final process
> is going to look like. Eventually, the initrd is going to be generated as
> part of the releng processess and '--instrepo' isn't going to be required.
> 
> I'm pretty sure that the downloaded initramfs/vmlinuz pair is verified using
> the sha256sum in .treeinfo but this is in the background, though.

Where will the .treeinfo file come from in the future? And will the rpm packages be verified?

Comment 3 Adam Williamson 2012-11-19 16:51:16 UTC
Discussed at 2012-11-19 QA meeting, acting as a blocker review meeting. We agreed we need more information here.

1) Till, what exactly are you concerned about? fedup downloads a couple of things: the initrd/vmlinuz pair, which as tflink notes should be served from the official repos (getting them from tflink's fp space is just a short-term hack), and then the packages for update. Could you clarify precisely what bit of the process you're concerned about, and what you think should be happening?

2) Will, what's your general take on this area? What verification of what bits is a) needed and b) currently implemented? Particularly, are package signatures currently being verified or not?

Thanks!

Comment 4 Till Maas 2012-11-19 17:33:09 UTC
(In reply to comment #3)
> Discussed at 2012-11-19 QA meeting, acting as a blocker review meeting. We
> agreed we need more information here.
> 
> 1) Till, what exactly are you concerned about? fedup downloads a couple of
> things: the initrd/vmlinuz pair, which as tflink notes should be served from
> the official repos (getting them from tflink's fp space is just a short-term
> hack), and then the packages for update. Could you clarify precisely what
> bit of the process you're concerned about, and what you think should be
> happening?

About the initrd/vmlinz pair:
It is completely unclear where it will be come from. If it comes from mirrors, some process to verify them is missing. For example ISO images currently need to be verified manually as described in https://fedoraproject.org/en/verify
There is no indication that this happens with the images at tflink's fp space or that it will be done where the files will be hosted in the future.

About the rpm packages:
Current git from git://github.com/wgwoods/fedup.git contains a comment in download.py:
# TODO check signatures of downloaded packages
See:
https://github.com/wgwoods/fedup/blob/master/fedup/download.py

Therefore the rpm packages seem not to be verified either. Nevertheless, any code downloaded from the Internet that is executed during installation needs to be verified by fedup, in case there are other files to be downloaded.

Comment 5 Till Maas 2012-11-19 17:37:00 UTC
Btw. my proposal would be to

1) Include all current Fedora GPG keys in fedora-release (in /etc/pki/rpm-gpg/)
2) Sign the initrd/vmlinuz files like the ISO images.
3) Patch fedup to verify the signatures of all downloaded files. For the packages it can probably borrow code from yum as soon as the GPG keys are available in /etc/pki/rpm-gpg

Comment 6 Tim Flink 2012-11-19 18:04:13 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > The current process on that wiki page is, AFAIK, not what the final process
> > is going to look like. Eventually, the initrd is going to be generated as
> > part of the releng processess and '--instrepo' isn't going to be required.
> > 
> > I'm pretty sure that the downloaded initramfs/vmlinuz pair is verified using
> > the sha256sum in .treeinfo but this is in the background, though.
> 
> Where will the .treeinfo file come from in the future? And will the rpm
> packages be verified?

The .treeinfo will be generated by pungi as it currently is.

Comment 7 Tim Flink 2012-11-19 18:08:09 UTC
(In reply to comment #4)
> About the initrd/vmlinz pair:
> It is completely unclear where it will be come from. If it comes from
> mirrors, some process to verify them is missing. For example ISO images
> currently need to be verified manually as described in
> https://fedoraproject.org/en/verify
> There is no indication that this happens with the images at tflink's fp
> space or that it will be done where the files will be hosted in the future.

There are still a couple details that need to be worked out but the current plan is to generate a separate upgrade.img with the current iso generation, using lorax. We are currently waiting for a new lorax build which is capable of doing this, though

> About the rpm packages:
> Current git from git://github.com/wgwoods/fedup.git contains a comment in
> download.py:
> # TODO check signatures of downloaded packages
> See:
> https://github.com/wgwoods/fedup/blob/master/fedup/download.py
> 
> Therefore the rpm packages seem not to be verified either. Nevertheless, any
> code downloaded from the Internet that is executed during installation needs
> to be verified by fedup, in case there are other files to be downloaded.

OK, I thought that this bug was about verifying the upgrade.img and vmlinuz. I'm not saying that verifying the signatures isn't important but I'm wondering if we really need it for beta or if it could be put off until final.

Comment 8 Till Maas 2012-11-19 18:18:36 UTC
(In reply to comment #7)

> There are still a couple details that need to be worked out but the current
> plan is to generate a separate upgrade.img with the current iso generation,
> using lorax. We are currently waiting for a new lorax build which is capable
> of doing this, though

Where will the upgrade.img file be stored and how will it be downloaded/verified by fedup?

> OK, I thought that this bug was about verifying the upgrade.img and vmlinuz.

The bug is about the verification of these files and the packages.

Comment 9 Tim Flink 2012-11-19 18:34:14 UTC
(In reply to comment #8)
> (In reply to comment #7)
> 
> > There are still a couple details that need to be worked out but the current
> > plan is to generate a separate upgrade.img with the current iso generation,
> > using lorax. We are currently waiting for a new lorax build which is capable
> > of doing this, though
> 
> Where will the upgrade.img file be stored and how will it be
> downloaded/verified by fedup?

My understanding is that the upgrade.img will be in the same images/pxeboot/ dir that the current vmlinuz and initrd.img are located in. Fedup will get the location of the images through the .treeinfo and verifys them with the sha256sum that is put in the .treeinfo by pungi.

I am not aware of any plans to use GPG checking, though.

> > OK, I thought that this bug was about verifying the upgrade.img and vmlinuz.
> 
> The bug is about the verification of these files and the packages.

Yeah, I'm seeing that now. Still wondering if package GPG checking has to be in place for beta, though.

Comment 10 Till Maas 2012-11-19 18:47:36 UTC
(In reply to comment #9)

> My understanding is that the upgrade.img will be in the same images/pxeboot/
> dir that the current vmlinuz and initrd.img are located in. Fedup will get
> the location of the images through the .treeinfo and verifys them with the
> sha256sum that is put in the .treeinfo by pungi.

Where will the .treeinfo file come from? Will it be shipped with fedup?

> Yeah, I'm seeing that now. Still wondering if package GPG checking has to be
> in place for beta, though.

IMHO it is an important feature and should therefore be carefully tested.

Comment 11 Tim Flink 2012-11-19 19:31:49 UTC
(In reply to comment #10)
> (In reply to comment #9)
> 
> > My understanding is that the upgrade.img will be in the same images/pxeboot/
> > dir that the current vmlinuz and initrd.img are located in. Fedup will get
> > the location of the images through the .treeinfo and verifys them with the
> > sha256sum that is put in the .treeinfo by pungi.
> 
> Where will the .treeinfo file come from? Will it be shipped with fedup?

The .treeinfo file is generated by pungi during compose. Off the top of my head, I don't know how long this has been done but I know it was done this way in F17 and I suspect that it started longer ago than that.

.treeinfo is not part of fedup nor should it be. Fedup gets the repo url and the .treeinfo file is in the base directory of that repo. Fedup will download the .treeinfo file with the sha256sums, grab the upgrade.img/vmlinuz and verify the checksums.

> > Yeah, I'm seeing that now. Still wondering if package GPG checking has to be
> > in place for beta, though.
> 
> IMHO it is an important feature and should therefore be carefully tested.

Like I said, I'm not arguing that it isn't important - just questioning whether it really needs to be in beta. However, that is a question for the blocker review on wednesday and/or fesco.

Comment 12 Till Maas 2012-11-19 21:57:35 UTC
(In reply to comment #11)

> .treeinfo is not part of fedup nor should it be. Fedup gets the repo url and
> the .treeinfo file is in the base directory of that repo. Fedup will
> download the .treeinfo file with the sha256sums, grab the
> upgrade.img/vmlinuz and verify the checksums.

So it does not verify the .treeinfo file? Then a malicious mirror can easily compromise the upgrade.img or vmlinuz file and create a matching .treeinfo file withouth fedup noticing.

Comment 13 Will Woods 2012-11-19 22:28:51 UTC
(the tl;dr version: 

  - this isn't any different from preupgrade/anaconda
  - it was never a release blocker before
  - it can be fixed, but rel-eng needs to sign some stuff

For more details, read on...)

The boot image checksums are currently verified against the checksum data in .treeinfo. 

This is equivalent to the DVD media check, and the same as what preupgrade,  anaconda, virt-install, etc. have always done during network installs and upgrades. If it wasn't a release blocker during any previous release it's probably not one now.

But Till is right - this is *not* sufficient to verify the *authenticity* of the files. For that, you do need to check signatures - either on the files themselves, or the checksums (e.g. signed .treeinfo data).

For example - for DVDs, https://fedoraproject.org/en/verify provides GPG-signed SHA256 hashes of the media; so if:

  a) the file containing the hash is verified using the Fedora Project key, and
  b) the hash in the file matches the hash of the image/media,

then the image/media is authentic.

So in order for fedup to verify the images as being authentic, **rel-eng will need to sign something** for it to check. There's a lot of ways this could work. Here's a couple options:

Option 1: rel-eng signs the .treeinfo files and puts a detached signature in .treeinfo.sig or .treeinfo.asc. fedup uses that signature to verify the authenticity of .treeinfo before downloading images. (If I understand correctly this is what Till is suggesting in comment #5.)

Option 2: rel-eng puts various data for each release in a "fedora-release-data" package: version number, codename, ISO checksums, image checksums, install URLs, etc. Since packages are signed, we can have RPM verify these file(s) and then compare those checksums to the downloaded images.

(The *package* signatures can easily be checked using yum; see the comment at the end of download_packages() in fedup/download.py. This is basically the infamous and ancient bug #998, which preupgrade and anaconda also share.)

Comment 14 Will Woods 2012-11-19 22:46:50 UTC
Note for the record that we had this conversation 3 years ago, when you filed the same bug against preupgrade:

  https://bugzilla.redhat.com/show_bug.cgi?id=509338

It's all still moot until there's a signature for the tool to verify.

So this isn't going anywhere until rel-eng et. al. decide which files to sign, and which key to sign them with.

(Keep in mind that the F17 system might not have the F18 key, so if the .treeinfo/images are signed with the F18 key then the user has to verify the key, and then you have bug #998 again..)

Comment 15 Till Maas 2012-11-19 23:40:31 UTC
(In reply to comment #13)

> This is equivalent to the DVD media check, and the same as what preupgrade, 
> anaconda, virt-install, etc. have always done during network installs and
> upgrades. If it wasn't a release blocker during any previous release it's
> probably not one now.

(In reply to comment #14)
> Note for the record that we had this conversation 3 years ago, when you
> filed the same bug against preupgrade:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=509338
> 
> It's all still moot until there's a signature for the tool to verify.
> 
> So this isn't going anywhere until rel-eng et. al. decide which files to
> sign, and which key to sign them with.

There was an upgrade method for previous releases that allowed to verify the update images which is afaik gone now. Therefore the situation has changed. And it is certainly not good PR for Fedora to drop the only left secure upgrade method. Btw. I can certainly not understand the resistance of rel-eng to properly sign distributed packages/code.

And a related note: Around the F13 schedule I noticed that preupgrade could use the signed netinstall.iso to get the image/kernel instaed of treeinfo. Maybe this is an option now as well.

Comment 16 Adam Williamson 2012-11-20 00:20:39 UTC
I'm -1 beta blocker on this, FWIW. +1 nth for any reasonably small changes to check signatures, I guess. I might be +1 final blocker on generally being able to check the integrity of the upgrade path, but not beta.

Comment 17 Jaroslav Reznik 2012-11-20 11:18:26 UTC
-1 beta blocker, not sure about nth as also work on rel-eng side is needed and we need it stable now (so slightly -1 nth for beta). But I agree we should have an option for secure updates, I'd say we can discuss it for final again.

Comment 18 Tim Flink 2012-11-20 16:12:07 UTC
-1 beta blocker. We can re-visit the issue for final but I don't think that we should be blocking beta for this.

That makes for -3 blocker, rejecting for beta and proposing for final.

Comment 19 Till Maas 2012-11-28 18:44:30 UTC
How can this be moved forward? It is easy to patch fedup in general, but this requires someone to provide the necessary data in a secure method. This is nothing one can provide a patch for.

Comment 20 Will Woods 2012-11-30 21:47:26 UTC
I've talked to Dennis Gilmore in rel-eng about this a few times; the current plan is to put a GPG-signed copy of .treeinfo at .treeinfo.signed.

Fedup will try to download and verify the latter and use the data therein; if it can't be verified, it'll refuse to continue. There will probably be some option (e.g. "--nogpgcheck") to fall back to using the unsigned .treeinfo.

Comment 21 Will Woods 2012-11-30 21:51:11 UTC
..err, a note: The previous comment was just describing how I think the check should work. So please read "fedup will..." as "fedup should...". Patches would be most welcome.

Comment 22 Adam Williamson 2012-12-03 20:14:11 UTC
Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until there is a clear action plan rather than a vague 'it needs to be more secure!' feeling, at least.

Comment 23 Will Woods 2012-12-03 22:13:59 UTC
..to clarify the plan/concept as described in comment #20:

  1) rel-eng will create a GPG clearsigned .treeinfo.signed
    * ex: `gpg --output .treeinfo.signed --clearsign .treeinfo`

  2) tools which use .treeinfo should try to fetch/verify .treeinfo.signed
    * ex: `gpg --output treeinfo.out --decrypt .treeinfo.signed` returns 0
    a) If it can be verified, its cleartext (treeinfo.out) should be used
    b) If not, the tool may abort, or warn the user and fall back to .treeinfo

Open questions are: *which* key is going to be used to sign .treeinfo, and how we expect to establish trust of that key to allow verification.

For example: if rel-eng uses the F18 key to sign F18 .treeinfo, there will need to be some way to get the F18 key onto the system, and possibly some way of asking the user whether they trust the key.

If we consider the contents of /etc/pki/rpm-gpg trusted, the F18 key(s) could be packaged for F17. The package would be signed with the F17 key, which sort of establishes a chain of trust from F17 to F18.

This won't work for completely offline systems, though. But if the F18 public key *itself* was signed with the F17 key, the F18 key could be included on the install media, and tools could use that to get the F18 key into the F17 keyring.

Another option is to sign .treeinfo with both the F17 and F18 keys, but this would require using a detached signature and tweaking things to accept the file if *any* signature can be verified.

And so on. There's a lot of options. This is a distro-wide policy decision about establishing trust between versions of Fedora, so I'm guessing it's going to require a bunch of meetings and discussions and voting.

Once the policy is set, though, we can work on implementing it.

Comment 24 Till Maas 2012-12-03 23:43:26 UTC
(In reply to comment #23)

> Open questions are: *which* key is going to be used to sign .treeinfo, and
> how we expect to establish trust of that key to allow verification.

fedup needs access to the F18 key to verify the downloaded packages, therefore just signing the .treeinfo might not be enough.

Comment 25 Will Woods 2012-12-04 16:12:41 UTC
Right, so we need the F18 key in any case - so .treeinfo might as well be signed with the F18 key. Which makes sense.

How we get the F18 key and establish trust for it is still an open question.

Comment 26 Tim Flink 2012-12-04 20:52:47 UTC
I count -3 blocker votes, moving to rejected

Comment 27 Till Maas 2012-12-07 19:50:36 UTC
(In reply to comment #25)
> Right, so we need the F18 key in any case - so .treeinfo might as well be
> signed with the F18 key. Which makes sense.
> 
> How we get the F18 key and establish trust for it is still an open question.

The key can be downloaded via HTTPS from https://fedoraproject.org/static/DE7F38BD.txt

Comment 28 Will Woods 2012-12-20 18:12:01 UTC
The current plan here seems to be:


== rel-eng is going to put the F18 key into the F17 fedora-release package ==

dgilmore has already gone ahead and put the F19 key into the F18 fedora-release package: http://koji.fedoraproject.org/koji/rpminfo?rpmID=3581396


== fedup will prompt the user to import the F18 key(s) ==

fedup (the front-end) is based on yum, so it'll be the normal yum "Hey, import this key?" dialog.

One problem here is that fedora*.repo have this as their gpgkey= lines:

  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch

Notice no "$releasever" - so when we look for the F18 key, we'll get the same path as the F17 key, and the F17 key won't verify the F18 packages.

If it was RPM-GPG-KEY-fedora-$releasever-$basearch, we wouldn't have to do anything at all to find the right key. Might need to file a bug against fedora-release about this?


== fedup will verify that the packages, kernel, and initramfs are authentic ==

Any package that comes from a repo with "gpgcheck=1" will have its signature verified; the kernel and initramfs will always be verified.

We'll verify the packages using the standard RPM/yum stuff. The kernel and initramfs will be verified like this:

  a) try to fetch and verify .treeinfo.signed (but see NOTE 1 below)
  b) if that succeeds, use its contents to find and verify the kernel/initramfs

If any of these things fails verification, the upgrade will refuse to continue.

(There might be a "--nogpgcheck" option if the user wants to force the upgrade to continue. It may or may not be documented.)


== NOTE 1 about media ==

There's a bit of a chicken-and-egg problem with ISOs/media. How do we get the signed .treeinfo into the ISO if .treeinfo isn't signed until after the .ISO is built? There's a *manual* way to verify media:

  https://fedoraproject.org/verify

But it would be much better if the media (like the RPMs) contained its own signature.

Until we have .treeinfo.signed inside the media (or a signature for the media itself), I can see two ways of handling media:

  a) assume the user is responsible for checking the media, and skip verification, or
  b) just let it fail, forcing everyone doing media upgrades to use --nogpgcheck

The third (and best) option, of course, is to actually get signatures into the media so we can check it.


== NOTE 2 about media ==

The other problem is: if you're doing the upgrade from media because the machine you're upgrading has no network connection, how do you get the F18 key onto it?

(You can't just trust the key that's on the media - we don't yet trust the media!)

One suggestion might be that we need to include the updated 'fedora-release-17-*' on the media. That package is a) signed with the F17 media, so it's trusted by the system, and b) contains the F18 key, which we can therefore assume to be authentic and trustworthy.

I'm open to other suggestions, of course. I know there will be plenty of them.

Comment 29 Till Maas 2012-12-20 20:25:03 UTC
(In reply to comment #28)

> == fedup will prompt the user to import the F18 key(s) ==

> If it was RPM-GPG-KEY-fedora-$releasever-$basearch, we wouldn't have to do
> anything at all to find the right key. Might need to file a bug against
> fedora-release about this?

This sounds good to me. Another option in the long term might be to make it understand gpg fingerprints, which would allow to fetch the key just from a gpg keyserver.

> == NOTE 1 about media ==
> 
> There's a bit of a chicken-and-egg problem with ISOs/media. How do we get
> the signed .treeinfo into the ISO if .treeinfo isn't signed until after the
> .ISO is built? There's a *manual* way to verify media:
> 
>   https://fedoraproject.org/verify
> 
> But it would be much better if the media (like the RPMs) contained its own
> signature.
> 
> Until we have .treeinfo.signed inside the media (or a signature for the
> media itself), I can see two ways of handling media:
> 
>   a) assume the user is responsible for checking the media, and skip
> verification, or
>   b) just let it fail, forcing everyone doing media upgrades to use
> --nogpgcheck
> 
> The third (and best) option, of course, is to actually get signatures into
> the media so we can check it.

I would favour the third option and maybe support a) with fedup, e.g. allow to easily verify the media with a CHECKSUM-file with fedup.

> == NOTE 2 about media ==
> 
> The other problem is: if you're doing the upgrade from media because the
> machine you're upgrading has no network connection, how do you get the F18
> key onto it?
> 
> (You can't just trust the key that's on the media - we don't yet trust the
> media!)
> 
> One suggestion might be that we need to include the updated
> 'fedora-release-17-*' on the media. That package is a) signed with the F17
> media, so it's trusted by the system, and b) contains the F18 key, which we
> can therefore assume to be authentic and trustworthy.
> 
> I'm open to other suggestions, of course. I know there will be plenty of
> them.

Another option would be to sign Fedora keys with other Fedora keys and distribute the key together with signatures to allow to use older keys to verify newer ones.

Btw. this bug needs probable be to be a NTH to allow for changes in the F18 release media, that allow to implement this.

Comment 30 Will Woods 2012-12-21 17:54:38 UTC
A planning note:

I'm out for the holidays, so I'm not even going to be able to start on this until Jan 2 or so. It will take at least a couple of weeks to write signature-verification code and get it working well enough to give to QA.

Let me also emphasize this fact: the signature checking situation *as it stands* for Fedora 18 is equivalent to *every previous release* since (and before) bug 998, filed in January 1999.

So if you want to block the release over this - even though it hasn't been a blocker for any other release in the past 15 years - you will need to delay F18 until February 2013, at the earliest.

Comment 31 Till Maas 2012-12-21 19:32:54 UTC
(In reply to comment #30)

> Let me also emphasize this fact: the signature checking situation *as it
> stands* for Fedora 18 is equivalent to *every previous release* since (and
> before) bug 998, filed in January 1999.

If fedup can now be used to update from an iso image that could be verified as described in https://fedoraproject.org/verify, then we are in the  similiar situation as previously.

Comment 32 Till Maas 2012-12-21 20:14:09 UTC
Another note from me: It would help to somehow get access to a repository with a sample .treeinfo.signed-file or at least such a file to help with development.

Comment 33 Till Maas 2012-12-21 20:16:04 UTC
Btw. does the filename need to be .treeinfo? This makes it less obvious that it even exists, because it is not shown in directory listings such as in http://tflink.fedorapeople.org/fedup/f18-upgrade/i386/

Comment 34 Martin Banas 2013-06-11 08:48:16 UTC
Hi Will,
any update on this one? Did you have a chance to work on signature verification? Is there already something to test?

Comment 35 Need Real Name 2013-07-03 07:13:30 UTC
How on earth did we get to the point where an insecure way of upgrading became widespread (and even trivially possible!)

I thought Fedora was secure by default? Isn't that the point of having SELinux installed and enabled by default? Isn't that the point of standardising on one cryto library?

Comment 36 Adam Williamson 2013-07-03 07:17:25 UTC
Well, the giant piles of hyperbole helped.

Comment 37 Fedora End Of Life 2013-07-04 04:22:04 UTC
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 38 Kevin Fenzi 2013-07-07 17:05:40 UTC
So, we didn't get this in for 19, can we try for 20?

I guess we need to discuss how best to sign treeinfo files in our processes?

The Fedora-N+1 key is already shipped with Fedora-N release (but then does this mean we wouldn't support fedup of say f17->f19?)

Is there anything we can setup to help code/develop this?

Comment 39 Till Maas 2013-07-07 17:19:09 UTC
(In reply to Kevin Fenzi from comment #38)
> So, we didn't get this in for 19, can we try for 20?
> 
> I guess we need to discuss how best to sign treeinfo files in our processes?

comment:23 gave already clear command line instructions.

> The Fedora-N+1 key is already shipped with Fedora-N release (but then does
> this mean we wouldn't support fedup of say f17->f19?)

For other reasons it would be good to always have all supported GPG keys in all fedora-release packages, e.g. it would make it easier to build packages with mock.

> Is there anything we can setup to help code/develop this?

As stated several times, I would also help if there was just some sample repo with a signed file. E.g. just signing the F19 repo would allow to easily test this with a sample F18 machine and help to implement this.

Comment 40 Kevin Fenzi 2013-07-07 17:27:56 UTC
(In reply to Till Maas from comment #39)
> (In reply to Kevin Fenzi from comment #38)
> > So, we didn't get this in for 19, can we try for 20?
> > 
> > I guess we need to discuss how best to sign treeinfo files in our processes?
> 
> comment:23 gave already clear command line instructions.

That doesn't help us any, we don't directly use gpg with the signing keys. ;) 

I meant: How are we going to handle signing .treeinfo in images. If we need them signed, it means the compose process has to stop and wait for rel-eng to sign that file, then continue to compose the image, which is a pretty anoying slowdown in workflow. Do we need the ones on the images signed? 

> > The Fedora-N+1 key is already shipped with Fedora-N release (but then does
> > this mean we wouldn't support fedup of say f17->f19?)
> 
> For other reasons it would be good to always have all supported GPG keys in
> all fedora-release packages, e.g. it would make it easier to build packages
> with mock.

Sure, but that means either we allocate keys at least 2 releases in advance or update fedora-release on old releases. 
 
> > Is there anything we can setup to help code/develop this?
> 
> As stated several times, I would also help if there was just some sample
> repo with a signed file. E.g. just signing the F19 repo would allow to
> easily test this with a sample F18 machine and help to implement this.

Not sure we want to change the f19 tree, but we could look at setting up some test tree, sure. ;)

Comment 41 Till Maas 2013-07-07 17:41:21 UTC
Just wondering, where is the .treeinfo-file for F19 supposed to be, e.g. for Fedora 18 I found one at:
http://download.fedoraproject.org/pub/fedora/linux/releases/18/Everything/x86_64/os/.treeinfo

But it does not work for Fedora 19:
$ curl -L http://download.fedoraproject.org/pub/fedora/linux/releases/19/Everything/x86_64/os/.treeinfo
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/0.7.67</center>
</body>
</html>

Comment 42 Kevin Fenzi 2013-07-07 17:47:36 UTC
Everything shouldn't normally have a .treeinfo (it's not directly installable). You want the Fedora/ tree.

http://download.fedoraproject.org/pub/fedora/linux/releases/19/Fedora/x86_64/os/.treeinfo

Comment 43 Till Maas 2013-07-07 18:04:03 UTC
(In reply to Kevin Fenzi from comment #40)
> (In reply to Till Maas from comment #39)
> > (In reply to Kevin Fenzi from comment #38)
> > > So, we didn't get this in for 19, can we try for 20?
> > > 
> > > I guess we need to discuss how best to sign treeinfo files in our processes?
> > 
> > comment:23 gave already clear command line instructions.
> 
> That doesn't help us any, we don't directly use gpg with the signing keys.
> ;) 
> 
> I meant: How are we going to handle signing .treeinfo in images. If we need
> them signed, it means the compose process has to stop and wait for rel-eng
> to sign that file, then continue to compose the image, which is a pretty
> anoying slowdown in workflow. Do we need the ones on the images signed? 

Is this workflow somewhere described? From only an package maintainers POV it is not very transparent what is happening where etc. I assume that you ask whether the signed ISO images need to contain signed treefinfo files? IMHO they are not strictly needed, but I assume the treeinfo signature could easily be added at the last step before running genisoimage for the ISO images, so that there is only a slight delay between signing the treeinfo file and signing the ISO images. However, if this is too much trouble, then just singing the normal treeinfo file would be a great improvement.

Comment 44 Kevin Fenzi 2013-07-07 18:09:52 UTC
https://fedoraproject.org/wiki/Composing_test_images (although it's somewhat out of date and could use an update). 

Any long running process that stops at some point for human input is anoying. :) 

Anyhow, I think that question can be hashed out by fedup maintainers and releng folks, so lets wait for their input.

Comment 45 Till Maas 2013-07-07 20:27:55 UTC
(In reply to Kevin Fenzi from comment #44)
> https://fedoraproject.org/wiki/Composing_test_images (although it's somewhat
> out of date and could use an update). 
> 
> Any long running process that stops at some point for human input is
> anoying. :) 

The described procedure does not explain how the images are signed, which is probably already a point where human input is required.

> Anyhow, I think that question can be hashed out by fedup maintainers and
> releng folks, so lets wait for their input.

At least the treeinfo part was already discussed according to comment:20.

Comment 46 Dennis Gilmore 2013-07-08 15:42:45 UTC
we don't sign the images at all, we dont sign the CHECKSUM files until we are staging the release as a final release. i.e. staging for the mirrors, TC's and RC's never have any of the CHECKSUM files signed, its a manual process thats run after. any signing step that we do requires manual intervention to input siging key pass phrases and thats not going to change.

so we could make a .treeinfo.signed file that's signed by the key when staging the release.

Comment 47 Till Maas 2013-07-09 16:00:48 UTC
(In reply to Dennis Gilmore from comment #46)
> we don't sign the images at all, we dont sign the CHECKSUM files until we
> are staging the release as a final release. i.e. staging for the mirrors,
> TC's and RC's never have any of the CHECKSUM files signed, its a manual
> process thats run after. any signing step that we do requires manual
> intervention to input siging key pass phrases and thats not going to change.

I see.

> so we could make a .treeinfo.signed file that's signed by the key when
> staging the release.

This would be fantastic. Can we get this done for an example repo? What needs to be patched to do this? Can you create a .treeinfo.signed file for F19? It does not necessarily need to be distributed by the mirrors, attaching it to the bug report here would help already.

Comment 48 Joel Rees 2013-08-05 01:29:11 UTC
I'm going to do the bull-in-the-china-shop thing and talk about idealisms here for a minute.

But I don't really see what is holding things up here.

Eventually, somebody needs to write a mirror security package, including functions for mirrors to identify themselves to each other periodically, and to periodically spot-check each others' downloads. But that's a separate problem.

Rolling keys over periodically is a good thing. Generating a new set of keys for the next release and signing them with the keys of the current release is just correct procedure when using keys and certificates and the like to validate the code. You don't revoke the current keys or old keys. (Unless there is some security event requiring revocation, in which case you want to generate replacement keys for the old releases anyway, if possible. You just generate the new keys and declare them valid for the release they are being generated for.

Which is the way it works now, right? Except you don't sign the new keys with the current keys, right?

I'm going to say that again: When rolling keys over, you usually sign the new keys with the old keys. It's that simple.

Isn't there already a package that has the list of keys that haven't been revoked and the releases they are valid for? Does it include fingerprints? If not, is there some reason it can't include fingerprints from part-way through F19, on?

(There definitely should be a list of revoked keys, probably as a package, and published in the on-line documentation. I know it's a short list, and we always want it to be a short list, but it should be there, for people to see when they search the web for a specific key or fingerprint. If it's a package, it should be on every valid mirror.)

If there is a package that will ultimately contain the next release's key and fingerprint, it will be at least be updated when the user performs the final update before the upgrade. (Which they are strongly advised to do, right?)

It is true that you can't upgrade a live CD or a write-protected system this way, but updating a live CD involves downloading and burning the CD, and updating a write-protected system always requires re-mount-ing rw.

I assume that there are non-obvious issues? Surely the release engineers are not so averse to generating and signing the keys?

Is the problem really just a matter of someone taking the time to do it?

Yes, whining, hoping that if the whining gets loud enough something will get done. 

This may not be a serious bug yet, but if/when it becomes a serious bug it will be a terrible bug to fix. Unless it has already been fixed by that time.

Comment 49 Joel Rees 2013-08-11 21:50:57 UTC
An addtional note:

Contrary to what it says at 

https://fedoraproject.org/wiki/FedUp

this is a regression.

In the past (F17 and before) the ${recommended_upgrade_method} allowed verification.

In the present (F18 and F19 and maybe F20) the ${recommended_upgrade_method} does not allow verification.

In other words, up through F17, downloading an install image, burning it into CD or USB, and using the upgrade option on that image was the recommended upgrade. 

Install images were verifiable. 

Not only were they verifiable, the checksums were posted on all the mirrors, and a rogue mirror would have to either post a perverted image and the checksum from the real image, hoping that the downloader would not check the checksum, or would have to post the checksum of the perverted image with the image, hoping that the downloader would not check to see that the checksum matched other mirrors. 

Or install the correct public key for the new version and check the fingerprint.

So the install images were verifiable and they were the recommended upgrade method.

Now, for whatever reason, fedup is the recommended upgrade method.

Install images are still verifiable, but they are not the recommended upgrade method. And, unless there is some magic not being mentioned involved where verifying the downloaded image magically verifies the separate tarballs that have the stuff that fedup uses -- the ramfs kernel, etc.. 

**** Verifying the installable media images is meaningless. ****

Well, there is some mention of using fedup from the downloaded image, but that is not explained, and most users will completely miss it. (Would it really work?)

Thus, the redommended method is no longer verifiable.

This absolutely is a regression.

Comment 50 Will Woods 2013-08-14 19:47:17 UTC
Actually, no. Verifying the downloaded image covers *all* the content of the image - including the kernel & initramfs used by fedup.

So `fedup --iso VERIFIED.ISO` (or `fedup --device` with verified media) is exactly as secure as upgrades using verified media from F17 and earlier.

Comment 51 Joel Rees 2013-08-19 12:46:34 UTC
Great. 

    fedup --iso VERIFIED.ISO

or

    fedup --device

Now put the command in the docs, so that users can see it. Otherwise, the average Jack reading the docs is going to think, "Okay, verify the media and fedup from the network." Which of course leaves an opening for a man-in-the-middle.

The existence of the man page is no help to the user reading the installation guide pages on the wike and the docs site. He's not going to be thinking about the man pages because he wants to believe that what he is reading should be sufficient for the average case.

Verification remains an afterthought all throughout Fedora.

Comment 52 chris 2013-09-07 02:37:44 UTC
Its crucial to have a verified online upgrade system that is easy to use for anyone. With rampant government surveillance, tainting system updates would be trivial as they already have MITM'd every stretch of the network. They apparently have no qualms over infecting systems of innocent people in order to pursue a few targets, with Stuxnet being one of the more famous examples involving pcs. Having NO encryption whatsoever is even worse than the md5 verification joke that MS was using. I can't imagine an enterprise distro like Fedora from which RHEL is forked from, not having such a basic security mechanism. Maybe it will be a reality by version 20? Is there any fundamental technical obstacles that prevent this?

Comment 53 Need Real Name 2013-09-08 15:56:04 UTC
I know this is how Fedora is these days: rip out a component and replace it with a new one that doesn't quite have all the functionality of the new one (and promise to add that functionality in the future), but...

Yeah.

Can we have this as a blocker?

Comment 54 Will Woods 2013-09-11 19:08:07 UTC
(In reply to Need Real Name from comment #53)
> I know this is how Fedora is these days: rip out a component and replace it
> with a new one that doesn't quite have all the functionality of the new one
> (and promise to add that functionality in the future), but...

I said it in comment 13, and comment 30, and comment 50, but I'll say it one last time: NOTHING HAS CHANGED. Anaconda has never, in the past 15 years, checked signatures on anything it downloaded.

The fact that you were not *aware* of the problems before still does not mean that they are *new* problems now.

(In reply to Need Real Name from comment #53)
> Can we have this as a blocker?

There's a procedure for proposing blockers. This isn't it.

Comment 55 Will Woods 2013-09-11 19:22:09 UTC
For those that actually care about helping get this bug fixed:

I just merged the "checksig" branch of fedup into master. It adds code to do the following:

1) Check signatures on downloaded packages,
2) Automatically import new GPG keys, *if* the requested key:
  a) already exists on disk,
  b) was installed by a package that was signed with a trusted key, and
  c) hasn't been tampered with since installation.
3) Check signatures on arbitrary files.

Once .treeinfo.signed exists fedup can use that to verify automatically verify both the downloaded boot images *and* all downloaded packages.  This would fix bug #998, at least for the upgrade case.

This may cause problems during pre-release testing, since there might be unsigned packages in the repos, but causing errors for unsigned packages is kind of the whole point here, right?

Comment 56 Need Real Name 2013-09-11 21:25:24 UTC
It was a blocker. Then someone removed it.

I'm not interested any more. I just don't think Bugzilla is useful for much Fedora-related this days.

Comment 57 chris 2013-09-12 01:27:05 UTC
(In reply to Will Woods from comment #55)

A few questions I would appreciate your responses for: 

1- So does that mean that Anaconda AND Fedup are verifying packages from this point on since your fix has been incorporated? (I'm totally fine with it giving an error for unsigned packages - not such a problem compared to the alternative if you ask me)

2- During an initial install of Fedora, are the packages downloaded through an encrypted/verified channel?

3- Are there still issues with using Fedup with a LUKS encrypted system?


TIA

Comment 58 Will Woods 2013-09-12 18:00:37 UTC
1) No, this means *fedup* will be verifying packages. For anaconda, see bug #998.

2) No, but it doesn't matter because all the data you download *is* verified. You don't have to trust the server; if the signature is good, the package is genuine and hasn't been tampered with. No matter where it came from.

3) Not that I know of.

Comment 59 Adam Williamson 2013-09-18 04:43:31 UTC
@56: it was proposed as a blocker, but rejected. A bug is not a blocker until it is proposed *and accepted*. https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process

Comment 60 Kevin Fenzi 2013-09-26 04:06:44 UTC
So, with the Fedora 20 Alpha release we did ship a .treeinfo.signed file: 

http://dl.fedoraproject.org/pub/fedora/linux/releases/test/20-Alpha/Fedora/x86_64/os/.treeinfo.signed
(for example)

Will: How can we test this out and confirm things? Are you planning a f19 updates-testing update with the checking enabled? Or a side repo with the package? Or should folks check out from git to test? 

Also, when we enable bodhi for a new branched release (at alpha change freeze time) all packages are signed. The mirrormanager fedora-install-N mapping only first appears at alpha release, so there shouldn't be any problems with unsigned packages at that point/using that instrepo.

Comment 61 Will Woods 2013-12-06 20:43:07 UTC
fedup-0.8.0 will validate downloaded RPMs and boot images (using .treeinfo.signed), *if* the keys are present at:
  
  /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$newversion-$basearch

fedora-release-19-5 has F20 keys at that path, so F19->F20 upgrades work fine.

fedora-release-18-1 has the F19 keys in the wrong place and no F20 keys, so upgrades from F18 will need '--nogpgcheck' until that gets fixed. But that's a separate bug.

Comment 62 Fedora Update System 2013-12-11 22:11:35 UTC
fedup-0.8.0-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc19

Comment 63 Fedora Update System 2013-12-11 22:19:55 UTC
fedup-0.8.0-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc20

Comment 64 Fedora Update System 2013-12-11 22:21:49 UTC
fedup-0.8.0-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc18

Comment 65 Fedora Update System 2013-12-13 05:08:54 UTC
Package fedup-0.8.0-3.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.8.0-3.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23316/fedup-0.8.0-3.fc19
then log in and leave karma (feedback).

Comment 66 Fedora Update System 2013-12-19 07:21:07 UTC
fedup-0.8.0-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 67 Fedora Update System 2013-12-20 01:43:54 UTC
fedup-0.8.0-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 68 Fedora Update System 2013-12-22 05:34:02 UTC
fedup-0.8.0-3.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.