Bug 1393545

fedora-repos has all of the gpg keys ever used in Fedora. 

how does the verification work?
can a commit be verified before a update is pulled down?
do we need to have all possible keys available? or only the one for the commit.

From 'man ostree':

GPG VERIFICATION
       OSTree supports signing commits with GPG. Operations on the system repository by default use keyring files in /usr/share/ostree/trusted.gpg.d. Any public key in a keyring file in that directory will be trusted by the client. No private keys should be present in this directory.

       In addition to the system repository, OSTree supports a per-remote remotename.trustedkeys.gpg file stored in the toplevel of the repository (alongside objects/ and such). This is particularly useful when downloading content that may not be fully trusted (e.g. you want to inspect it but not deploy it as an OS), or use it for containers. This file is written via ostree remote add --gpg-import.

> can a commit be verified before a update is pulled down?

You can pull the commit metadata, which includes the signature, and verify that commit before pulling all of the file objects.  

In the example below, I have pulled down all the commit history for the branch, but not the file objects.  Then I use 'ostree show' to verify the signature of the HEAD-1 commit.

# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.40 (2016-11-08 10:20:02)
        Commit: b02be04acde2a04a96d9e969edf3b73753967c06811fc51d65c2b2b4985dff4a
        OSName: fedora-atomic
# ostree pull --commit-metadata-only --depth=-1 fedora-atomic:fedora-atomic/25/x86_64/docker-host

80 metadata, 0 content objects fetched; 14 KiB transferred in 31 seconds
# ostree log fedora-atomic/25/x86_64/docker-host
commit b02be04acde2a04a96d9e969edf3b73753967c06811fc51d65c2b2b4985dff4a
Date:  2016-11-08 10:20:02 +0000
Version: 25.40
(no subject)

commit c8379fb28b7102367d1f4dbdb8f8082628d098aba8969f4c98ee7677b068a2c7
Date:  2016-11-06 10:22:13 +0000
Version: 25.39
(no subject)

commit 7f19ed0a253b9023dd1d41acd9043deadb08a70d379f424b14fff61703f47827
Date:  2016-11-01 10:26:04 +0000
Version: 25.38
(no subject)

...
# ostree show c8379fb28b7102367d1f4dbdb8f8082628d098aba8969f4c98ee7677b068a2c7
commit c8379fb28b7102367d1f4dbdb8f8082628d098aba8969f4c98ee7677b068a2c7
Date:  2016-11-06 10:22:13 +0000
Version: 25.39
(no subject)

Found 1 signature:

  Signature made Sun 06 Nov 2016 10:22:22 AM UTC using RSA key ID 4089D8F2FDB19C98
  Can't check signature: public key not found

> do we need to have all possible keys available? or only the one for the
> commit.

We would only need a keyring that has the GPG key used to sign the commit.

See https://github.com/projectatomic/centos-release-atomic-host-devel/blob/master/centos-devel-atomic-host-release.spec#L88

The problem is right now doing that would require fedora-repos to depend on ostree.  (Although just for file path ownership, which honestly, I'm not sure if we really care right now if someone removes fedora-repos and /usr/share/ostree/trusted.gpg.d stays around).

(In reply to Colin Walters from comment #3)
> See
> https://github.com/projectatomic/centos-release-atomic-host-devel/blob/
> master/centos-devel-atomic-host-release.spec#L88
> 
> The problem is right now doing that would require fedora-repos to depend on
> ostree.  (Although just for file path ownership, which honestly, I'm not
> sure if we really care right now if someone removes fedora-repos and
> /usr/share/ostree/trusted.gpg.d stays around).

Is there a reason that couldn't just be moved into the `filesystem` package?

A root problem is the split between the fedora-release packages and fedora-repos only works for rpms.  If we wanted default Docker registries or ostree remotes, there's no mechanism to have those only apply to one product.

(In reply to Colin Walters from comment #5)
> A root problem is the split between the fedora-release packages and
> fedora-repos only works for rpms.  If we wanted default Docker registries or
> ostree remotes, there's no mechanism to have those only apply to one product.

Sure there is. Each of the Editions has their own fedora-release-$EDITION subpackage that we could add a Requires: to.

Here's a patch I started on which should illustrate what I'm talking about; there may be a better approach, but:

--- a/fedora-release.spec
+++ b/fedora-release.spec
@@ -28,6 +28,7 @@ Obsoletes:      fedora-release-standard < 22-0.8
 
 
 Requires:       fedora-repos(%{version})
+BuildRequires:  fedora-repos(%{version})
 BuildArch:      noarch
 
 %description
@@ -43,6 +44,7 @@ Requires:       fedora-release = %{version}-%{release}
 %description atomichost
 Provides a base package for Fedora Atomic Host-specific configuration files to
 depend on.
+Requires:       ostree
 
 %package cloud
 Summary:        Base package for Fedora Cloud-specific default configurations
@@ -218,6 +220,13 @@ install -m 0644 org.gnome.shell.gschema.override $RPM_BUILD_ROOT%{_datadir}/glib
 mkdir -p $RPM_BUILD_ROOT/%{_prefix}/sbin/
 install -m 0755 convert-to-edition $RPM_BUILD_ROOT/%{_prefix}/sbin/
 
+# This should likely go into fedora-repos, though that'd be more awkward
+# as it'd require a dependency on ostree there, or a subpackage
+mkdir -p -m 755 %{buildroot}/%{_datadir}/ostree/trusted.gpg.d
+for file in /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-%{dist_version}-primary; do
+    gpg --dearmor < $file > %{buildroot}/%{_datadir}/ostree/trusted.gpg.d/$(basename $file).gpg
+done
+
 %post -p <lua>
 %include %{_sourcedir}/convert-to-edition.lua
 -- On initial installation, we'll at least temporarily put the non-product
@@ -316,6 +325,7 @@ glib-compile-schemas %{_datadir}/glib-2.0/schemas &> /dev/null || :
 %{!?_licensedir:%global license %%doc}
 %license LICENSE
 %config %attr(0644,root,root) /usr/lib/os.release.d/os-release-atomichost
+%{_datadir}/ostree/trusted.gpg.d/*
 
 
 %files cloud

(In reply to Colin Walters from comment #7)
> Here's a patch I started on which should illustrate what I'm talking about;
> there may be a better approach, but:
> 
> --- a/fedora-release.spec
> +++ b/fedora-release.spec
> @@ -28,6 +28,7 @@ Obsoletes:      fedora-release-standard < 22-0.8
>  
>  
>  Requires:       fedora-repos(%{version})
> +BuildRequires:  fedora-repos(%{version})
This is a non starter, you have a chicken and egg problem fedora-repos and fedora-release both Require each other and require that they be the same version, it  could be bootstrapped though 

>  BuildArch:      noarch
>  
>  %description
> @@ -43,6 +44,7 @@ Requires:       fedora-release = %{version}-%{release}
>  %description atomichost
>  Provides a base package for Fedora Atomic Host-specific configuration files
> to
>  depend on.
> +Requires:       ostree
>  
>  %package cloud
>  Summary:        Base package for Fedora Cloud-specific default
> configurations
> @@ -218,6 +220,13 @@ install -m 0644 org.gnome.shell.gschema.override
> $RPM_BUILD_ROOT%{_datadir}/glib
>  mkdir -p $RPM_BUILD_ROOT/%{_prefix}/sbin/
>  install -m 0755 convert-to-edition $RPM_BUILD_ROOT/%{_prefix}/sbin/
>  
> +# This should likely go into fedora-repos, though that'd be more awkward
> +# as it'd require a dependency on ostree there, or a subpackage
> +mkdir -p -m 755 %{buildroot}/%{_datadir}/ostree/trusted.gpg.d
> +for file in /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-%{dist_version}-primary; do
> +    gpg --dearmor < $file >
> %{buildroot}/%{_datadir}/ostree/trusted.gpg.d/$(basename $file).gpg
> +done
> +
>  %post -p <lua>
>  %include %{_sourcedir}/convert-to-edition.lua
>  -- On initial installation, we'll at least temporarily put the non-product
> @@ -316,6 +325,7 @@ glib-compile-schemas %{_datadir}/glib-2.0/schemas &>
> /dev/null || :
>  %{!?_licensedir:%global license %%doc}
>  %license LICENSE
>  %config %attr(0644,root,root) /usr/lib/os.release.d/os-release-atomichost
> +%{_datadir}/ostree/trusted.gpg.d/*
>  
>  
>  %files cloud
the problem should be solved entirely in fedora-repos.  we could easily make a fedora-repos-atomic sub package. that has the bits in it. I just need to know which of the gpog keys we need to pull into the keyring

Conceptually I think it should be fedora-repos-ostree, and we might as well also take the next step of including the remote config there, like we do for CentOS.  As far as the keys, each branch should be signed by the same key that's signing RPMs, right?

yes each release has its own keys. It comes down to is there value in having the f26 key installed on a f25 system, or f25 on f24 to facilitate transitioning from one release to the next. or is it all self contained within a release and we only need the key for the release in question.

Unfortunately, right now each release is a separate ostree repository.  That means we need separate remotes, and hence having separate keyrings is forced (if we use per-remote keyrings).

If we choose the /usr/share/ostree/trusted.gpg.d route, then indeed it'd be useful to have future release keyrings in there.

But we should try to do the per-remote keyring I think.  Hm...this would likely be a lot simpler if we could refer to an existing file path in the remote spec, then we could just point it at the rpm-md key.

Filed that as https://github.com/ostreedev/ostree/issues/573

I started working on this, but it blocks on https://pagure.io/releng/issue/6598

Ok, so now we'd need to rebuild rpm-ostree with the current ostree, so we pick up `ostree_repo_verify_commit_for_remote()`.  See https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2016.13/1.fc25/data/logs/x86_64/build.log for where it's too old.

That said I'd like to try to bump to 2017.1.  But that blocks on https://pagure.io/releng/issue/6602

This should be ready for more work now.  I'd like to finally add the remote config to a package so that we can change it on upgrades.

A conceptual issue remains in that fedora-repos is currently RPM specific, and we still need to have an ostree remote config.  If we add /etc/ostree/remotes.d/fedora-atomic.conf into this package, it'd show up in the container base image, Server etc.  That doesn't seem like a huge problem, but we might as well create a subpackage fedora-repos-ostree.  Agree?

Alternatively, create fedora-repos-atomic, which could have a future configuration for registry.fedoraproject.org for container images too.  Then it could be inherited by Server/Workstation too.  Yes, the ostree remote would show up there but I doubt anyone's going to really care about a new file in either of those; they're both already huge.

I don't mind doing the work on this to create a `fedora-repos-atomic` subpackage if that's fine with everyone else.  Open to other ideas as well.

i'm +1 to another rpm but maybe would prefer we don't add container image registries to it. Seems like it would be best if we don't intertwine the two. i.e. there is a case where you want container registry information on a non-atomic host system. WDYT?

If we want to create a `fedora-repos-ostree` or `fedora-repos-atomic` package then let's create a work item for it on pagure.io/atomic-wg.

I had a attempt at it with https://pagure.io/fork/ausil/fedora-repos/c/4f2e301032b7766e454a4f2e4279ae82dd8e34ab?branch=ostree-keys

Hi @ausil, thanks for working on this.  However there's a bit of confusion in this bug; originally we were talking about needing a separate *keys* subpackage. 

But in https://bugzilla.redhat.com/show_bug.cgi?id=1393545#c12 everything changed, since ostree learned how to use the keys from /etc/pki/rpm-gpg like the libdnf side does.

Then the conversation turned to having the *ostree remote files* in a package, which can just reference the rpm-gpg keys.

I think a variant of your patch which adds a -repos subpackage and includes content like:

```
$ cat /etc/ostree/remotes.d/fedora-atomic-27.conf 
[remote "fedora-atomic-27"]
url=https://kojipkgs.fedoraproject.org/atomic/rawhide/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary
```

is what we're talking about.

However...I have to say again everything would be dramatically simpler if there was just one ostree repo, or at least one for FAH and one for FWS.  But that does run up against the key-per-release issue.

(In reply to Colin Walters from comment #20)

> But in https://bugzilla.redhat.com/show_bug.cgi?id=1393545#c12 everything changed, since ostree learned how to use the keys from /etc/pki/rpm-gpg like the libdnf side does.

So we don't need this work at all now, right? I like the gpgkeypath approach personally. 

> But that does run up against the key-per-release issue.

doesn't gpgkeypath=key1,key2 work?

That's https://github.com/ostreedev/ostree/issues/773
Which would be pretty easy to implement, and I'm happy to do so but we need to be sure how we're going to use it first.

Say we had a unified repo, it might look like this:

```
[remote "fedora-atomic"]
url=https://kojipkgs.fedoraproject.org/atomic/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary,/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary
```

But what happens when f28 comes out?  Do we rotate that key list, and it'd end up in the new tree?  But the problem is that the way ostree works we'll be reading the remote config from the *old* tree.  We could just always have the file specify "N-1,N,N+1" keys?

Or perhaps it's simpler to have something like a directory:
/etc/pki/rpm-gpg/fedora-supported/
  RPM-GPG-KEY-fedora-26-primary
  RPM-GPG-KEY-fedora-27-primary

Then we could say:
gpgkeypath=/etc/pki/rpm-gpg/fedora-supported

Note that currently libdnf actually auto-imports everything in /etc/pki/rpm-gpg...which means it trusts RPM-GPG-KEY-fedora-10-primary too which it likely shouldn't.

(In reply to Colin Walters from comment #22)
> 
> Say we had a unified repo, it might look like this:
> 
> ```
> [remote "fedora-atomic"]
> url=https://kojipkgs.fedoraproject.org/atomic/
> gpg-verify=true
> gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary,/etc/pki/rpm-gpg/
> RPM-GPG-KEY-fedora-27-primary
> ```

Exactly.

> 
> But what happens when f28 comes out?  Do we rotate that key list, and it'd
> end up in the new tree?  But the problem is that the way ostree works we'll
> be reading the remote config from the *old* tree.  We could just always have
> the file specify "N-1,N,N+1" keys?

Why not just N,N+1 ?

basically if we start delivering the repo file via an rpm then we have
this:

I'm on Fedora 26 /etc/ostree/remotes.d/fedora-atomic.conf is provided
by rpm foo-fc26.rpm. and has gpgkeypath=f26key,f27key. 

rpm-ostree upgrade from f26 to f27. Now /etc/ostree/remotes.d/fedora-atomic.conf
is provided  by foo-fc27.rpm and has gpgkeypath=f27key,f28key.

Thoughts?

I think that'd work, yep.

What do you think, Dennis?

We probably should make a single key that we sign ostree commits with, like we have a single key for all modules.

This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

(In reply to Dennis Gilmore from comment #26)
> We probably should make a single key that we sign ostree commits with, like
> we have a single key for all modules.

Not sure how likely this is to happen anytime soon (or even if it's a good idea). Can we go with proposal in https://bugzilla.redhat.com/show_bug.cgi?id=1393545#c23 until we figure that out?

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.