Signing is not yet enabled on CI. Currently blocker is that we need to get the SRE hosted signer up.
I ran the signer again, all current releases should be signed by 11:05am
Also, this has nothing to do with GA signing. Nightlies will not be signed. We will only sign GA content.
Updating title
An option is for us to have ART sign rc.4 and rc.5. We should be able to do that now.
@mike - Can you use the latest RC to test upgrades. Looks like that seem to be working. Signing aspect is different as you already aware.
Have a try on path 4.1.0-rc.4 to 4.1.0-rc.5, still need "--force". scenario1: without "--force",fail [root@preserve-jliu-worker 20190523_15261]# ./oc adm upgrade --to=4.1.0-rc.5 Updating to 4.1.0-rc.5 [root@preserve-jliu-worker 20190523_15261]# ./oc adm upgrade info: An upgrade is in progress. Unable to apply 4.1.0-rc.5: the image may not be safe to use Updates: VERSION IMAGE 4.1.0-rc.5 registry.svc.ci.openshift.org/ocp/release@sha256:dc67ad5edd91ca48402309fe0629593e5ae3333435ef8d0bc52c2b62ca725021 scenario2: with "--force" succeed. [root@preserve-jliu-worker 20190523_15261]# ./oc adm upgrade --to=4.1.0-rc.5 --force Updating to 4.1.0-rc.5 [root@preserve-jliu-worker 20190523_15261]# ./oc adm upgrade info: An upgrade is in progress. Working towards 4.1.0-rc.5: downloading update Updates: VERSION IMAGE 4.1.0-rc.5 registry.svc.ci.openshift.org/ocp/release@sha256:dc67ad5edd91ca48402309fe0629593e5ae3333435ef8d0bc52c2b62ca725021 [root@preserve-jliu-worker 20190523_15261]# ./oc adm upgrade Cluster version is 4.1.0-rc.5
Signatures are now posted! https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/ 1dabe42b5c94841fd8736d8f3a80afeaf5f5ad3833cef8d304c419a97b0efbc3 is for rc.6 7e1e73c66702daa39223b3e6dd2cf5e15c057ef30c988256f55fae27448c3b01 is for rc.7 This is following the format prescribed in the update keys operator: https://github.com/openshift/cluster-update-keys/blob/master/manifests.rhel/0000_90_cluster-update-keys_configmap.yaml#L5 There are now signed sha256sum.txt.sig files in the rc.6 and rc.7 client tool directories: https://mirror.openshift.com/pub/openshift-v4/clients/ocp/ You can verify these were signed with the 'beta2' key by downloading each file and then running 'gpg --verify <the_file>'. You will see a header similar to the following: > gpg: Signature made Thu 30 May 2019 06:21:48 PM CDT using RSA key ID F21541EB But with different dates depending on the artifact. This matches the official published key id posted on the Red Hat Product Signing Keys page: https://access.redhat.com/security/team/key > 4096R/938a80caf21541eb (2009-02-24): > Red Hat, Inc. (beta key 2) <security>
Attempted an upgrade of an rc6 cluster installed before the signatures were posted (to simulate beta 6 users who installed some time ago) to rc7 after changes in comment 12 were made. Still seeing "image may not be safe to use": [root@ip-172-31-53-199 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-rc.6 True False 8h Cluster version is 4.1.0-rc.6 [root@ip-172-31-53-199 ~]# oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 Updating to release image quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 [root@ip-172-31-53-199 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version True True 10s Unable to apply quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7: the image may not be safe to use
Installed a cluster after rc6 was signed and tried an upgrade to signed rc7 and the result was the same as comment 15: [root@ip-172-31-53-199 ~]# oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 Updating to release image quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 [root@ip-172-31-53-199 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version True True 5s Unable to apply quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7: the image may not be safe to use
openshift/cluster-update-keys/pull/11/12 contain at least part of the required fixes. They're attached to this BZ now. One has not been merged yet.
Verified using 4.1.0.rc7, 4.1.0.rc8 and 4.1.0.rc9. - rc7 and rc9 are both publicly signed and are valid targets of upgrades and downgrades - rc8 and rc9 have the right key in their internal configmap to use during upgrade with the public keys The tested upgrade/downgrade paths that work without --force are: upgrade rc8 -> rc9 downgrade rc9 or rc8 -> rc7 All of these worked using these builds. Note that no other builds can upgrade to rc9 without --force.
1) 4.1.0-rc.8 -> 4.1.0-rc.9 -> 4.1.0-rc.7: upgrade first then downgrade, the downgrade path 4.1.0-rc.9 -> 4.1.0-rc.7 doesn't work without '--force' option, it has errors $ oc adm upgrade info: An upgrade is in progress. Unable to apply registry.svc.ci.openshift.org/ocp/release:4.1.0-rc.7: the image may not be safe to use Updates: VERSION IMAGE 4.1.0-0.nightly-2019-05-31-174150 registry.svc.ci.openshift.org/ocp/release@sha256:ba48e4781a7c7327fd5a155744b467019f10ba1be0b2d3bd59726afb1f898295 4.1.0-0.nightly-2019-05-29-220142 registry.svc.ci.openshift.org/ocp/release@sha256:bcd33081bc69c146bcad9f31d8a236a03996822b8e052640ccf15d78666df759 2) 4.1.0-rc.9 -> 4.1.0-rc.8 doesn't work without '--force' option, it has error $ oc adm upgrade info: An upgrade is in progress. Unable to apply registry.svc.ci.openshift.org/ocp/release:4.1.0-rc.8: the image may not be safe to use Updates: VERSION IMAGE 4.1.0-0.nightly-2019-05-31-174150 registry.svc.ci.openshift.org/ocp/release@sha256:ba48e4781a7c7327fd5a155744b467019f10ba1be0b2d3bd59726afb1f898295 4.1.0-0.nightly-2019-05-29-220142 registry.svc.ci.openshift.org/ocp/release@sha256:bcd33081bc69c146bcad9f31d8a236a03996822b8e052640ccf15d78666df759 3) 4.1.0-rc.9 -> 4.1.0-rc.7 direct downgrade doesn't work without '--force' option, it has error $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version True True 8m7s Unable to apply registry.svc.ci.openshift.org/ocp/release:4.1.0-rc.7: the image may not be safe to use $ oc version Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0-201905191700+7bd2e5b-dirty", GitCommit:"7bd2e5b", GitTreeState:"dirty", BuildDate:"2019-05-19T23:52:43Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Upgrade from rc.8 to rc.9 successful. oc adm upgrade --to=4.1.0-rc.9 Upgrade not safe error from rc.9 to rc.7 if attempted with tag as below: oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 Uppgrade successful from rc.9 to rc.7 with sha image id as below. $ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:7e1e73c66702daa39223b3e6dd2cf5e15c057ef30c988256f55fae27448c3b01 Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:7e1e73c66702daa39223b3e6dd2cf5e15c057ef30c988256f55fae27448c3b01 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-rc.7 True False 5m38s Cluster version is 4.1.0-rc.7 $ oc version Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0-201905021432+7903b3e-dirty", GitCommit:"7903b3e", GitTreeState:"dirty", BuildDate:"2019-05-02T19:09:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Same behaviour while upgrading from rc.8 to rc.7. With tag, it reports "image not safe. With image sha256 ID, it is successful. $ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 Updating to release image quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version True True 2s Unable to apply quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7: the image may not be safe to use $ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:7e1e73c66702daa39223b3e6dd2cf5e15c057ef30c988256f55fae27448c3b01 Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:7e1e73c66702daa39223b3e6dd2cf5e15c057ef30c988256f55fae27448c3b01 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-rc.7 True False 27s Cluster version is 4.1.0-rc.7
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1382