Bug 1705984
| Summary: | Image Signature Verification is no longer working, latest signatures are not being downloaded. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Pedro Amoedo <pamoedom> |
| Component: | ImageStreams | Assignee: | Oleg Bulatov <obulatov> |
| Status: | CLOSED WONTFIX | QA Contact: | Wenjing Zheng <wzheng> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.11.0 | CC: | aos-bugs, bparees, dornelas, dwalsh, gblomqui, jbruzl, jluza, jokerman, jshepherd, mitr, mmccomas, mpatel, obulatov, rsandu, sgraf, wzheng |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-16 13:54:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1722568, 1722780, 1748812 | ||
| Bug Blocks: | 1726784, 1748134 | ||
~~~
# cat /etc/containers/registries.d/registry.redhat.io.yaml
docker:
registry.redhat.io:
sigstore: https://registry.redhat.io/containers/sigstore
~~~
Sigstore for registry.redhat.io is at url https://access.redhat.com/webassets/docker/content/sigstore
(In reply to Stanislav Graf from comment #1) > ~~~ > # cat /etc/containers/registries.d/registry.redhat.io.yaml > docker: > registry.redhat.io: > sigstore: https://registry.redhat.io/containers/sigstore > ~~~ > > Sigstore for registry.redhat.io is at url > https://access.redhat.com/webassets/docker/content/sigstore Hi Stanislav, thanks for your quick reply, I've extracted that from official documentation [1]. However, this is irrelevant, I have already tried with both URLs and even with legacy registry.access.redhat.com and the problem is still present. [1] - https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore Thanks for your report. (In reply to Pedro Amoedo from comment #0) > Version-Release number of selected component (if applicable): > OCP 3.x Please be precise. > 1. Set the proper sigstore as documented [1] within > /etc/containers/registries.d/, for example: (I confirm that these entries work for Skopeo.) > 2. Import any latest image, for example: > > # oc import-image registry.redhat.io/rhel7/etcd:latest --confirm > # oc import-image registry.access.redhat.com/rhel7/etcd:latest --confirm > > 3. Check istag and the image signatures presence (if any): > > # oc get istag > NAME DOCKER REF > UPDATED IMAGENAME > etcd:latest > registry.redhat.io/rhel7/etcd@sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 9 seconds > ago sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > # oc describe istag etcd:latest | grep -A3 Signatures > (void) If I understand OpenShift’s https://github.com/openshift/origin/blob/6324991f123ca5fabdceded15dfde8445bce7a81/pkg/image/importer/importer.go , (oc import-image) only causes the manifest and blobs to be imported. Signatures are imported separately by https://github.com/openshift/origin/blob/c68d654128cc4ec776a183d20db1d24b51db07d5/pkg/image/controller/signature/container_image_downloader.go . Did you wait long enough for that controller to run? (I don’t really know what “long enough” means for that controller, but 9 seconds seems borderline at best). --- > Image signatures are no longer being downloaded for new registry.redhat.io > neither for the legacy registry.access.redhat.com, however, if you manually > download them with skopeo commands you can see that new images have more > signatures associated than previous versions: That’s expected. e.g. registry.access.redhat.com/rhel7/etcd:3.2.22-28 has signatures for - registry.access.redhat.com/rhel7/etcd:3.2.22 - registry.access.redhat.com/rhel7/etcd:3.2.22-28 - registry.access.redhat.com/rhel7/etcd:latest while the current etcd:latest (i.e. 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec ) has - [1] registry.access.redhat.com/rhel7/etcd:3.2.22 - [2] registry.redhat.io/rhel7/etcd:3.2.22 - [3] registry.access.redhat.com/rhel7/etcd:3.2.22-30 - [4] registry.redhat.io/rhel7/etcd:3.2.22-30 - [5] registry.access.redhat.com/rhel7/etcd:latest - [6] registry.redhat.io/rhel7/etcd:latest > Optionally, if you try to push this image to a registry and manually verify > the signatures, you'll see unexpected conrtadictory repository errors (those > same commands were working for me 2 weeks ago with previous image versions): > [ocp-ocr]# oc adm verify-image-signature > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > --expected-identity > registry.access.redhat.com/rhel7/etcd@sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release > image > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > identity is now confirmed (signed by GPG key "199E2F91FD431D51") [1] above > error verifying signature > sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@7a6699867797 > 8f6b9985e86304bede59 for image > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > (verification status will be removed): signature rejected: Signature for > identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted [2] above > image > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > identity is now confirmed (signed by GPG key "199E2F91FD431D51") [3] above > error verifying signature > sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@eb06e3c03f0b > 182972dbd9626500596a for image > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > (verification status will be removed): signature rejected: Signature for > identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted [4] above (… and [5], [6] snipped) This works as expected AFAICT; with --expected-identity registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (using a digest reference, which means the tag is ignored as long as the digest matches), signatures for registry.access.redhat.com/rhel7/etcd:* match the --expected-identity, and signatures for redhat.io/rhel7/etcd:* don’t match the --expected identity. Admittedly the output of the command could be more readable. --- > There is also an error in the documentation related with "oc adm > verify-image-signature" command, it states that "expected-identity" should > be extracted from "Docker Pull Spec" value from imagestream description, but > once imported, that value will always be the local registry and not the > original identity that should be verified against the signature: > > $ oc adm verify-image-signature <image> --expected-identity=<pull_spec> > [--save] [options] Is this about this text in the help output? > and matching the provided expected identity > with the identity (pull spec) of the given image. That does not quite say “extract it from the imastream”, but, sure, being more precise wouldn’t hurt. BTW, ideally --expected-identity should also include the original tag, if the intent was to pull by tag (even if the tag is :latest), so that the tag in the signature it is not completely ignored. --- So, overall, this report contains: - Signature import controller not working, or maybe just delayed a bit after (oc import-image) (in which case documentation might need improving) - (oc adm verify-image-signature) works as expected AFAICT, improving the UI is worth considering - A documentation suggestion for (oc add verify-image-signature). All of this is primarily targeted in the OpenShift image-related code, reassigning there (but feel free to send this back to me if the import failures are a problem in the underlying c/image code). (In reply to Miloslav Trmač from comment #3) > Thanks for your report. > > (In reply to Pedro Amoedo from comment #0) > > Version-Release number of selected component (if applicable): > > OCP 3.x > Please be precise. All versions that I could try (3.7, 3.9, 3.10, 3.11), that's the reason of 3.x, apologies. > > > > 1. Set the proper sigstore as documented [1] within > > /etc/containers/registries.d/, for example: > (I confirm that these entries work for Skopeo.) > Yes, they manually work with Skopeo but not with "oc import-image" as oficially documented [1] and previously working 2 weeks ago. > > 2. Import any latest image, for example: > > > > # oc import-image registry.redhat.io/rhel7/etcd:latest --confirm > > # oc import-image registry.access.redhat.com/rhel7/etcd:latest --confirm > > > > 3. Check istag and the image signatures presence (if any): > > > > # oc get istag > > NAME DOCKER REF > > UPDATED IMAGENAME > > etcd:latest > > registry.redhat.io/rhel7/etcd@sha256: > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 9 seconds > > ago sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > > > # oc describe istag etcd:latest | grep -A3 Signatures > > (void) > > If I understand OpenShift’s > https://github.com/openshift/origin/blob/ > 6324991f123ca5fabdceded15dfde8445bce7a81/pkg/image/importer/importer.go , > (oc import-image) only causes the manifest and blobs to be imported. > > Signatures are imported separately by > https://github.com/openshift/origin/blob/ > c68d654128cc4ec776a183d20db1d24b51db07d5/pkg/image/controller/signature/ > container_image_downloader.go . Did you wait long enough for that controller > to run? (I don’t really know what “long enough” means for that controller, > but 9 seconds seems borderline at best). > Absolutely, I waited long time enough (days), signatures are not being downloaded as they were, and still are with older versions, as I have demonstrated with registry.access.redhat.com/rhel7/etcd:3.2.22-28. > --- > > > Image signatures are no longer being downloaded for new registry.redhat.io > > neither for the legacy registry.access.redhat.com, however, if you manually > > download them with skopeo commands you can see that new images have more > > signatures associated than previous versions: > That’s expected. e.g. registry.access.redhat.com/rhel7/etcd:3.2.22-28 has > signatures for > - registry.access.redhat.com/rhel7/etcd:3.2.22 > - registry.access.redhat.com/rhel7/etcd:3.2.22-28 > - registry.access.redhat.com/rhel7/etcd:latest > > while the current etcd:latest (i.e. > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec ) has > - [1] registry.access.redhat.com/rhel7/etcd:3.2.22 > - [2] registry.redhat.io/rhel7/etcd:3.2.22 > - [3] registry.access.redhat.com/rhel7/etcd:3.2.22-30 > - [4] registry.redhat.io/rhel7/etcd:3.2.22-30 > - [5] registry.access.redhat.com/rhel7/etcd:latest > - [6] registry.redhat.io/rhel7/etcd:latest > > > Optionally, if you try to push this image to a registry and manually verify > > the signatures, you'll see unexpected conrtadictory repository errors (those > > same commands were working for me 2 weeks ago with previous image versions): > > > [ocp-ocr]# oc adm verify-image-signature > > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > --expected-identity > > registry.access.redhat.com/rhel7/etcd@sha256: > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release > > image > > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > > identity is now confirmed (signed by GPG key "199E2F91FD431D51") > [1] above > > error verifying signature > > sha256: > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@7a6699867797 > > 8f6b9985e86304bede59 for image > > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > (verification status will be removed): signature rejected: Signature for > > identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted > [2] above > > image > > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > > identity is now confirmed (signed by GPG key "199E2F91FD431D51") > [3] above > > error verifying signature > > sha256: > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@eb06e3c03f0b > > 182972dbd9626500596a for image > > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > (verification status will be removed): signature rejected: Signature for > > identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted > [4] above > (… and [5], [6] snipped) > > This works as expected AFAICT; with --expected-identity > registry.access.redhat.com/rhel7/etcd@sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (using a > digest reference, which means the tag is ignored as long as the digest > matches), signatures for registry.access.redhat.com/rhel7/etcd:* match the > --expected-identity, and signatures for redhat.io/rhel7/etcd:* don’t match > the --expected identity. > > Admittedly the output of the command could be more readable. > When you use registry.access.redhat.com as "expected-identity" it fails pointing to redhat.io, but if you use redhat.io, it fails pointing to registry.access.redhat.com, how's that possible?, has the way of using "oc adm verify-image-signature" changed? As per documentation [2], the "expected-identity" should be extracted from "Docker Pull Spec" value from imagestream description, but once an image is imported, that value will always be the local registry and not the original identity that should be used to match with the proper "public-key". $ oc adm verify-image-signature <image> --expected-identity=<pull_spec> [--save] [options] > --- > > > There is also an error in the documentation related with "oc adm > > verify-image-signature" command, it states that "expected-identity" should > > be extracted from "Docker Pull Spec" value from imagestream description, but > > once imported, that value will always be the local registry and not the > > original identity that should be verified against the signature: > > > > $ oc adm verify-image-signature <image> --expected-identity=<pull_spec> > > [--save] [options] > > Is this about this text in the help output? > > and matching the provided expected identity > > with the identity (pull spec) of the given image. > > That does not quite say “extract it from the imastream”, but, sure, being > more precise wouldn’t hurt. BTW, ideally --expected-identity should also > include the original tag, if the intent was to pull by tag (even if the tag > is :latest), so that the tag in the signature it is not completely ignored. > Again, as per official documentation [2]: ~~~ The <pull_spec> can be found *by describing the image stream*. The <image> may be found by describing the image stream tag. See the following example command output. ~~~ > --- > > So, overall, this report contains: > - Signature import controller not working, or maybe just delayed a bit after > (oc import-image) (in which case documentation might need improving) > - (oc adm verify-image-signature) works as expected AFAICT, improving the UI > is worth considering > - A documentation suggestion for (oc add verify-image-signature). > > All of this is primarily targeted in the OpenShift image-related code, > reassigning there (but feel free to send this back to me if the import > failures are a problem in the underlying c/image code). Again, I can demonstrate a working example with old image version on which the signatures are being downloaded as expected, and even verified if needed using the digest. ~~~ # oc import-image rhel7/etcd:3.2.22-28 --from=registry.access.redhat.com/rhel7/etcd:3.2.22-28 --confirm The import completed successfully. Name: etcd Namespace: test Created: 2 minutes ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2019-05-07T09:05:58Z Docker Pull Spec: docker-registry.default.svc:5000/test/etcd Image Lookup: local=false Unique Images: 2 Tags: 2 latest tagged from registry.access.redhat.com/rhel7/etcd * registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 2 minutes ago 3.2.22-28 tagged from registry.access.redhat.com/rhel7/etcd:3.2.22-28 * registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Less than a second ago Image Name: etcd:3.2.22-28 Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Created: Less than a second ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.8MB (first layer 75.82MB, last binary layer 15.97MB) Image Created: 2 months ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-03-06T14:49:26.310415 com.redhat.build-host=cpt-0004.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=28 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-28 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker [root@ocp-1 ~]# oc describe istag etcd:3.2.22-28 Image Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Created: 9 seconds ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.8MB (first layer 75.82MB, last binary layer 15.97MB) Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@79b327ec1b3b48744d8d7a4b7f88b1f3322af19bf06bf93f8842fbf59eaf148d Type: AtomicImageV1 Status: Unverified Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@f0550dd7edbeddcb36b60644a4668acb1862018e539319ecf7c69101a05bfdb5 Type: AtomicImageV1 Status: Unverified Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@babd9cd80f0022e16fca51a90ee9a3ec0f8155321fa1d51477ae46081b0f9d34 Type: AtomicImageV1 Status: Unverified Image Created: 2 months ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-03-06T14:49:26.310415 com.redhat.build-host=cpt-0004.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=28 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-28 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker # oc adm verify-image-signature sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a --expected-identity registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") ~~~ With latest one, the signatures are not even present in order to verify them: ~~~ # oc import-image rhel7/etcd --from=registry.access.redhat.com/rhel7/etcd --confirm The import completed successfully. Name: etcd Namespace: test Created: Less than a second ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2019-05-07T09:03:54Z Docker Pull Spec: docker-registry.default.svc:5000/test/etcd Image Lookup: local=false Unique Images: 1 Tags: 1 latest tagged from registry.access.redhat.com/rhel7/etcd * registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Less than a second ago Image Name: etcd:latest Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: Less than a second ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.82MB (first layer 75.85MB, last binary layer 15.96MB) Image Created: 2 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker # oc describe istag etcd:latest Image Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: 36 seconds ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.82MB (first layer 75.85MB, last binary layer 15.96MB) Image Created: 2 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker ~~~ NOTE: "oc import-image" seems to ignore "--loglevel", is any other option to increase verbosity on this command output that can help us determine the root cause? [1] - https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore [2] - https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#verifying-image-signatures-using-openshift-cli Thanks and regards. (In reply to Pedro Amoedo from comment #4) > > > Image signatures are no longer being downloaded for new registry.redhat.io > > > neither for the legacy registry.access.redhat.com, however, if you manually > > > download them with skopeo commands you can see that new images have more > > > signatures associated than previous versions: > > That’s expected. e.g. registry.access.redhat.com/rhel7/etcd:3.2.22-28 has > > signatures for > > - registry.access.redhat.com/rhel7/etcd:3.2.22 > > - registry.access.redhat.com/rhel7/etcd:3.2.22-28 > > - registry.access.redhat.com/rhel7/etcd:latest > > > > while the current etcd:latest (i.e. > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec ) has > > - [1] registry.access.redhat.com/rhel7/etcd:3.2.22 > > - [2] registry.redhat.io/rhel7/etcd:3.2.22 > > - [3] registry.access.redhat.com/rhel7/etcd:3.2.22-30 > > - [4] registry.redhat.io/rhel7/etcd:3.2.22-30 > > - [5] registry.access.redhat.com/rhel7/etcd:latest > > - [6] registry.redhat.io/rhel7/etcd:latest > … > When you use registry.access.redhat.com as "expected-identity" it fails > pointing to redhat.io, but if you use redhat.io, it fails pointing to > registry.access.redhat.com, how's that possible? Well, because "redhat.io/rhel7/etcd" and "registry.access.redhat.com/rhel7/etcd" are literally different strings. It’s not surprising that this difference causes the “expected identity” to not match; it would be surprising if this difference was ignored without complaint (assuming default configuration, or the one hard-coded in (oc add verify-image-signature)). > has the way of using "oc adm verify-image-signature" changed? Not to my knowledge; the existence of redhat.io, and the signatures being created for redhat.io, are fairly new, though. > As per documentation [2], the > "expected-identity" should be extracted from "Docker Pull Spec" value from > imagestream description, but once an image is imported, that value will > always be the local registry and not the original identity that should be > used to match with the proper "public-key". Yes, this is, _in general_, incorrect (and it has always been incorrect): the expected identity must match the identity used when creating the signature (which can be redhat.com, isv.example.com, but it could also be the in-cluster registry, exactly the way the documentation says). (In reply to Miloslav Trmač from comment #9) > (In reply to Pedro Amoedo from comment #4) > > > > Image signatures are no longer being downloaded for new registry.redhat.io > > > > neither for the legacy registry.access.redhat.com, however, if you manually > > > > download them with skopeo commands you can see that new images have more > > > > signatures associated than previous versions: > > > That’s expected. e.g. registry.access.redhat.com/rhel7/etcd:3.2.22-28 has > > > signatures for > > > - registry.access.redhat.com/rhel7/etcd:3.2.22 > > > - registry.access.redhat.com/rhel7/etcd:3.2.22-28 > > > - registry.access.redhat.com/rhel7/etcd:latest > > > > > > while the current etcd:latest (i.e. > > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec ) has > > > - [1] registry.access.redhat.com/rhel7/etcd:3.2.22 > > > - [2] registry.redhat.io/rhel7/etcd:3.2.22 > > > - [3] registry.access.redhat.com/rhel7/etcd:3.2.22-30 > > > - [4] registry.redhat.io/rhel7/etcd:3.2.22-30 > > > - [5] registry.access.redhat.com/rhel7/etcd:latest > > > - [6] registry.redhat.io/rhel7/etcd:latest > > … > > When you use registry.access.redhat.com as "expected-identity" it fails > > pointing to redhat.io, but if you use redhat.io, it fails pointing to > > registry.access.redhat.com, how's that possible? > > Well, because "redhat.io/rhel7/etcd" and > "registry.access.redhat.com/rhel7/etcd" are literally different strings. > It’s not surprising that this difference causes the “expected identity” to > not match; it would be surprising if this difference was ignored without > complaint (assuming default configuration, or the one hard-coded in (oc add > verify-image-signature)). > > > has the way of using "oc adm verify-image-signature" changed? > > Not to my knowledge; the existence of redhat.io, and the signatures being > created for redhat.io, are fairly new, though. > > > > As per documentation [2], the > > "expected-identity" should be extracted from "Docker Pull Spec" value from > > imagestream description, but once an image is imported, that value will > > always be the local registry and not the original identity that should be > > used to match with the proper "public-key". > > Yes, this is, _in general_, incorrect (and it has always been incorrect): > the expected identity must match the identity used when creating the > signature (which can be redhat.com, isv.example.com, but it could also be > the in-cluster registry, exactly the way the documentation says). Thanks Miloslav, you have given me the signature relationship that I needed to confirm my theory: If we remove the extra non-necessary signatures that come from other registry (in this case the ones coming from legacy registry.access.redhat.com) and manually push the image only with the registry.redhat.io signatures, the "oc adm verify-image-signature" works again: ~~~ [pamoedo@p50 etcd_latest_test] $ ls 96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 manifest.json signature-4 a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 signature-1 signature-5 d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 signature-2 signature-6 d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 signature-3 version [pamoedo@p50 etcd_latest_test] $ rm signature-1 signature-3 signature-5 [pamoedo@p50 etcd_latest_test] $ mv signature-2 signature-1 [pamoedo@p50 etcd_latest_test] $ mv signature-4 signature-2 [pamoedo@p50 etcd_latest_test] $ mv signature-6 signature-3 [pamoedo@p50 etcd_latest_test] $ skopeo copy --dest-tls-verify=false --dest-creds openshift:<token> dir:/home/pamoedo/mirror/etcd_latest_test atomic:docker-registry-default.192.168.0.105.nip.io/imported/etcd:latest Getting image source signatures Checking if image destination supports signatures Skipping fetch of repeat blob sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 Skipping fetch of repeat blob sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 Skipping fetch of repeat blob sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 Copying config sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 0 B / 4.63 KB [------------------------------------------------------------] 0s Writing manifest to image destination Storing signatures [root@ocp-ocr ~]# oc describe istag etcd:latest Image Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: 2 minutes ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.82 MB (first layer 75.85 MB, last binary layer 15.96 MB) Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@84e28e88b7ae4540e8795d6039244e49 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@0716d6dc887985c87440ce2af6a096ba Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@63d191698330a86d2a2b3338ed5c64fd Type: atomic Status: Unverified Image Created: 2 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker [root@ocp-ocr ~]# oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity registry.redhat.io/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") ~~~ This confirms that the addition of new registry.redhat.io signatures into the same sigstore (https://access.redhat.com/webassets/docker/content/sigstore), even if you point to new the one as stated in the documentation (https://registry.redhat.io/containers/sigstore), has caused an unexpected problem at least on 2 oc commands: 1) "oc import-image" no longer downloads the signatures because, somehow, the image-signature-import controller is not able to determine which signatures are the proper ones, I suposse that apart from regular skopeo copy command (which downloads all signatures), it probably makes some underlying docker signature-verification that fails. NOTE: this command doesn't accept "loglevel" global-option in order to further investigate, we'll probably need to get the logs from controllers, right? 2) "oc adm verify-image-signature" is not able to properly verify 2 different identities for the same image, it was designed to only provide one "--expected-identity" value, so verification will "succeed" only on half of the signatures but fail on the other half, which makes not possible the use of "--save" option to change the status of those signatures to Verified. Kind Regards. (In reply to Pedro Amoedo from comment #11) > Thanks Miloslav, you have given me the signature relationship that I needed > to confirm my theory: > > If we remove the extra non-necessary signatures that come from other > registry (in this case the ones coming from legacy > registry.access.redhat.com) and manually push the image only with the > registry.redhat.io signatures, the "oc adm verify-image-signature" works > again: AFAICS it has “worked” before as well; sure, it was noisy about half of the signatures being rejected. (Comments below are based only on reading the code as linked; I don’t nearly understand enough of the OpenShift code base to be authoritative, and have basically zero practical experience. Still, actually gathering logs would seem to be more productive than forming hypotheses from limited data at this point.) > 1) "oc import-image" no longer downloads the signatures because, somehow, > the image-signature-import controller is not able to determine which > signatures are the proper ones, You haven’t show anything to suggest that “not able to determine” is the case. > I suposse that apart from regular skopeo > copy command (which downloads all signatures), it probably makes some > underlying docker signature-verification that fails. That seems unlikely; as far as I understand the code, it simply downloads all signatures, without doing any kind of verification at all. > 2) "oc adm verify-image-signature" is not able to properly verify 2 > different identities for the same image, it was designed to only provide one > "--expected-identity" value, so verification will "succeed" only on half of > the signatures but fail on the other half, which makes not possible the use > of "--save" option to change the status of those signatures to Verified. Looking at the code, nothing prevents using --save if some of the verifications fail; the “verification status will be removed” text in the failures quoted earlier refers to what would happen with --save. (It is possible that it does actually fail, (I don’t quite see how the status removal code path works), but you haven’t shown any output like that.) Sure, the outcome of that could be that only half of the signatures are marked as “verified” in the UI, but that should not be a _big_ problem - one of the major design points of the multiple-signature support is that it is always safe and harmless to add another, valid or invalid, signature. (Of course, it may be the case that shipping OpenShift with an UI that shows half of the signatures as rejected in the default configuration gives bad impression, and that you want all green checkmarks or something; a code to support marking signatures as verified in such combinations can of course be written — but be _very_ careful about the semantics of this; this should not be offered as a generic “any name of” tool that breaks the association between an image, its expected vendor and expected signing key.) (In reply to Miloslav Trmač from comment #12) > (In reply to Pedro Amoedo from comment #11) > > Thanks Miloslav, you have given me the signature relationship that I needed > > to confirm my theory: > > > > If we remove the extra non-necessary signatures that come from other > > registry (in this case the ones coming from legacy > > registry.access.redhat.com) and manually push the image only with the > > registry.redhat.io signatures, the "oc adm verify-image-signature" works > > again: > > AFAICS it has “worked” before as well; sure, it was noisy about half of the > signatures being rejected. > > > (Comments below are based only on reading the code as linked; I don’t nearly > understand enough of the OpenShift code base to be authoritative, and have > basically zero practical experience. Still, actually gathering logs would > seem to be more productive than forming hypotheses from limited data at this > point.) > > > 1) "oc import-image" no longer downloads the signatures because, somehow, > > the image-signature-import controller is not able to determine which > > signatures are the proper ones, > > You haven’t show anything to suggest that “not able to determine” is the > case. > > > I suposse that apart from regular skopeo > > copy command (which downloads all signatures), it probably makes some > > underlying docker signature-verification that fails. > > That seems unlikely; as far as I understand the code, it simply downloads > all signatures, without doing any kind of verification at all. > > > 2) "oc adm verify-image-signature" is not able to properly verify 2 > > different identities for the same image, it was designed to only provide one > > "--expected-identity" value, so verification will "succeed" only on half of > > the signatures but fail on the other half, which makes not possible the use > > of "--save" option to change the status of those signatures to Verified. > > Looking at the code, nothing prevents using --save if some of the > verifications fail; the “verification status will be removed” text in the > failures quoted earlier refers to what would happen with --save. (It is > possible that it does actually fail, (I don’t quite see how the status > removal code path works), but you haven’t shown any output like that.) > > Sure, the outcome of that could be that only half of the signatures are > marked as “verified” in the UI, but that should not be a _big_ problem - one > of the major design points of the multiple-signature support is that it is > always safe and harmless to add another, valid or invalid, signature. (Of > course, it may be the case that shipping OpenShift with an UI that shows > half of the signatures as rejected in the default configuration gives bad > impression, and that you want all green checkmarks or something; a code to > support marking signatures as verified in such combinations can of course be > written — but be _very_ careful about the semantics of this; this should not > be offered as a generic “any name of” tool that breaks the association > between an image, its expected vendor and expected signing key.) Thanks Miloslav, I'm trying to provide as much details as possible but those commands has no "debug" mode and also there are no related entries within controllers nor api logs (PFA) about any failure with the import-image procedure. Can you please confirm that the easy-to-follow reproduction steps that I have provided are working from your side?, FWIW, here you have again my tests against latest 3.11.98 version to discard any missed errata or patch. *** FIRST TEST *** Create a new clean project, configure both sigstores and "oc import-image": ~~~ [quicklab@master-1 ~]$ oc version oc v3.11.98 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://<obfuscated> openshift v3.11.98 kubernetes v1.11.0+d4cacc0 [quicklab@master-1 ~]$ oc new-project imported Now using project "imported" on server "https://<obfuscated> [...] [quicklab@master-1 ~]$ cat /etc/containers/registries.d/registry.access.redhat.com.yaml docker: registry.access.redhat.com: sigstore: https://access.redhat.com/webassets/docker/content/sigstore [quicklab@master-1 ~]$ cat /etc/containers/registries.d/registry.redhat.io.yaml docker: registry.redhat.io: sigstore: https://registry.redhat.io/containers/sigstore [quicklab@master-1 ~]$ oc import-image rhel7/etcd --from=registry.access.redhat.com/rhel7/etcd --confirm imagestream.image.openshift.io/etcd imported Name: etcd Namespace: imported Created: Less than a second ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2019-05-08T08:57:40Z Docker Pull Spec: docker-registry.default.svc:5000/imported/etcd Image Lookup: local=false Unique Images: 1 Tags: 1 latest tagged from registry.access.redhat.com/rhel7/etcd * registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Less than a second ago Image Name: etcd:latest Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: Less than a second ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.82MB in 3 layers Layers: 75.85MB sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 1.264kB sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 15.96MB sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 Image Created: 2 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker [quicklab@master-1 ~]$ oc describe istag etcd:latest Image Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: 15 seconds ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.82MB in 3 layers Layers: 75.85MB sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 1.264kB sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 15.96MB sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 Image Created: 2 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker [quicklab@master-1 ~]$ oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity registry.acces.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release error: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec does not have any signature ~~~ As you can see above, the "oc adm verify-image-signature" fails with: ~~~ error: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec does not have any signature ~~~ *** SECOND TEST *** Delete the imported is/images & manually push with skopeo the same one containing the signatures downloaded from https://access.redhat.com/webassets/docker/content/sigstore: ~~~ [pamoedo@p50 mirror] $ skopeo copy --dest-tls-verify=false --dest-creds openshift:<token> dir:/home/pamoedo/mirror/etcd_latest atomic:docker-registry-default.apps.<obfuscated>/imported/etcd:latest Getting image source signatures Checking if image destination supports signatures Skipping fetch of repeat blob sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 Skipping fetch of repeat blob sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 Skipping fetch of repeat blob sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 Copying config sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 0 B / 4.63 KB [------------------------------------------------------------] 0s Writing manifest to image destination Storing signatures [quicklab@master-1 ~]$ oc describe istag etcd:latest Image Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Docker Image: docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: 15 seconds ago Annotations: image.openshift.io/dockerLayersOrder=ascending image.openshift.io/manifestBlobStored=true openshift.io/image.managed=true Image Size: 91.82MB in 3 layers Layers: 75.85MB sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 1.264kB sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 15.96MB sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@66950bcce62456b4b004a7a3f3accc10 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d8aa4b464edc63c015e52644269ef316 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@127e8defde8dca6e894d78a21079e769 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@57d122c4cbea8fdf8c66b163223c8d69 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e1a67160e4a9a13b69a25d5041da6006 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@2cc57a7796f8e2fb6768df089bc4afd0 Type: atomic Status: Unverified Image Created: 2 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker [quicklab@master-1 ~]$ oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity registry.acces.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@66950bcce62456b4b004a7a3f3accc10 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d8aa4b464edc63c015e52644269ef316 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@127e8defde8dca6e894d78a21079e769 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@57d122c4cbea8fdf8c66b163223c8d69 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e1a67160e4a9a13b69a25d5041da6006 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@2cc57a7796f8e2fb6768df089bc4afd0 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec e37fe1db-716f-11e9-bc6c-fa163e95446d 968179 0 2019-05-08 05:01:38 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[2379/tcp:{} 2380/tcp:{} 4001/tcp:{} 7001/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[summary:A highly-available key value store for shared configuration name:rhel7/etcd version:3.2.22 install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component:etcd-container distribution-scope:public usage:etcd -help maintainer:Avesh Agarwal io.openshift.expose-services:2379:tcp,2380:tcp io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name:etcd vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 authoritative-source-url:registry.access.redhat.com release:30 build-date:2019-04-17T13:04:09.833457 url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. vendor:Red Hat, Inc. run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 architecture:x86_64 vcs-type:git io.openshift.tags:etcd com.redhat.license_terms:https://www.redhat.com/licenses/eulas]} 1.13.1 Avesh Agarwal <avagarwa> 0xc42145b1e0 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } [quicklab@master-1 ~]$ oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity registry.acces.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --save error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@66950bcce62456b4b004a7a3f3accc10 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d8aa4b464edc63c015e52644269ef316 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@127e8defde8dca6e894d78a21079e769 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@57d122c4cbea8fdf8c66b163223c8d69 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e1a67160e4a9a13b69a25d5041da6006 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@2cc57a7796f8e2fb6768df089bc4afd0 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): failed to get image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" manifest: Get https://docker-registry.default.svc:5000/v2/: x509: certificate signed by unknown authority The Image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" is invalid: * signatures[0].metadata.name: Required value: name or generateName is required * signatures[0].metadata.name: Invalid value: "": name must be of format <imageName>@<signatureName> * signatures[0].type: Required value * signatures[0].content: Required value * signatures[1].metadata.name: Required value: name or generateName is required * signatures[1].metadata.name: Invalid value: "": name must be of format <imageName>@<signatureName> * signatures[1].type: Required value * signatures[1].content: Required value * signatures[2].metadata.name: Required value: name or generateName is required * signatures[2].metadata.name: Invalid value: "": name must be of format <imageName>@<signatureName> * signatures[2].type: Required value * signatures[2].content: Required value * signatures[3].metadata.name: Required value: name or generateName is required * signatures[3].metadata.name: Invalid value: "": name must be of format <imageName>@<signatureName> * signatures[3].type: Required value * signatures[3].content: Required value * signatures[4].metadata.name: Required value: name or generateName is required * signatures[4].metadata.name: Invalid value: "": name must be of format <imageName>@<signatureName> * signatures[4].type: Required value * signatures[4].content: Required value * signatures[5].metadata.name: Required value: name or generateName is required * signatures[5].metadata.name: Invalid value: "": name must be of format <imageName>@<signatureName> * signatures[5].type: Required value * signatures[5].content: Required value ~~~ As you can see above, "oc adm verify-image-signature" command breaks both with "--save" option or without it, and consequently, signatures remain "unverified" (all of them): ~~~ [quicklab@master-1 ~]$ oc describe istag etcd:latest | grep -A3 Signatures Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@66950bcce62456b4b004a7a3f3accc10 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d8aa4b464edc63c015e52644269ef316 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@127e8defde8dca6e894d78a21079e769 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@57d122c4cbea8fdf8c66b163223c8d69 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e1a67160e4a9a13b69a25d5041da6006 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@2cc57a7796f8e2fb6768df089bc4afd0 Type: atomic Status: Unverified ~~~ With all due respect, is this not enough proof that both commands fail with the new "extra" signatures? NOTE: I'm privately attaching controllers & api logs FWIW Best Regards. The first test case: `oc import-image` doesn't import signatures, you need to wait about 1 hour before re-sync of the image-signature-import controller imports signatures. If it doesn't happen, you need to run the controller with -v=5 and collect verbose logs. Logs without enabled verbosity are not helpful for debugging. The second test case: https://access.redhat.com/solutions/3809961 (In reply to Oleg Bulatov from comment #16) > The first test case: `oc import-image` doesn't import signatures, you need > to wait about 1 hour before re-sync of the image-signature-import controller > imports signatures. If it doesn't happen, you need to run the controller > with -v=5 and collect verbose logs. Logs without enabled verbosity are not > helpful for debugging. > > The second test case: https://access.redhat.com/solutions/3809961 Thanks Oleg, For the first test: I will try to increase the loglevel for the controller and wait one hour just in case. Regarding the second test, please note that provided KCS solution also fails in this case: ~~~ [quicklab@master-1 ~]$ oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity docker-registry.default.svc:5000/imported/etcd --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --insecure error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:latest is not accepted Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[2379/tcp:{} 2380/tcp:{} 4001/tcp:{} 7001/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[vcs-type:git description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.openshift.tags:etcd run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.openshift.expose-services:2379:tcp,2380:tcp name:rhel7/etcd architecture:x86_64 release:30 version:3.2.22 summary:A highly-available key value store for shared configuration usage:etcd -help authoritative-source-url:registry.access.redhat.com io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. com.redhat.component:etcd-container distribution-scope:public maintainer:Avesh Agarwal url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 vendor:Red Hat, Inc. io.k8s.display-name:etcd com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 build-date:2019-04-17T13:04:09.833457 com.redhat.license_terms:https://www.redhat.com/licenses/eulas]} 1.13.1 Avesh Agarwal <avagarwa> 0xc421644dc0 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } ~~~ Regards. You need to pick one of these identities: registry.access.redhat.com/rhel7/etcd:3.2.22 registry.redhat.io/rhel7/etcd:3.2.22 registry.access.redhat.com/rhel7/etcd:3.2.22-30 registry.redhat.io/rhel7/etcd:3.2.22-30 registry.access.redhat.com/rhel7/etcd:latest registry.redhat.io/rhel7/etcd:latest This image is not signed for the identity docker-registry.default.svc:5000/imported/etcd. (In reply to Oleg Bulatov from comment #18) > You need to pick one of these identities: > > registry.access.redhat.com/rhel7/etcd:3.2.22 > registry.redhat.io/rhel7/etcd:3.2.22 > registry.access.redhat.com/rhel7/etcd:3.2.22-30 > registry.redhat.io/rhel7/etcd:3.2.22-30 > registry.access.redhat.com/rhel7/etcd:latest > registry.redhat.io/rhel7/etcd:latest > > This image is not signed for the identity > docker-registry.default.svc:5000/imported/etcd. FWIW, it fails with all of them: ~~~ [quicklab@master-1 ~]$ for i in `cat identities_list`; do echo "*** $i ***" && oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity $i --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --insecure && echo ""; done *** registry.access.redhat.com/rhel7/etcd:3.2.22 *** image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:latest is not accepted Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[2379/tcp:{} 2380/tcp:{} 4001/tcp:{} 7001/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[distribution-scope:public name:rhel7/etcd usage:etcd -help version:3.2.22 io.openshift.tags:etcd com.redhat.component:etcd-container release:30 com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 com.redhat.license_terms:https://www.redhat.com/licenses/eulas vcs-type:git io.k8s.display-name:etcd run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 authoritative-source-url:registry.access.redhat.com build-date:2019-04-17T13:04:09.833457 url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 architecture:x86_64 install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. maintainer:Avesh Agarwal description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. vendor:Red Hat, Inc. io.openshift.expose-services:2379:tcp,2380:tcp summary:A highly-available key value store for shared configuration]} 1.13.1 Avesh Agarwal <avagarwa> 0xc4207e4160 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 6d273f8b-7182-11e9-a129-fa163efb4c0c 0 2019-05-08 07:14:20 -0400 EDT <nil> <nil> map[] map[] [] nil [] } atomic [163 1 155 192 203 204 192 193 40 57 79 127 226 95 103 217 64 198 181 140 62 73 98 241 69 5 185 197 153 233 186 69 169 133 186 169 85 89 198 85 6 122 73 153 121 49 251 5 108 171 149 146 139 50 75 50 147 19 115 148 172 20 170 149 50 115 19 211 83 193 172 148 252 228 236 212 34 221 220 196 188 204 180 212 226 18 221 148 204 116 32 5 148 82 42 206 72 52 50 53 179 50 53 72 52 73 51 51 48 51 79 74 49 54 53 51 73 50 51 50 79 53 53 179 72 49 78 52 75 49 77 74 78 50 52 50 54 181 48 183 76 78 52 73 50 178 76 54 74 179 76 74 52 72 54 55 178 48 78 180 72 178 180 72 77 86 170 213 81 80 42 169 44 0 89 167 148 88 146 159 155 153 172 144 156 159 87 146 152 153 151 90 164 0 116 109 94 98 73 105 81 170 18 80 85 102 74 106 94 73 102 73 37 178 195 138 82 211 82 139 82 243 146 193 218 139 82 211 51 139 75 138 42 245 18 147 147 83 139 139 245 138 82 83 50 18 75 244 146 243 115 245 139 50 82 115 204 245 83 75 146 83 172 140 245 140 244 140 140 148 106 65 22 231 23 148 100 230 231 65 125 157 92 148 10 116 64 17 200 164 160 212 20 5 143 196 18 133 32 103 95 133 128 210 36 5 3 61 99 51 61 3 221 180 76 160 90 5 160 214 78 38 81 102 86 6 80 200 193 67 152 227 193 103 254 127 38 219 38 220 96 74 154 63 53 98 130 254 111 37 227 199 143 101 102 170 233 214 183 245 106 206 95 226 208 49 125 233 181 223 114 155 180 190 253 60 80 39 54 189 60 215 251 23 143 211 1 35 134 47 225 198 19 125 100 46 125 188 232 168 187 99 111 192 73 177 7 17 235 22 223 62 122 38 153 185 226 195 179 117 28 183 74 10 175 176 53 204 91 208 254 73 59 199 231 156 110 127 79 140 59 179 113 209 25 177 74 205 222 32 165 51 110 172 14 189 177 11 59 94 222 153 85 102 187 70 210 210 186 56 208 160 187 238 180 204 222 229 238 194 47 119 102 75 92 180 123 250 46 171 164 228 240 132 200 125 33 213 155 142 11 113 155 77 109 223 208 244 235 55 95 170 172 235 233 14 175 47 71 249 10 103 75 172 53 94 223 126 51 187 250 194 100 165 52 47 215 251 39 23 26 178 175 179 155 210 207 162 164 60 47 53 240 145 69 209 202 150 245 191 174 109 136 40 88 190 200 101 69 202 30 81 165 169 226 5 51 15 223 110 58 83 179 193 231 248 221 175 183 37 246 122 253 123 194 164 219 234 175 84 25 48 39 138 173 109 63 87 86 242 185 166 121 242 23 154 102 190 86 92 149 22 175 180 210 103 157 237 254 219 102 215 82 139 34 53 103 62 92 169 199 97 116 219 78 250 212 135 79 166 203 255 155 159 159 20 164 112 82 184 43 164 74 112 182 210 244 211 121 218 61 78 250 223 118 200 220 23 239 91 158 251 203 240 80 86 157 230 211 140 226 249 181 203 79 153 245 148 172 57 231 46 220 146 126 105 67 142 31 243 204 189 91 56 127 76 251 189 193 217 175 180 167 98 237 202 173 155 231 88 215 244 52 190 255 227 105 194 226 189 230 205 151 72 214 101 155 239 86 110 127 110 236 83 177 189 238 200 169 123 74 45 247 247 108 61 187 96 219 22 97 151 247 183 172 30 111 138 84 235 214 171 56 202 150 249 80 33 142 113 249 74 123 129 76 207 87 97 201 231 179 216 90 39 230 191 216 180 121 106 207 141 135 219 20 24 79 111 253 159 176 98 250 162 57 31 37 63 70 255 175 179 60 108 126 95 119 226 219 227 250 23 184 123 63 23 36 158 251 244 238 250 84 111 245 191 206 125 207 157 103 189 90 126 201 107 201 162 185 173 78 44 249 170 31 231 237 2 0] [{Trusted True 2019-05-08 09:24:17.53446401 -0400 EDT m=+0.504801582 2019-05-08 09:24:17.53446401 -0400 EDT m=+0.504801582 manually verified verified by user "quicklab"} {ForImage True 2019-05-08 09:24:17.53446401 -0400 EDT m=+0.504801582 2019-05-08 09:24:17.53446401 -0400 EDT m=+0.504801582 }] map[] <nil> 0xc4204c57c0 <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } *** registry.redhat.io/rhel7/etcd:3.2.22 *** error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:latest is not accepted Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[openshift.io/image.managed:true image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[4001/tcp:{} 7001/tcp:{} 2379/tcp:{} 2380/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com io.k8s.display-name:etcd io.openshift.expose-services:2379:tcp,2380:tcp io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 maintainer:Avesh Agarwal url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 authoritative-source-url:registry.access.redhat.com build-date:2019-04-17T13:04:09.833457 description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope:public summary:A highly-available key value store for shared configuration run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 vcs-type:git architecture:x86_64 usage:etcd -help io.openshift.tags:etcd release:30 com.redhat.component:etcd-container name:rhel7/etcd com.redhat.license_terms:https://www.redhat.com/licenses/eulas vendor:Red Hat, Inc. version:3.2.22]} 1.13.1 Avesh Agarwal <avagarwa> 0xc4214c0580 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c 6d432dac-7182-11e9-a129-fa163efb4c0c 0 2019-05-08 07:14:20 -0400 EDT <nil> <nil> map[] map[] [] nil [] } atomic [163 1 155 192 203 204 192 193 40 57 79 127 226 95 103 217 64 198 181 140 46 73 98 241 69 5 185 197 153 233 186 69 169 133 186 62 230 33 129 238 73 122 73 153 121 49 251 5 108 171 149 146 139 50 75 50 147 19 115 148 172 20 170 149 50 115 19 211 83 193 172 148 252 228 236 212 34 221 220 196 188 204 180 212 226 18 221 148 204 116 32 5 148 82 42 206 72 52 50 53 179 50 53 72 52 73 51 51 48 51 79 74 49 54 53 51 73 50 51 50 79 53 53 179 72 49 78 52 75 49 77 74 78 50 52 50 54 181 48 183 76 78 52 73 50 178 76 54 74 179 76 74 52 72 54 55 178 48 78 180 72 178 180 72 77 86 170 213 81 80 42 169 44 0 89 167 148 88 146 159 155 153 172 144 156 159 87 146 152 153 151 90 164 0 116 109 94 98 73 105 81 170 18 80 85 102 74 106 94 73 102 73 37 178 195 138 82 211 82 139 82 243 146 193 218 139 82 211 51 139 75 138 42 245 138 82 83 50 18 75 244 50 243 245 139 50 82 115 204 245 83 75 146 83 172 140 245 140 244 140 140 148 106 65 22 230 23 148 100 230 231 65 125 155 92 148 10 180 184 8 100 66 80 106 138 130 71 98 137 66 144 179 175 66 64 105 146 130 129 158 177 153 158 129 110 90 38 80 173 2 80 107 39 147 40 51 43 3 40 196 224 33 203 209 127 141 255 127 248 245 77 223 165 63 174 21 155 251 97 87 173 96 227 175 181 111 26 47 216 223 238 57 63 35 51 121 214 133 51 47 203 234 54 79 251 120 81 66 253 135 192 198 149 70 39 223 110 111 187 101 109 243 226 39 183 215 204 91 42 235 154 26 30 254 112 58 114 206 235 95 207 204 189 222 254 233 82 19 227 173 10 153 98 230 126 121 17 173 84 35 174 148 35 118 187 224 226 110 221 132 181 12 189 66 243 36 158 199 28 121 61 41 91 166 115 255 175 173 95 25 133 42 186 68 159 184 54 247 247 244 126 185 243 154 161 232 153 160 173 155 235 103 153 23 9 211 166 110 121 147 249 61 122 246 180 117 37 79 13 76 52 180 76 166 153 200 8 207 92 185 249 197 89 227 59 214 217 109 172 43 184 140 106 126 75 11 77 126 255 237 193 215 39 124 247 242 206 233 172 249 223 247 232 164 182 228 146 220 99 7 190 239 127 216 126 114 201 149 210 24 30 195 171 207 146 255 189 59 89 119 130 47 112 194 218 208 197 210 127 101 207 53 206 147 139 159 83 214 201 230 173 232 144 203 199 177 245 236 134 239 17 59 99 106 143 29 217 248 253 103 222 204 13 138 23 95 57 30 145 244 95 176 90 47 124 179 144 159 209 181 71 123 126 40 45 51 236 187 48 241 83 92 107 175 248 141 9 175 84 155 28 175 36 216 23 249 78 122 197 106 240 114 57 127 133 92 239 220 61 209 193 157 41 175 150 22 123 111 79 185 253 115 226 174 233 130 58 85 142 113 221 33 50 39 56 174 238 153 175 215 181 147 175 122 81 123 197 143 205 252 169 251 36 255 90 172 63 160 243 114 179 201 11 181 206 99 154 55 52 39 41 31 108 44 103 47 148 242 178 48 59 54 187 87 232 196 239 235 234 169 146 11 230 186 164 239 237 121 193 191 183 51 179 99 201 254 223 62 229 115 75 159 250 222 83 238 40 77 237 142 19 219 167 55 199 176 170 170 54 81 118 195 163 211 231 103 248 222 240 141 208 96 189 44 199 124 248 201 243 190 243 75 20 35 207 69 21 249 240 213 217 76 232 14 124 212 98 231 191 65 251 72 172 156 245 225 217 51 222 94 88 155 117 130 115 206 198 144 192 89 139 244 143 56 218 41 44 153 184 104 122 134 250 146 164 139 69 254 202 191 46 172 57 232 221 127 119 199 34 33 0] [{Trusted True 2019-05-08 09:24:18.27609 -0400 EDT m=+0.472297135 2019-05-08 09:24:18.27609 -0400 EDT m=+0.472297135 manually verified verified by user "quicklab"} {ForImage True 2019-05-08 09:24:18.27609 -0400 EDT m=+0.472297135 2019-05-08 09:24:18.27609 -0400 EDT m=+0.472297135 }] map[] <nil> 0xc421676640 <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } *** registry.access.redhat.com/rhel7/etcd:3.2.22-30 *** error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:latest is not accepted Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[2380/tcp:{} 4001/tcp:{} 7001/tcp:{} 2379/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[release:30 version:3.2.22 description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. summary:A highly-available key value store for shared configuration url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 distribution-scope:public io.openshift.expose-services:2379:tcp,2380:tcp io.openshift.tags:etcd com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com authoritative-source-url:registry.access.redhat.com build-date:2019-04-17T13:04:09.833457 com.redhat.component:etcd-container maintainer:Avesh Agarwal install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 vendor:Red Hat, Inc. architecture:x86_64 io.k8s.display-name:etcd io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. usage:etcd -help uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type:git name:rhel7/etcd com.redhat.license_terms:https://www.redhat.com/licenses/eulas run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3]} 1.13.1 Avesh Agarwal <avagarwa> 0xc4210d6c60 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a 6d5f2164-7182-11e9-a129-fa163efb4c0c 0 2019-05-08 07:14:20 -0400 EDT <nil> <nil> map[] map[] [] nil [] } atomic [163 1 155 192 203 204 192 193 40 57 79 127 226 95 103 217 64 198 181 140 254 73 98 241 69 5 185 197 153 233 186 69 169 133 186 22 121 230 17 30 198 122 73 153 121 49 251 5 156 171 149 146 139 50 75 50 147 19 115 148 172 20 170 149 50 115 19 211 83 193 172 148 252 228 236 212 34 221 220 196 188 204 180 212 226 18 221 148 204 116 32 5 148 82 42 206 72 52 50 53 179 50 53 72 52 73 51 51 48 51 79 74 49 54 53 51 73 50 51 50 79 53 53 179 72 49 78 52 75 49 77 74 78 50 52 50 54 181 48 183 76 78 52 73 50 178 76 54 74 179 76 74 52 72 54 55 178 48 78 180 72 178 180 72 77 86 170 213 81 80 42 169 44 0 89 167 148 88 146 159 155 153 172 144 156 159 87 146 152 153 151 90 164 0 116 109 94 98 73 105 81 170 18 80 85 102 74 106 94 73 102 73 37 178 195 138 82 211 82 139 82 243 146 193 218 139 82 211 51 139 75 138 42 245 18 147 147 83 139 139 245 138 82 83 50 18 75 244 146 243 115 245 139 50 82 115 204 245 83 75 146 83 172 140 245 140 244 140 140 116 141 13 148 106 65 118 231 23 148 100 230 231 65 61 158 92 148 10 116 67 17 200 176 160 212 20 5 143 196 18 133 32 103 95 133 128 210 36 5 3 61 99 51 61 3 221 180 76 160 90 5 160 214 78 38 81 102 86 6 80 224 193 3 153 67 188 94 128 161 241 77 144 248 234 156 85 250 173 204 245 44 155 182 85 45 120 108 155 187 112 251 197 107 239 182 43 44 49 116 231 237 204 190 63 205 72 179 117 235 5 115 190 210 125 236 13 142 193 201 137 17 119 124 184 119 48 236 191 88 177 82 65 95 161 205 127 25 171 11 227 202 227 75 243 174 73 123 93 233 112 96 237 79 230 145 18 173 174 138 202 153 186 161 211 35 86 161 121 123 244 163 172 12 182 25 181 55 116 30 221 109 123 214 161 148 244 35 212 249 218 163 165 182 143 36 46 188 222 201 83 114 100 195 45 111 182 156 235 86 6 59 226 227 107 54 197 90 252 183 218 242 178 90 73 239 2 159 233 35 31 145 223 33 203 181 39 152 112 60 143 116 234 104 240 12 122 232 186 62 150 57 179 112 213 190 87 47 93 254 204 104 19 221 184 76 79 250 205 161 144 21 179 228 156 143 232 110 22 239 94 150 162 117 121 241 247 55 178 118 172 247 46 252 173 159 20 32 203 99 88 195 84 61 49 70 43 172 238 106 186 184 112 238 189 218 118 127 238 142 192 165 61 97 137 177 235 239 41 217 174 100 157 84 169 172 34 245 244 106 128 146 189 29 139 94 222 127 71 43 255 105 147 38 238 85 251 229 45 177 224 167 212 20 41 227 173 246 143 206 207 90 112 187 77 224 233 62 189 217 82 172 222 235 219 53 133 252 254 239 218 224 83 181 213 255 115 142 120 200 148 163 207 171 14 176 241 150 77 251 40 115 254 217 33 23 55 137 150 199 29 203 214 10 127 169 246 17 203 218 234 16 203 246 135 139 183 78 203 255 248 203 132 191 175 234 183 158 189 16 21 167 209 222 112 215 232 181 145 208 242 38 174 167 242 207 190 229 105 105 28 178 58 18 21 234 242 124 154 251 19 243 155 108 159 146 183 183 36 44 255 212 109 155 101 166 91 247 69 230 255 45 159 204 207 183 75 182 159 204 255 125 160 72 230 238 201 171 75 127 63 83 16 156 115 192 244 247 243 183 146 193 197 171 25 56 235 173 39 137 88 93 48 207 9 140 244 174 88 244 231 242 158 67 1 76 239 5 239 221 226 146 186 41 250 211 227 248 36 222 229 229 65 146 153 115 123 247 29 118 230 175 253 57 193 222 59 96 249 97 107 187 127 187 101 189 45 255 203 170 174 219 187 237 82 126 138 18 0] [{Trusted True 2019-05-08 09:24:19.305823472 -0400 EDT m=+0.755408599 2019-05-08 09:24:19.305823472 -0400 EDT m=+0.755408599 manually verified verified by user "quicklab"} {ForImage True 2019-05-08 09:24:19.305823472 -0400 EDT m=+0.755408599 2019-05-08 09:24:19.305823472 -0400 EDT m=+0.755408599 }] map[] <nil> 0xc4216e59a0 <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } *** registry.redhat.io/rhel7/etcd:3.2.22-30 *** error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-30 is not accepted image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:latest is not accepted Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[2379/tcp:{} 2380/tcp:{} 4001/tcp:{} 7001/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[summary:A highly-available key value store for shared configuration install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 maintainer:Avesh Agarwal vcs-type:git description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 authoritative-source-url:registry.access.redhat.com vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 io.openshift.tags:etcd architecture:x86_64 build-date:2019-04-17T13:04:09.833457 com.redhat.component:etcd-container url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage:etcd -help distribution-scope:public io.k8s.display-name:etcd vendor:Red Hat, Inc. io.openshift.expose-services:2379:tcp,2380:tcp name:rhel7/etcd com.redhat.license_terms:https://www.redhat.com/licenses/eulas version:3.2.22 release:30]} 1.13.1 Avesh Agarwal <avagarwa> 0xc421469760 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 6e7aec81-7182-11e9-a129-fa163efb4c0c 0 2019-05-08 07:14:22 -0400 EDT <nil> <nil> map[] map[] [] nil [] } atomic [163 1 155 192 203 204 192 193 40 57 79 127 226 95 103 217 64 198 181 140 238 73 98 241 69 5 185 197 153 233 186 69 169 133 186 110 81 174 97 201 65 122 73 153 121 49 251 5 92 170 149 146 139 50 75 50 147 19 115 148 172 20 170 149 50 115 19 211 83 193 172 148 252 228 236 212 34 221 220 196 188 204 180 212 226 18 221 148 204 116 32 5 148 82 42 206 72 52 50 53 179 50 53 72 52 73 51 51 48 51 79 74 49 54 53 51 73 50 51 50 79 53 53 179 72 49 78 52 75 49 77 74 78 50 52 50 54 181 48 183 76 78 52 73 50 178 76 54 74 179 76 74 52 72 54 55 178 48 78 180 72 178 180 72 77 86 170 213 81 80 42 169 44 0 89 167 148 88 146 159 155 153 172 144 156 159 87 146 152 153 151 90 164 0 116 109 94 98 73 105 81 170 18 80 85 102 74 106 94 73 102 73 37 178 195 138 82 211 82 139 82 243 146 193 218 139 82 211 51 139 75 138 42 245 138 82 83 50 18 75 244 50 243 245 139 50 82 115 204 245 83 75 146 83 172 140 245 140 244 140 140 116 141 13 148 106 65 118 230 23 148 100 230 231 65 61 156 92 148 10 180 187 8 100 72 80 106 138 130 71 98 137 66 144 179 175 66 64 105 146 130 129 158 177 153 158 129 110 90 38 80 173 2 80 107 39 147 40 51 43 3 40 208 224 129 203 113 69 153 255 159 138 190 183 136 70 234 123 134 15 93 27 107 148 126 221 213 22 217 63 209 85 99 113 211 99 39 223 134 198 61 186 91 215 113 47 63 202 121 196 255 192 103 142 168 95 25 247 158 220 178 183 144 14 173 159 62 181 139 171 195 107 225 147 215 83 94 44 170 148 209 229 43 124 96 156 255 100 203 229 208 35 247 222 6 70 205 123 193 23 124 198 102 122 158 203 123 67 134 108 190 187 15 246 58 36 164 117 126 40 137 171 95 246 109 9 103 209 125 187 233 204 2 215 78 198 198 110 20 55 53 92 211 254 217 236 252 153 229 28 191 223 189 84 235 94 240 253 90 147 199 141 244 142 219 139 158 25 164 51 10 132 254 250 124 229 186 203 231 186 175 150 235 170 138 62 47 237 94 35 149 116 53 61 139 165 79 61 240 120 129 233 213 61 254 193 215 231 159 251 28 245 101 170 255 76 166 18 134 215 117 54 172 47 189 3 2 79 179 132 121 61 154 47 121 149 183 234 224 227 205 234 22 247 202 156 142 68 77 61 80 113 195 117 153 165 210 178 191 140 203 45 118 235 245 31 108 208 120 184 87 53 185 187 240 247 214 172 121 77 237 13 70 54 138 63 248 69 101 56 181 191 115 134 101 214 114 159 8 223 111 52 229 209 209 43 242 225 91 171 251 23 31 19 16 212 77 10 98 15 80 145 137 84 185 191 224 153 253 150 183 202 199 184 155 127 207 253 220 196 126 48 84 232 68 101 243 195 138 236 22 179 39 245 190 59 189 191 61 136 223 114 64 214 110 225 167 91 23 250 116 156 91 207 157 211 58 214 153 181 67 255 120 83 243 210 143 206 123 53 79 116 169 178 235 255 237 202 108 201 121 185 34 234 187 115 141 198 147 223 247 150 93 127 179 183 232 83 139 252 117 246 109 199 84 207 166 217 107 43 74 92 82 75 244 187 48 243 236 113 187 218 233 107 38 10 228 164 239 59 200 220 191 180 98 151 146 225 150 167 15 62 204 253 110 36 198 175 180 224 198 68 241 50 139 68 47 253 6 193 56 245 136 104 25 209 75 156 135 98 50 87 6 89 5 124 223 25 198 121 46 124 142 204 43 45 69 246 94 158 198 147 145 134 31 214 62 59 174 171 61 117 233 179 236 91 15 142 45 180 102 95 30 179 93 157 245 240 81 215 171 11 34 151 154 248 9 204 147 188 12 0] [{Trusted True 2019-05-08 09:24:20.065924205 -0400 EDT m=+0.580567793 2019-05-08 09:24:20.065924205 -0400 EDT m=+0.580567793 manually verified verified by user "quicklab"} {ForImage True 2019-05-08 09:24:20.065924205 -0400 EDT m=+0.580567793 2019-05-08 09:24:20.065924205 -0400 EDT m=+0.580567793 }] map[] <nil> 0xc420555d80 <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } *** registry.access.redhat.com/rhel7/etcd:latest *** error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:latest is not accepted Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[2380/tcp:{} 4001/tcp:{} 7001/tcp:{} 2379/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[name:rhel7/etcd version:3.2.22 vendor:Red Hat, Inc. uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 usage:etcd -help url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 com.redhat.license_terms:https://www.redhat.com/licenses/eulas io.openshift.expose-services:2379:tcp,2380:tcp maintainer:Avesh Agarwal architecture:x86_64 vcs-type:git com.redhat.component:etcd-container com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope:public release:30 vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 summary:A highly-available key value store for shared configuration build-date:2019-04-17T13:04:09.833457 io.openshift.tags:etcd io.k8s.display-name:etcd install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 authoritative-source-url:registry.access.redhat.com]} 1.13.1 Avesh Agarwal <avagarwa> 0xc421142c60 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 6ea9b974-7182-11e9-a129-fa163efb4c0c 0 2019-05-08 07:14:22 -0400 EDT <nil> <nil> map[] map[] [] nil [] } atomic [163 1 155 192 203 204 192 193 40 57 79 127 226 95 103 217 64 198 181 140 62 73 98 241 69 5 185 197 153 233 186 69 169 133 186 94 150 25 41 158 22 122 73 153 121 49 251 5 92 170 149 146 139 50 75 50 147 19 115 148 172 20 170 149 50 115 19 211 83 193 172 148 252 228 236 212 34 221 220 196 188 204 180 212 226 18 221 148 204 116 32 5 148 82 42 206 72 52 50 53 179 50 53 72 52 73 51 51 48 51 79 74 49 54 53 51 73 50 51 50 79 53 53 179 72 49 78 52 75 49 77 74 78 50 52 50 54 181 48 183 76 78 52 73 50 178 76 54 74 179 76 74 52 72 54 55 178 48 78 180 72 178 180 72 77 86 170 213 81 80 42 169 44 0 89 167 148 88 146 159 155 153 172 144 156 159 87 146 152 153 151 90 164 0 116 109 94 98 73 105 81 170 18 80 85 102 74 106 94 73 102 73 37 178 195 138 82 211 82 139 82 243 146 193 218 139 82 211 51 139 75 138 42 245 18 147 147 83 139 139 245 138 82 83 50 18 75 244 146 243 115 245 139 50 82 115 204 245 83 75 146 83 172 114 18 75 64 94 168 5 89 156 95 80 146 153 159 7 245 117 114 81 42 208 1 69 32 147 130 82 83 20 60 18 75 20 130 156 125 21 2 74 147 20 12 244 140 205 244 12 116 211 50 129 106 21 128 90 59 153 68 153 89 25 64 33 7 15 97 142 253 249 252 255 84 158 159 224 137 206 253 167 245 33 128 125 114 235 196 47 89 139 123 51 215 196 180 214 110 200 224 215 224 60 184 80 234 240 103 118 225 218 142 184 172 48 182 164 77 115 87 45 77 252 241 73 213 190 98 223 35 39 181 221 171 181 178 142 255 242 91 246 67 37 152 173 124 241 205 197 71 228 163 143 238 75 250 227 103 205 39 87 189 238 250 28 198 235 190 50 83 189 188 98 153 37 207 249 239 156 38 37 189 49 215 216 71 237 93 250 182 169 236 51 238 158 184 247 240 201 246 14 177 19 187 45 88 143 91 27 213 29 221 217 216 172 112 200 111 213 237 171 65 251 151 223 225 44 61 99 184 39 191 91 255 198 137 165 155 248 83 107 25 250 39 21 207 169 252 21 82 180 42 243 101 123 221 126 235 15 22 199 95 41 104 22 52 199 207 154 117 179 167 53 104 122 99 253 139 172 226 227 79 184 147 95 54 38 51 148 86 48 46 224 254 188 84 161 242 83 17 239 170 183 203 53 68 246 45 61 155 215 124 166 60 82 103 185 226 229 255 46 74 199 207 69 115 229 28 45 255 35 187 49 174 215 222 252 248 142 155 94 189 135 214 117 234 202 123 107 255 158 248 186 233 219 245 200 120 239 19 235 175 217 105 127 123 80 156 34 215 242 224 94 222 198 87 34 34 41 123 239 251 106 112 70 206 100 78 57 205 37 159 82 159 30 191 115 31 131 204 204 235 255 205 143 205 213 98 230 19 191 18 190 169 180 172 235 253 79 189 143 255 18 24 111 69 62 86 12 127 244 118 57 235 166 192 48 23 131 192 75 90 47 58 90 175 174 13 210 252 177 111 70 73 156 107 146 201 195 196 37 107 4 94 220 233 124 169 86 111 255 96 202 150 87 230 113 2 161 173 203 66 175 132 41 75 235 125 185 163 124 69 64 248 126 217 212 173 222 61 55 77 214 173 56 43 115 212 81 250 115 108 225 113 145 201 255 86 89 171 28 55 45 185 34 97 255 223 95 110 171 50 215 195 174 105 246 111 55 207 58 238 222 189 71 46 210 62 115 193 228 21 46 73 169 91 46 204 89 153 31 149 44 54 71 250 243 38 221 116 69 7 30 151 141 12 183 47 207 61 201 115 195 241 218 226 25 51 54 136 170 221 182 223 244 183 170 254 92 237 123 137 11 75 78 126 186 188 80 115 227 12 177 151 0] [{Trusted True 2019-05-08 09:24:20.805381942 -0400 EDT m=+0.631621341 2019-05-08 09:24:20.805381942 -0400 EDT m=+0.631621341 manually verified verified by user "quicklab"} {ForImage True 2019-05-08 09:24:20.805381942 -0400 EDT m=+0.631621341 2019-05-08 09:24:20.805381942 -0400 EDT m=+0.631621341 }] map[] <nil> 0xc4205ef360 <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } *** registry.redhat.io/rhel7/etcd:latest *** error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") Neither --save nor --remove-all were passed, image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" not updated to &{{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 6908dbe2-7182-11e9-bc6c-fa163e95446d 990718 0 2019-05-08 07:14:13 -0400 EDT <nil> <nil> map[] map[image.openshift.io/dockerLayersOrder:ascending image.openshift.io/manifestBlobStored:true openshift.io/image.managed:true] [] nil [] } docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec {{ } sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 2019-04-17 09:04:43 -0400 EDT {ed75d4430067 0 0 0 false false false [] map[7001/tcp:{} 2379/tcp:{} 2380/tcp:{} 4001/tcp:{}] false false false [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker] [/bin/sh -c rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'] [] sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095 map[] [] false [] [] map[io.openshift.expose-services:2379:tcp,2380:tcp vendor:Red Hat, Inc. vcs-ref:7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type:git description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. architecture:x86_64 authoritative-source-url:registry.access.redhat.com release:30 io.k8s.display-name:etcd name:rhel7/etcd com.redhat.component:etcd-container summary:A highly-available key value store for shared configuration io.openshift.tags:etcd com.redhat.license_terms:https://www.redhat.com/licenses/eulas url:https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage:etcd -help version:3.2.22 build-date:2019-04-17T13:04:09.833457 install:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description:etcd is a distributed reliable key-value store for the most critical data of a distributed system. maintainer:Avesh Agarwal run:/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 com.redhat.build-host:cpt-0006.osbs.prod.upshift.rdu2.redhat.com distribution-scope:public uninstall:/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3]} 1.13.1 Avesh Agarwal <avagarwa> 0xc421465a20 amd64 91824976} 1.0 [{sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 75854078 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1264 application/vnd.docker.image.rootfs.diff.tar.gzip} {sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15964893 application/vnd.docker.image.rootfs.diff.tar.gzip}] [{{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } { 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } [] [] map[] <nil> <nil> <nil>} {{ } {sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 6ec543c2-7182-11e9-a129-fa163efb4c0c 0 2019-05-08 07:14:22 -0400 EDT <nil> <nil> map[] map[] [] nil [] } atomic [163 1 155 192 203 204 192 193 40 57 79 127 226 95 103 217 64 198 181 140 46 73 98 241 69 5 185 197 153 233 186 69 169 133 186 165 134 129 149 129 241 122 73 153 121 49 251 5 92 171 149 146 139 50 75 50 147 19 115 148 172 20 170 149 50 115 19 211 83 193 172 148 252 228 236 212 34 221 220 196 188 204 180 212 226 18 221 148 204 116 32 5 148 82 42 206 72 52 50 53 179 50 53 72 52 73 51 51 48 51 79 74 49 54 53 51 73 50 51 50 79 53 53 179 72 49 78 52 75 49 77 74 78 50 52 50 54 181 48 183 76 78 52 73 50 178 76 54 74 179 76 74 52 72 54 55 178 48 78 180 72 178 180 72 77 86 170 213 81 80 42 169 44 0 89 167 148 88 146 159 155 153 172 144 156 159 87 146 152 153 151 90 164 0 116 109 94 98 73 105 81 170 18 80 85 102 74 106 94 73 102 73 37 178 195 138 82 211 82 139 82 243 146 193 218 139 82 211 51 139 75 138 42 245 138 82 83 50 18 75 244 50 243 245 139 50 82 115 204 245 83 75 146 83 172 114 18 75 64 78 175 5 89 152 95 80 146 153 159 7 245 109 114 81 42 208 226 34 144 9 65 169 41 10 30 137 37 10 65 206 190 10 1 165 73 10 6 122 198 102 122 6 186 105 153 64 181 10 64 173 157 76 162 204 172 12 160 16 131 135 44 71 195 51 1 134 230 5 154 153 66 249 167 46 182 156 184 192 183 251 68 212 199 112 254 176 80 206 29 91 156 132 207 110 50 190 119 64 62 141 67 90 214 189 107 179 98 149 216 186 118 57 119 193 199 237 19 87 61 53 123 45 177 216 142 231 207 243 3 202 45 147 204 103 46 114 110 145 158 87 124 81 41 64 109 245 146 45 121 125 126 174 55 250 11 21 244 18 43 133 207 58 59 184 196 172 144 190 119 254 210 214 221 23 127 121 77 218 197 252 53 208 169 77 116 159 247 20 46 165 245 151 239 168 100 77 41 105 124 190 104 119 139 181 176 115 195 219 116 151 211 161 201 203 158 124 249 211 180 51 124 189 109 97 99 128 222 53 65 173 224 207 146 205 177 90 167 214 134 220 159 46 156 180 143 127 245 218 115 171 61 152 119 166 42 93 223 251 155 79 35 64 253 241 142 169 70 73 7 185 100 188 130 21 155 29 87 189 138 120 167 210 246 119 241 194 248 85 121 179 215 93 186 28 162 192 181 51 89 175 197 253 194 73 41 53 231 159 243 146 54 149 10 109 63 24 229 216 91 254 226 111 209 185 151 18 47 116 103 212 180 187 205 222 192 119 224 201 102 53 241 95 79 42 58 43 28 15 134 6 79 93 176 175 238 13 179 222 242 147 177 39 207 93 23 102 127 106 193 209 248 98 243 129 90 187 198 143 251 252 37 175 253 171 212 151 188 199 54 239 98 136 220 100 183 35 255 3 211 106 101 88 38 239 45 92 25 193 225 116 34 35 49 188 44 142 87 38 92 53 230 161 210 30 182 146 245 93 166 192 16 182 204 143 191 117 128 195 113 203 20 251 172 27 239 231 115 25 243 253 96 75 213 57 237 215 29 239 19 210 241 238 194 183 98 165 98 201 77 153 26 121 87 254 127 208 142 126 44 251 115 123 160 187 218 35 1 38 54 233 176 179 43 191 71 60 125 124 191 236 241 52 190 237 174 51 206 213 204 49 157 97 241 82 133 251 157 246 86 93 1 222 13 90 213 55 118 254 59 191 191 108 95 133 80 209 77 158 63 118 246 79 251 47 117 221 245 182 173 58 205 219 240 240 96 138 233 188 234 47 82 189 50 199 85 246 51 166 49 136 49 9 235 29 213 220 144 185 51 187 177 231 93 85 186 127 236 140 197 243 212 202 110 108 157 120 101 141 254 254 240 51 223 46 2 0] [{Trusted True 2019-05-08 09:24:21.597790026 -0400 EDT m=+0.721038529 2019-05-08 09:24:21.597790026 -0400 EDT m=+0.721038529 manually verified verified by user "quicklab"} {ForImage True 2019-05-08 09:24:21.597790026 -0400 EDT m=+0.721038529 2019-05-08 09:24:21.597790026 -0400 EDT m=+0.721038529 }] map[] <nil> 0xc42054d260 <nil>}] [] application/vnd.docker.distribution.manifest.v2+json } [quicklab@master-1 ~]$ oc describe istag etcd:latest | grep -A3 Signatures Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@f50dd685c567f98868bf6f1fc5b05f68 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@256db0f78ba71bb9f92f947198b0e19c Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@3f974ba29b876064bb8a0388ccd1814a Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@d9e54c0e73477cafbec435f25475dd64 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@6884164c593bea122d0894b0ebb2b932 Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e251ebb5c45c20d58271228af29f2af8 Type: atomic Status: Unverified ~~~ @Oleg, I have set global DEBUG_LOGLEVEL to 5 into /etc/origin/master/master.env and restarted both api & controllers, after that I ran "oc import-image rhel7/etcd --from=registry.access.redhat.com/rhel7/etcd --confirm" as usual and waited one hour as suggested, the signatures are still not present into the imagestream and I couldn't find any related entry into the controllers logs on at first sight (PFA just in case).
~~~
[quicklab@master-1 ~]$ cat /etc/containers/registries.d/registry.*
docker:
registry.access.redhat.com:
sigstore: https://access.redhat.com/webassets/docker/content/sigstore
docker:
registry.redhat.io:
sigstore: https://registry.redhat.io/containers/sigstore
[quicklab@master-1 ~]$ oc describe istag etcd:latest
Image Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec
Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec
Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec
Created: About an hour ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
Image Size: 91.82MB in 3 layers
Layers: 75.85MB sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526
1.264kB sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2
15.96MB sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78
Image Created: 3 weeks ago
Author: Avesh Agarwal <avagarwa>
Arch: amd64
Command: /usr/bin/etcd-env.sh /usr/bin/etcd
Working Dir: <none>
User: <none>
Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp
Docker Labels: architecture=x86_64
authoritative-source-url=registry.access.redhat.com
build-date=2019-04-17T13:04:09.833457
com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com
com.redhat.component=etcd-container
com.redhat.license_terms=https://www.redhat.com/licenses/eulas
description=etcd is a distributed reliable key-value store for the most critical data of a distributed system.
distribution-scope=public
install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3
io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system.
io.k8s.display-name=etcd
io.openshift.expose-services=2379:tcp,2380:tcp
io.openshift.tags=etcd
maintainer=Avesh Agarwal
name=rhel7/etcd
release=30
run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3
summary=A highly-available key value store for shared configuration
uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3
url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30
usage=etcd -help
vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2
vcs-type=git
vendor=Red Hat, Inc.
version=3.2.22
Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=docker
[quicklab@master-1 ~]$ oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity registry.access.redhat.com/rhel7/etcd:latest --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
error: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec does not have any signature
~~~
Regards.
(In reply to Pedro Amoedo from comment #13) > (In reply to Miloslav Trmač from comment #12) > > > 2) "oc adm verify-image-signature" is not able to properly verify 2 > > > different identities for the same image, it was designed to only provide one > > > "--expected-identity" value, so verification will "succeed" only on half of > > > the signatures but fail on the other half, which makes not possible the use > > > of "--save" option to change the status of those signatures to Verified. > > > > Looking at the code, nothing prevents using --save if some of the > > verifications fail; the “verification status will be removed” text in the > > failures quoted earlier refers to what would happen with --save. (It is > > possible that it does actually fail, (I don’t quite see how the status > > removal code path works), but you haven’t shown any output like that.) <snip> > [quicklab@master-1 ~]$ oc adm verify-image-signature > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > --expected-identity > registry.acces.redhat.com/rhel7/etcd@sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --save > error verifying signature > sha256: > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@66950bcce624 > 56b4b004a7a3f3accc10 for image > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > (verification status will be removed): failed to get image > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > manifest: Get https://docker-registry.default.svc:5000/v2/: x509: > certificate signed by unknown authority … > error verifying signature > The Image > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" is > invalid: > * signatures[0].metadata.name: Required value: name or generateName is > required > * signatures[0].metadata.name: Invalid value: "": name must be of format > <imageName>@<signatureName> > * signatures[0].type: Required value > * signatures[0].content: Required value … Thanks, this confirms my suspicion: https://github.com/openshift/origin/blob/882ed02142fbf7ba16da9f8efeb31dab8cfa8889/pkg/oc/cli/admin/verifyimagesignature/verify-signature.go#L217 is, when trying to “remove the verification status”, creating an invalid value. That’s a real bug, and it probably prevents using --save in most of the many-different-signatures setups. (I don’t see that this can explain why signatures are not imported at all.) (In reply to Pedro Amoedo from comment #19) > FWIW, it fails with all of them: > > ~~~ > [quicklab@master-1 ~]$ for i in `cat identities_list`; do echo "*** $i ***" > && oc adm verify-image-signature > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > --expected-identity $i --public-key > /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --insecure && echo ""; done > > *** registry.access.redhat.com/rhel7/etcd:3.2.22 *** > image > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > identity is now confirmed (signed by GPG key "199E2F91FD431D51") ^^^ It does not fail; every iteration has one signature matching (and the others rejected). That’s exactly how it is supposed to work. > [quicklab@master-1 ~]$ oc describe istag etcd:latest | grep -A3 Signatures --save was not used, so of course nothing was changed. (But, per comment#24, you are right that --save probably doesn’t work in this situation; it should mark _one_ signature as verified in any of these commands.) (In reply to Pedro Amoedo from comment #19) > FWIW, it fails with all of them: > > ~~~ > [quicklab@master-1 ~]$ for i in `cat identities_list`; do echo "*** $i ***" > && oc adm verify-image-signature > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > --expected-identity $i --public-key > /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --insecure && echo ""; done > > *** registry.access.redhat.com/rhel7/etcd:3.2.22 *** > image > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > identity is now confirmed (signed by GPG key "199E2F91FD431D51") ^^^ It does not fail; every iteration has one signature matching (and the others rejected). That’s exactly how it is supposed to work. > [quicklab@master-1 ~]$ oc describe istag etcd:latest | grep -A3 Signatures --save was not used, so of course nothing was changed. (But, per comment#24, you are right that --save probably doesn’t work in this situation; it should mark _one_ signature as verified in any of these commands.) (In reply to Miloslav Trmač from comment #24) > (In reply to Pedro Amoedo from comment #13) > > (In reply to Miloslav Trmač from comment #12) > > > > 2) "oc adm verify-image-signature" is not able to properly verify 2 > > > > different identities for the same image, it was designed to only provide one > > > > "--expected-identity" value, so verification will "succeed" only on half of > > > > the signatures but fail on the other half, which makes not possible the use > > > > of "--save" option to change the status of those signatures to Verified. > > > > > > Looking at the code, nothing prevents using --save if some of the > > > verifications fail; the “verification status will be removed” text in the > > > failures quoted earlier refers to what would happen with --save. (It is > > > possible that it does actually fail, (I don’t quite see how the status > > > removal code path works), but you haven’t shown any output like that.) > <snip> > > > [quicklab@master-1 ~]$ oc adm verify-image-signature > > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > --expected-identity > > registry.acces.redhat.com/rhel7/etcd@sha256: > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --save > > error verifying signature > > sha256: > > 50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@66950bcce624 > > 56b4b004a7a3f3accc10 for image > > sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec > > (verification status will be removed): failed to get image > > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" > > manifest: Get https://docker-registry.default.svc:5000/v2/: x509: > > certificate signed by unknown authority > … > > error verifying signature > > The Image > > "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" is > > invalid: > > * signatures[0].metadata.name: Required value: name or generateName is > > required > > * signatures[0].metadata.name: Invalid value: "": name must be of format > > <imageName>@<signatureName> > > * signatures[0].type: Required value > > * signatures[0].content: Required value > … > > Thanks, this confirms my suspicion: > https://github.com/openshift/origin/blob/ > 882ed02142fbf7ba16da9f8efeb31dab8cfa8889/pkg/oc/cli/admin/ > verifyimagesignature/verify-signature.go#L217 is, when trying to “remove the > verification status”, creating an invalid value. That’s a real bug, and it > probably prevents using --save in most of the many-different-signatures > setups. (I don’t see that this can explain why signatures are not imported > at all.) Thanks Miloslav for confirming the bug, maybe we can adapt verify-signature code to support multiple "expected-identities" for the same image? or to discard them and only verified the ones matching the "expected-identity? whatever you consider more appropriate. FYI, on my customer's case, the problem is only the manual verification step, they are using fully disconnected environments so the "oc import-image" is not relevant. However, as part of my debugging and in order to match the official documented procedure I thought that was better to raise the BZ mainly focused on that part. I'm still working with Oleg on the "import-image" part to extract more detailed logs and determine the root cause. Thanks you both for the help, much appreciated! Hi again Oleg, apologies for the delay, I preferred to set up a new cluster from scratch to have more clear logs, please find attached controllers_loglevel_5.log.gz file which BTW contains related entries like the following: ~~~ [quicklab@master-0 ~]$ zgrep signature controllers_loglevel_5.log.gz | grep etcd time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1" time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2" time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-3" time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-4" time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-5" time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-6" time="2019-05-09T16:40:10-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-7" time="2019-05-09T16:48:47-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1" time="2019-05-09T16:48:47-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2" time="2019-05-09T16:48:47-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-3" time="2019-05-09T16:48:47-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-4" time="2019-05-09T16:48:47-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-5" time="2019-05-09T16:48:47-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-6" time="2019-05-09T16:48:48-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-7" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-3" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-4" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-5" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-6" time="2019-05-09T16:57:56-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-7" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-3" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-4" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-5" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-6" time="2019-05-09T17:07:39-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-7" time="2019-05-09T17:16:53-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1" time="2019-05-09T17:16:53-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2" time="2019-05-09T17:16:53-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-3" time="2019-05-09T17:16:53-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-4" time="2019-05-09T17:16:53-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-5" time="2019-05-09T17:16:54-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-6" time="2019-05-09T17:16:54-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-7" time="2019-05-09T17:45:34-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1" time="2019-05-09T17:45:35-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2" time="2019-05-09T17:45:35-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-3" time="2019-05-09T17:45:35-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-4" time="2019-05-09T17:45:35-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-5" time="2019-05-09T17:45:35-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-6" time="2019-05-09T17:45:36-04:00" level=debug msg="GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-7" ~~~ This confirms that the image-signature-import controller is working as expected and able to get the proper signatures, but at least on my latest tests (3.11.98), the image stream still doesn't present the signatures attached (even after one hour): ~~~ [quicklab@master-0 ~]$ oc describe istag etcd:latest Image Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec Created: About an hour ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 91.82MB in 3 layers Layers: 75.85MB sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 1.264kB sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 15.96MB sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 Image Created: 3 weeks ago Author: Avesh Agarwal <avagarwa> Arch: amd64 Command: /usr/bin/etcd-env.sh /usr/bin/etcd Working Dir: <none> User: <none> Exposes Ports: 2379/tcp, 2380/tcp, 4001/tcp, 7001/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-04-17T13:04:09.833457 com.redhat.build-host=cpt-0006.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=etcd-container com.redhat.license_terms=https://www.redhat.com/licenses/eulas description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. distribution-scope=public install=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3 io.k8s.description=etcd is a distributed reliable key-value store for the most critical data of a distributed system. io.k8s.display-name=etcd io.openshift.expose-services=2379:tcp,2380:tcp io.openshift.tags=etcd maintainer=Avesh Agarwal name=rhel7/etcd release=30 run=/usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p 2380:2380 --name $NAME $IMAGE $OPT2 $OPT3 summary=A highly-available key value store for shared configuration uninstall=/usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host -e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30 usage=etcd -help vcs-ref=7746b3779792565daa82fd5e33511746bd0be2a2 vcs-type=git vendor=Red Hat, Inc. version=3.2.22 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=docker ~~~ Is there any other function than we can also debug, like for example, the JoinImageStreamImage[1]? or maybe even force a dump of the []imagev1.ImageSignature array to see if the content corresponds with previous downloaded signatures? ~~~ ret := []imagev1.ImageSignature{} for _, blob := range signatures { sig := imagev1.ImageSignature{Type: imageapi.ImageSignatureTypeAtomicImageV1} // This will use the name of the image (sha256:xxxx) and the SHA256 of the // signature itself as the signature name has to be unique for each // signature. sig.Name = imageapi.JoinImageStreamImage(image.Name, fmt.Sprintf("%x", sha256.Sum256(blob))) sig.Content = blob sig.CreationTimestamp = metav1.Now() ret = append(ret, sig) } ~~~ [1] - https://github.com/openshift/origin/blob/c68d654128cc4ec776a183d20db1d24b51db07d5/pkg/image/controller/signature/container_image_downloader.go Regards. This doesn't look like logs from controllers. They should be like this: I0917 15:56:22.462982 1 controller_manager.go:134] Started "openshift.io/image-signature-import" I0917 15:56:22.492234 1 controller_manager.go:134] Started "openshift.io/unidling" I0917 15:56:22.506170 1 controller_manager.go:134] Started "openshift.io/ingress-ip" ... I0917 15:56:25.285509 1 signature_import_controller.go:130] Initiating download of signatures for sha256:8263a1b234bb459167601a504cbfacd05228bfbe7a2eab91d25b4c39fbb152ea I0917 15:56:25.770804 1 signature_import_controller.go:151] No signatures downloaded for sha256:8263a1b234bb459167601a504cbfacd05228bfbe7a2eab91d25b4c39fbb152ea I0917 15:56:25.851197 1 signature_import_controller.go:130] Initiating download of signatures for sha256:ad997a6644e1f270ec67c4b85a911bfcce046fd94b7435eaecdb20118cbcaaa2 I0917 15:56:26.317562 1 signature_import_controller.go:151] No signatures downloaded for sha256:ad997a6644e1f270ec67c4b85a911bfcce046fd94b7435eaecdb20118cbcaaa2 I0917 15:56:26.464964 1 signature_import_controller.go:130] Initiating download of signatures for sha256:d7177a11a2e74945883f3338d06e3e02be37db278f8d57e0c19f99af19dce092 Have you tried https://access.redhat.com/articles/3663751 ? To get all stored information about the image you can use `oc get image sha256:XXXXXX -o yaml` Hi Oleg, that article is basically the same I've done to gather the logs manually, my steps were:
~~~
# sed -i -e 's/DEBUG_LOGLEVEL=2/DEBUG_LOGLEVEL=5/g' /etc/origin/master/master.env
# master-restart api
# master-restart controllers
# oc import-image .....
# master-logs controllers controllers 2>&1 | tee controllers_loglevel_5.log
~~~
However, I have tried again exactly as the article does, redirecting only stderr, and now it seems we have the expected signature_import_controller entries, PFA again, my apologies.
Regarding the output of "oc get image sha256:XXXXXX -o yaml", here you have it:
~~~
[quicklab@master-0 ~]$ oc get image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec -o yaml
apiVersion: image.openshift.io/v1
dockerImageLayers:
- mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
name: sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526
size: 75854078
- mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
name: sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2
size: 1264
- mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
name: sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78
size: 15964893
dockerImageManifestMediaType: application/vnd.docker.distribution.manifest.v2+json
dockerImageMetadata:
Architecture: amd64
Author: Avesh Agarwal <avagarwa>
Config:
Cmd:
- /usr/bin/etcd-env.sh
- /usr/bin/etcd
Env:
- PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
- container=docker
ExposedPorts:
2379/tcp: {}
2380/tcp: {}
4001/tcp: {}
7001/tcp: {}
Hostname: ed75d4430067
Image: 83868aa442ebe30c9a5678d594197e9e521420f414958d7fb9894ea496e38518
Labels:
architecture: x86_64
authoritative-source-url: registry.access.redhat.com
build-date: 2019-04-17T13:04:09.833457
com.redhat.build-host: cpt-0006.osbs.prod.upshift.rdu2.redhat.com
com.redhat.component: etcd-container
com.redhat.license_terms: https://www.redhat.com/licenses/eulas
description: etcd is a distributed reliable key-value store for the most critical
data of a distributed system.
distribution-scope: public
install: /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host
-e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3
io.k8s.description: etcd is a distributed reliable key-value store for the most
critical data of a distributed system.
io.k8s.display-name: etcd
io.openshift.expose-services: 2379:tcp,2380:tcp
io.openshift.tags: etcd
maintainer: Avesh Agarwal
name: rhel7/etcd
release: "30"
run: /usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p
2380:2380 --name $NAME $IMAGE $OPT2 $OPT3
summary: A highly-available key value store for shared configuration
uninstall: /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host
-e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3
url: https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30
usage: 'etcd -help '
vcs-ref: 7746b3779792565daa82fd5e33511746bd0be2a2
vcs-type: git
vendor: Red Hat, Inc.
version: 3.2.22
ContainerConfig:
Cmd:
- /bin/sh
- -c
- rm -f '/etc/yum.repos.d/extras-latest-7.6.z-6c3cc.repo' '/etc/yum.repos.d/odcs-92808-3b88b.repo'
Env:
- PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
- container=docker
ExposedPorts:
2379/tcp: {}
2380/tcp: {}
4001/tcp: {}
7001/tcp: {}
Hostname: ed75d4430067
Image: sha256:0a987feadd3ebb0195490fe507343bc66621848a38633335583771cb4b109095
Labels:
architecture: x86_64
authoritative-source-url: registry.access.redhat.com
build-date: 2019-04-17T13:04:09.833457
com.redhat.build-host: cpt-0006.osbs.prod.upshift.rdu2.redhat.com
com.redhat.component: etcd-container
com.redhat.license_terms: https://www.redhat.com/licenses/eulas
description: etcd is a distributed reliable key-value store for the most critical
data of a distributed system.
distribution-scope: public
install: /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host
-e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/install.sh $OPT3
io.k8s.description: etcd is a distributed reliable key-value store for the most
critical data of a distributed system.
io.k8s.display-name: etcd
io.openshift.expose-services: 2379:tcp,2380:tcp
io.openshift.tags: etcd
maintainer: Avesh Agarwal
name: rhel7/etcd
release: "30"
run: /usr/bin/docker run -d $OPT1 -p 4001:4001 -p 7001:7001 -p 2379:2379 -p
2380:2380 --name $NAME $IMAGE $OPT2 $OPT3
summary: A highly-available key value store for shared configuration
uninstall: /usr/bin/docker run --rm $OPT1 --privileged -v /:/host -e HOST=/host
-e NAME=$NAME -e IMAGE=$IMAGE $IMAGE $OPT2 /usr/bin/uninstall.sh $OPT3
url: https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/etcd/images/3.2.22-30
usage: 'etcd -help '
vcs-ref: 7746b3779792565daa82fd5e33511746bd0be2a2
vcs-type: git
vendor: Red Hat, Inc.
version: 3.2.22
Created: 2019-04-17T13:04:43Z
DockerVersion: 1.13.1
Id: sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635
Size: 91824976
apiVersion: "1.0"
kind: DockerImage
dockerImageMetadataVersion: "1.0"
dockerImageReference: registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec
kind: Image
metadata:
annotations:
image.openshift.io/dockerLayersOrder: ascending
creationTimestamp: 2019-05-10T14:22:00Z
name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec
resourceVersion: "146211"
selfLink: /apis/image.openshift.io/v1/images/sha256%3A50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec
uid: f96d5eaf-732e-11e9-af67-fa163e421b5d
~~~
Regards.
Thanks, Pedro. > I0509 16:40:10.910197 1 signature_import_controller.go:176] Image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec reached signature limit (max:3, want:6) … and the controller does not save any of the signatures, even the 3 that would fit into the limit, in that case. (It can’t decide which 3 of the 6 are the most important anyway). The limit of 3 is currently hard-coded in https://github.com/openshift/origin/blob/2899b4531c11d6e6a74c9ce1aa3dbfd367aa9090/pkg/cmd/openshift-controller-manager/controller/image.go#L146 . (In reply to Miloslav Trmač from comment #36) > Thanks, Pedro. > > > I0509 16:40:10.910197 1 signature_import_controller.go:176] Image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec reached signature limit (max:3, want:6) > > … and the controller does not save any of the signatures, even the 3 that > would fit into the limit, in that case. (It can’t decide which 3 of the 6 > are the most important anyway). > > The limit of 3 is currently hard-coded in > https://github.com/openshift/origin/blob/ > 2899b4531c11d6e6a74c9ce1aa3dbfd367aa9090/pkg/cmd/openshift-controller- > manager/controller/image.go#L146 . Thanks Miloslav, good catch, I couldn't imagine that we had a hardcoded limit for that, I'd have sworn to have seen images with more signatures attached on the past. Anyway, if there's anything I could do to expedite those bugs resolution, please let me know. Regards. Just to summarize it sounds like there are two bugs/limitations here: 1) A limit of 3 signatures on import and when this limit is exceeded, no signatures are imported. Limit should be higher and at least the signatures up to the limit should be imported when the limit is exceeded. 2) Inability to verify signatures when signatures from multiple identities exist. Should be able to either a) pass multiple expected identities or b) --save the validation for the signatures that could be verified and leave the others alone. Pedro, thank you for all your investigation/debugging/reproduction efforts! One more bug: 3) https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore says that --expected-identity can be found by describing the image stream, which is not in general the case. if we're piling on usability issues then: 4) output when we're only able to verify a subset of the signatures is borderline unreadable/unusable. At a minimum it needs some newlines in between the output for each signature verification result. Hi all, would be possible to obtain some estimation about the pending fix? Thanks and regards. As I can see, there are a lot of problems with signatures, and they won't be fixed at the same time. So let's use this bug as an umbrella issue. > 1) A limit of 3 signatures on import and when this limit is exceeded, no signatures are imported. Limit should be higher and at least the signatures up to the limit should be imported when the limit is exceeded. I created a separate bug to increase the limit: https://bugzilla.redhat.com/show_bug.cgi?id=1722568 > 2) Inability to verify signatures when signatures from multiple identities exist. Should be able to either a) pass multiple expected identities or b) --save the validation for the signatures that could be verified and leave the others alone. So, I see two use cases for `oc adm verify-image-signature`: 1. simulate the process of verification that happen in Docker to debug signatures, 2. set the "verified" flag in the image resource. The first one should be replaced with real pulling images from the registry. There is no reasons have signatures for access.registry.redhat.com/* and registry.redhat.io/* references in our database, because these signatures will be served from the integrated registry which has a different hostname (and therefore these signatures are not valid for the registry). And even if `oc adm verify-image-signature` says that the signature is valid, the real push most likely will fail. For the second one: who uses this flag? When this flag should be switched back to "false"? > 3) https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore says that --expected-identity can be found by describing the image stream, which is not in general the case. Are there any better recommendations where to get the value from? If the image was imported from registry.redhat.io/foo/bar, this Docker reference will be in the image stream. (In reply to Oleg Bulatov from comment #42) > > 2) Inability to verify signatures when signatures from multiple identities exist. Should be able to either a) pass multiple expected identities or b) --save the validation for the signatures that could be verified and leave the others alone. > > So, I see two use cases for `oc adm verify-image-signature`: > > 1. simulate the process of verification that happen in Docker to debug > signatures, > 2. set the "verified" flag in the image resource. > > The first one should be replaced with real pulling images from the registry. > There is no reasons have signatures for access.registry.redhat.com/* and > registry.redhat.io/* references in our database, because these signatures > will be served from the integrated registry which has a different hostname > (and therefore these signatures are not valid for the registry). The signatures exist to provide _end-to-end_ verification; the node can verify the original RH-created signature for access.registry.redhat.com. RH obviously won’t sign images with the hostname of the integrated registry, and a cluster-created signature for the integrated registry wouldn’t be by RH. So, the registry should store and provide the original signatures with the original identities. Nodes can configure policy.json to expect the appropriate identity for every repository. Yes, it’s a hassle. (All of this is not to say that `oc adm verify-image-signature` is necessarily worth recommending for “simulating the verification”; I’d just use podman pull / crictl pull / … directly on the appropriately configured node, to actually test the node’s policy.json instead of the pretty fake environment internally used by `oc adm verify-image-signature`. But “there is no reason to have [the original] signatures” is very much missing the point of having signatures.) > > 3) https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore says that --expected-identity can be found by describing the image stream, which is not in general the case. > > Are there any better recommendations where to get the value from? If the > image was imported from registry.redhat.io/foo/bar, this Docker reference > will be in the image stream. That’s not what the dump above shows: > Docker Pull Spec: docker-registry.default.svc:5000/test/etcd nor what the example in the documentation shows: > Docker Pull Spec: 172.30.1.1:5000/openshift/nodejs Both may be incorrect, I haven’t checked. (In reply to Miloslav Trmač from comment #43) > (In reply to Oleg Bulatov from comment #42) > > > 2) Inability to verify signatures when signatures from multiple identities exist. Should be able to either a) pass multiple expected identities or b) --save the validation for the signatures that could be verified and leave the others alone. > > > > So, I see two use cases for `oc adm verify-image-signature`: > > > > 1. simulate the process of verification that happen in Docker to debug > > signatures, > > 2. set the "verified" flag in the image resource. > > > > The first one should be replaced with real pulling images from the registry. > > There is no reasons have signatures for access.registry.redhat.com/* and > > registry.redhat.io/* references in our database, because these signatures > > will be served from the integrated registry which has a different hostname > > (and therefore these signatures are not valid for the registry). > > The signatures exist to provide _end-to-end_ verification; the node can > verify the original RH-created signature for access.registry.redhat.com. RH > obviously won’t sign images with the hostname of the integrated registry, > and a cluster-created signature for the integrated registry wouldn’t be by > RH. So, the registry should store and provide the original signatures with > the original identities. > > Nodes can configure policy.json to expect the appropriate identity for every > repository. Yes, it’s a hassle. > > (All of this is not to say that `oc adm verify-image-signature` is > necessarily worth recommending for “simulating the verification”; I’d just > use podman pull / crictl pull / … directly on the appropriately configured > node, to actually test the node’s policy.json instead of the pretty fake > environment internally used by `oc adm verify-image-signature`. But “there > is no reason to have [the original] signatures” is very much missing the > point of having signatures.) > > > > 3) https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore says that --expected-identity can be found by describing the image stream, which is not in general the case. > > > > Are there any better recommendations where to get the value from? If the > > image was imported from registry.redhat.io/foo/bar, this Docker reference > > will be in the image stream. > > That’s not what the dump above shows: > > Docker Pull Spec: docker-registry.default.svc:5000/test/etcd > nor what the example in the documentation shows: > > Docker Pull Spec: 172.30.1.1:5000/openshift/nodejs > > Both may be incorrect, I haven’t checked. My 2 cents regarding point 3) Both examples from documentation are incorrect, "--expected-identity" should always be (for official RedHat images) "registry.access.redhat.com" or "registry.redhat.io", depending on the set of signatures that you want to verify (once the "oc adm verify-image-signature" command allows multiple sets, of course). Also, could be a good example in terms of documentation, the following info extracted from the istag, all of our docker images contains the same "authoritative-source-url" and the docker image string is exactly the one that varies according to the repository: ~~~ [root@ocp-ocr ~]# oc describe istag etcd:latest | grep "Docker Image\|authoritative" Docker Image: registry.redhat.io/rhel7/etcd@sha256:bc5ef0f75b94086479b4029da9628c21d8cd8a1eb60ec88730cbd82c727e415d authoritative-source-url=registry.access.redhat.com ~~~ Let me show you a real working example with an old image that only contains the old set of 3 signatures: ~~~ [root@ocp-ocr ~]# oc import-image rhel7/etcd --from=registry.access.redhat.com/rhel7/etcd:3.2.22-28 --confirm The import completed successfully. Name: etcd Namespace: imported Created: Less than a second ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2019-05-09T20:00:46Z Docker Pull Spec: docker-registry.default.svc:5000/imported/etcd Image Lookup: local=false Unique Images: 1 Tags: 1 latest tagged from registry.access.redhat.com/rhel7/etcd:3.2.22-28 * registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Less than a second ago Image Name: etcd:3.2.22-28 Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a [...] [root@ocp-ocr ~]# oc describe istag etcd:latest | grep -A1 "Docker Image\|authoritative\|Signatures" Docker Image: registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a -- Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@79b327ec1b3b48744d8d7a4b7f88b1f3322af19bf06bf93f8842fbf59eaf148d -- Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@f0550dd7edbeddcb36b60644a4668acb1862018e539319ecf7c69101a05bfdb5 -- Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@babd9cd80f0022e16fca51a90ee9a3ec0f8155321fa1d51477ae46081b0f9d34 -- authoritative-source-url=registry.access.redhat.com build-date=2019-03-06T14:49:26.310415 [root@ocp-ocr ~]# oc adm verify-image-signature --expected-identity `oc describe istag etcd:latest | grep "Docker Image:" | cut -f3` `oc describe istag etcd:latest | grep "Name:" | cut -f3` --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") ~~~ Regards. (In reply to Pedro Amoedo from comment #44) > Also, could be a good example in terms of documentation, the following info > extracted from the istag, all of our docker images contains the same > "authoritative-source-url" and the docker image string is exactly the one > that varies according to the repository: Do I understand correctly that authoritative-source-url is a label contained in the image? If so, it is insecure to rely on it and it must not be used: that would allow a malicious image to choose its own identity. Pedro, can you elaborate on use cases of `oc adm verify-image-signature`? Is it to debug that images may eventually be deployed on nodes or to have green labels in the web console? I'm trying to figure out what are expectations for this utility. (In reply to Miloslav Trmač from comment #45) > (In reply to Pedro Amoedo from comment #44) > > Also, could be a good example in terms of documentation, the following info > > extracted from the istag, all of our docker images contains the same > > "authoritative-source-url" and the docker image string is exactly the one > > that varies according to the repository: > Do I understand correctly that authoritative-source-url is a label contained > in the image? If so, it is insecure to rely on it and it must not be used: > that would allow a malicious image to choose its own identity. Hi Miloslav, see my responses below: > Do I understand correctly that authoritative-source-url is a label contained > in the image? If so, it is insecure to rely on it and it must not be used: AFAIK, that string is a docker label contained on every image from RedHat but I can't provide you more detail, this is out of my scope. > that would allow a malicious image to choose its own identity. I suppose that precisely for that reason we have the option to verify the signatures of the image, right? Regards. (In reply to Oleg Bulatov from comment #47) > Pedro, can you elaborate on use cases of `oc adm verify-image-signature`? Is > it to debug that images may eventually be deployed on nodes or to have green > labels in the web console? > > I'm trying to figure out what are expectations for this utility. Hi Oleg, let me explain: I have a customer that requires the maximum level of security, that's the reason they use completely disconnected environments. They have an isolated standalone registry that provides the images to the rest of the isolated clusters, in order to load the images on this registry, they need a proper method to manually push the images and verify them after that to corroborate the integrity. I already have a documented solution here: https://access.redhat.com/solutions/4039981, pending now on the new bug fixes. Apart from that, please note that by default, the docker daemon packed with OCP versions doesn't force a signature verification: ~~~ [root@ocp-ocr ~]# cat /etc/sysconfig/docker | grep OPTIONS OPTIONS=' --selinux-enabled --log-driver=journald --signature-verification=False' ~~~ And as you already know, we don't even download the signatures with "oc import-image" by default unless it's manually configured first the sigstore for each repository under "/etc/containers/registries.d/". For all of that reasons (and maybe others), I believed that "oc adm verify-image-signature" is a necessary command. Regards. (In reply to Pedro Amoedo from comment #48) > (In reply to Miloslav Trmač from comment #45) > > (In reply to Pedro Amoedo from comment #44) > > > Also, could be a good example in terms of documentation, the following info > > > extracted from the istag, all of our docker images contains the same > > > "authoritative-source-url" and the docker image string is exactly the one > > > that varies according to the repository: > > > > Do I understand correctly that authoritative-source-url is a label contained > > in the image? If so, it is insecure to rely on it and it must not be used: > > AFAIK, that string is a docker label contained on every image from RedHat > but I can't provide you more detail, this is out of my scope. > > > that would allow a malicious image to choose its own identity. > > I suppose that precisely for that reason we have the option to verify the > signatures of the image, right? The expected identity is an _input_ to verifying the image; you can't use anything the image claims about itself until that verification happens. So, reading a label from an image and using its value as an input to --expected-identity completely defeats the point. (In reply to Miloslav Trmač from comment #50) > (In reply to Pedro Amoedo from comment #48) > > (In reply to Miloslav Trmač from comment #45) > > > (In reply to Pedro Amoedo from comment #44) > > > > Also, could be a good example in terms of documentation, the following info > > > > extracted from the istag, all of our docker images contains the same > > > > "authoritative-source-url" and the docker image string is exactly the one > > > > that varies according to the repository: > > > > > > Do I understand correctly that authoritative-source-url is a label contained > > > in the image? If so, it is insecure to rely on it and it must not be used: > > > > AFAIK, that string is a docker label contained on every image from RedHat > > but I can't provide you more detail, this is out of my scope. > > > > > that would allow a malicious image to choose its own identity. > > > > I suppose that precisely for that reason we have the option to verify the > > signatures of the image, right? > > The expected identity is an _input_ to verifying the image; you can't use > anything the image claims about itself until that verification happens. So, > reading a label from an image and using its value as an input to > --expected-identity completely defeats the point. Hi Miloslav, it was just an example of what string should be used for "--expected-identity" value in terms of documentation, the signature is always the key, the identity is also hardcoded there and the verification will fail if you don't match it, for example, here you have an image manually downloaded with skopeo and pushed again to an standalone registry which modifies the docker labels: ~~~ [root@ocp-ocr ~]# oc describe istag etcd:3.2.22-28 | grep "Docker Image:" | cut -f3 docker-registry.default.svc:5000/imported/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a [root@ocp-ocr ~]# oc adm verify-image-signature --expected-identity docker-registry.default.svc:5000/imported/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release error verifying signature sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@374d76803f914165a828534748d665d5 for image sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted error verifying signature sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@1f7401996e1b33788b71d911cdef04f7 for image sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22-28 is not accepted error verifying signature sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@bd319a3c54f4311e2914d3e1b37a552c for image sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a (verification status will be removed): signature rejected: Signature for identity registry.access.redhat.com/rhel7/etcd:latest is not accepted ~~~ As you can see, the verification command fails stating that my provided identity is not the one expected by the signatures: ~~~ Signature for identity registry.access.redhat.com/rhel7/etcd:3.2.22 is not accepted" ~~~ Regards. (In reply to Pedro Amoedo from comment #51) > Hi Miloslav, it was just an example of what string should be used for > "--expected-identity" value in terms of documentation No, it must not be documented like that. >, the signature is > always the key, the identity is also hardcoded there and the verification > will fail if you don't match it That’s not good enough. The --expected-identity value is used to select what public keys, IF ANY, and even what identity matching rules should be applied. The signature verification _starts_ with the USER-SUPPLIED expected identity, and only LATER involves cryptography. Consider a policy.json like the one in https://github.com/containers/image/blob/master/docs/containers-policy.json.5.md , where images with the docker.io/library/busybox identity are accepted without requiring any signature. In that case, an attacker could substitute the legitimate etcd image with a malicious one, with a "authoritative-source-url: docker.io/library/busybox" label, and a configuration that used the expected identity from inside the label would therefore end up using the "docker.io/library/busybox: accept anything without signatures" policy, regardless of the users’ intent to use an etcd image that was signed by RH. True, `oc adm verify-image-signature` does not currently use `policy.json`, but that approach is still wrong and dangerous. (In reply to Miloslav Trmač from comment #52) > (In reply to Pedro Amoedo from comment #51) > > Hi Miloslav, it was just an example of what string should be used for > > "--expected-identity" value in terms of documentation > > No, it must not be documented like that. > > >, the signature is > > always the key, the identity is also hardcoded there and the verification > > will fail if you don't match it > > That’s not good enough. The --expected-identity value is used to select what > public keys, IF ANY, and even what identity matching rules should be > applied. The signature verification _starts_ with the USER-SUPPLIED > expected identity, and only LATER involves cryptography. > > Consider a policy.json like the one in > https://github.com/containers/image/blob/master/docs/containers-policy.json. > 5.md , where images with the docker.io/library/busybox identity are accepted > without requiring any signature. In that case, an attacker could substitute > the legitimate etcd image with a malicious one, with a > "authoritative-source-url: docker.io/library/busybox" label, and a > configuration that used the expected identity from inside the label would > therefore end up using the "docker.io/library/busybox: accept anything > without signatures" policy, regardless of the users’ intent to use an etcd > image that was signed by RH. > > > True, `oc adm verify-image-signature` does not currently use `policy.json`, > but that approach is still wrong and dangerous. Thanks Miloslav, I understand your point with the "docker.io/library/busybox" example, but with our default policy configuration I don't think it matches the current situation: ~~~ $ cat /etc/containers/policy.json { "default": [ { "type": "insecureAcceptAnything" } ], "transports": { "docker-daemon": { "": [{"type":"insecureAcceptAnything"}] } } } $ grep OPTIONS /etc/sysconfig/docker OPTIONS=' --selinux-enabled --signature-verification=False' ~~~ Correct me if I wrong, but if the current value of using "--expected-identity" is used to select what public keys should be verified, and that is precisely the part that is currently failing when having multiple sets of signatures, what about removing the human part of selecting the identity and try to automate the verification based only on the signature content and the public key? something like this? ~~~ $ oc adm verify-image-signature IMAGE --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release ~~~ On this manner if there are multiple sets of signatures (as we currently have with registry.redhat.io), the algorithm will just have to try to verify all of them against their own identity (expected-identity = signature-identity?) and don't rely on external input. NOTE: please take in consideration that I haven't digged further into the source code to know if this is viable, it's just an idea. Best Regards. (In reply to Pedro Amoedo from comment #53) > (In reply to Miloslav Trmač from comment #52) > > (In reply to Pedro Amoedo from comment #51) > > > Hi Miloslav, it was just an example of what string should be used for > > > "--expected-identity" value in terms of documentation > > > > No, it must not be documented like that. > > > > >, the signature is > > > always the key, the identity is also hardcoded there and the verification > > > will fail if you don't match it > > > > That’s not good enough. The --expected-identity value is used to select what > > public keys, IF ANY, and even what identity matching rules should be > > applied. The signature verification _starts_ with the USER-SUPPLIED > > expected identity, and only LATER involves cryptography. … > Correct me if I wrong, but if the current value of using > "--expected-identity" is used to select what public keys should be verified, It’s not _only_ that. It’s also to ensure that the user got the image that the user wanted to pull. > and that is precisely the part that is currently failing when having > multiple sets of signatures, what about removing the human part of selecting > the identity and try to automate the verification based only on the > signature content and the public key? something like this? > > ~~~ > $ oc adm verify-image-signature IMAGE --public-key > /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release > ~~~ > > On this manner if there are multiple sets of signatures (as we currently > have with registry.redhat.io), the algorithm will just have to try to verify > all of them against their own identity (expected-identity = > signature-identity?) and don't rely on external input. That would allow an attacker to substitute a years-old, but correctly RH-sigbned, rhel7:7.0.0 image, with many known vulnerabilities fixed in later releases, when the user wanted to use rhel50:50.7.3 released yesterday. No, the identity must express the user’s _intent_; no part of it may be derived from the untrusted image data that is yet to be validated. Ben, This issue was given as a possible blocker for double signing of container images for both registries, see: https://projects.engineering.redhat.com/browse/DELIVERY-6699 Can we get this prioritized for for OCP 4.2? Specifically this part: 2) Inability to verify signatures when signatures from multiple identities exist. Should be able to either a) pass multiple expected identities or b) --save the validation for the signatures that could be verified and leave the others alone. As #1722568 seems to be verified in OCP 4.2, which I guess is a blocker for this issue. Ideally it would also be backported to 4.1 and 3.11. Per comment 38-40 there are a number of issues here. Which aspect are you looking to have prioritized? Note also that there is a BZ targeted at 4.2 for part of this problem: https://bugzilla.redhat.com/show_bug.cgi?id=1722568 If https://bugzilla.redhat.com/show_bug.cgi?id=1722568 doesn't cover your concern, we should create additional clones of this BZ, targeted at the releases you want, with specific explanation of which signature improvements/fixes are needed, since again, there are like 4 different improvements described in this BZ. Hi Ben, Please prioritise the bug describe in comment #c24. Probably best if we can just --save the validation if any signatures match the expected identity. Results of the --save should be that 'oc describe istag <is-name>' shows 'Signature Status' as 'verified'. Pedro, would that satisfy your customers requirements if the signature limit fix in OCP 4.2 BZ#1722568 was also backported to OCP 3.11? Regards, Jason (In reply to Jason Shepherd from comment #62) > Hi Ben, > > Please prioritise the bug describe in comment #c24. Probably best if we can > just --save the validation if any signatures match the expected identity. > Results of the --save should be that 'oc describe istag <is-name>' shows > 'Signature Status' as 'verified'. > > Pedro, would that satisfy your customers requirements if the signature limit > fix in OCP 4.2 BZ#1722568 was also backported to OCP 3.11? > > Regards, > Jason Hi Jason, IMHO, that solution would be perfect if we also allow the command to be run multiple times with different expected identities and do not remove the previous verified signatures, as per current design, if any signature fails to be verified, that status will be removed[1]. Maybe we can just add a previous check that skips the signature verification if already trusted? BTW, please note that version 3.11.129 already contain the limit fix (it was incremented to 10)[2], fixed via BZ#1722581 & pushed by RHBA-2019:1753[3] [1] - https://github.com/openshift/origin/blob/release-3.11/pkg/oc/cli/admin/verifyimagesignature/verify-signature.go#L217 [2] - https://github.com/openshift/origin/pull/23125/files [3] - https://access.redhat.com/errata/RHBA-2019:1753 Best Regards. Jason, i have broken out the bug described in comment 24 to its own BZ here: https://bugzilla.redhat.com/show_bug.cgi?id=1748812 the BZ is currently targeted for 4.2 (but Oleg will have to evaluate). If you need it backported to 4.1 or 3.11, please comment on that BZ. Our existing tool verify-image-signature is supposed to set boolean the "Valid" boolean flag on image objects. It wasn't a good idea: * A signature may expire or be revoked, but existing API doesn't take this into account and images will stay "Valid". That may be misleading. * Mirroring policy affects how the signature should be verified and this policy may change over time, but verify-image-signature doesn't know about it. * Policies (registries.conf) exist on cluster nodes and they are not available to verify-image-signature that is supposed to be run on a client machine. * verify-image-signature expects a provided expected-identity that customers tend to misunderstand. Verifier should work with imagestreams, not images, to know real image references. So signatures are supposed to be linked to imagestreams, not image objects. Overall, on image API side signatures are fairly broken, they work well on containers runtime side. As Red Hat with other companies is working on the next generation signatures - Notary v2 - they are our main focus, for existing signatures only vulnerabilities will be fixed on image API side. To avoid regressions after upgrades from 3.x to 4.x, we won't fix it in 3.x. |
Description of problem: Documented steps [1] for automatic signature download within "oc import-image" command are no longer working. Version-Release number of selected component (if applicable): OCP 3.x How reproducible: Always with more recent images, it still works with old ones (see below). Steps to Reproduce: 1. Set the proper sigstore as documented [1] within /etc/containers/registries.d/, for example: # cat /etc/containers/registries.d/registry.redhat.io.yaml docker: registry.redhat.io: sigstore: https://registry.redhat.io/containers/sigstore # cat /etc/containers/registries.d/registry.access.redhat.com.yaml docker: registry.access.redhat.com: sigstore: https://access.redhat.com/webassets/docker/content/sigstore 2. Import any latest image, for example: # oc import-image registry.redhat.io/rhel7/etcd:latest --confirm OR # oc import-image registry.access.redhat.com/rhel7/etcd:latest --confirm NOTE: Obviously, for new registry.redhat.io you must be logged in and properly set the corresponding secret for pull operation [2]. 3. Check istag and the image signatures presence (if any): # oc get istag NAME DOCKER REF UPDATED IMAGENAME etcd:latest registry.redhat.io/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 9 seconds ago sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec # oc describe istag etcd:latest | grep -A3 Signatures (void) Actual results: Image signatures are no longer being downloaded for new registry.redhat.io neither for the legacy registry.access.redhat.com, however, if you manually download them with skopeo commands you can see that new images have more signatures associated than previous versions: [pamoedo@p50 mirror] $ skopeo --debug copy docker://registry.access.redhat.com/rhel7/etcd:latest dir:/home/pamoedo/mirror/etcd DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0000] Using "docker" namespace registry.access.redhat.com DEBU[0000] Using https://access.redhat.com/webassets/docker/content/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com DEBU[0000] GET https://registry.access.redhat.com/v2/ DEBU[0000] Ping https://registry.access.redhat.com/v2/ err <nil> DEBU[0000] Ping https://registry.access.redhat.com/v2/ status 200 DEBU[0000] GET https://registry.access.redhat.com/v2/rhel7/etcd/manifests/latest DEBU[0001] Source is a manifest list; copying (only) instance sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec DEBU[0001] GET https://registry.access.redhat.com/v2/rhel7/etcd/manifests/sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec DEBU[0002] IsRunningImageAllowed for image docker:registry.access.redhat.com/rhel7/etcd:latest DEBU[0002] Using transport "docker" specific policy section registry.access.redhat.com DEBU[0002] GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-1 DEBU[0002] GET https://access.redhat.com/webassets/docker/content/sigstore/rhel7/etcd@sha256=50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec/signature-2 [...] [pamoedo@p50 mirror] $ ls etcd/ 96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 manifest.json signature-4 a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 signature-1 signature-5 d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 signature-2 signature-6 d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 signature-3 version Optionally, if you try to push this image to a registry and manually verify the signatures, you'll see unexpected conrtadictory repository errors (those same commands were working for me 2 weeks ago with previous image versions): [mirror] $ skopeo copy --dest-tls-verify=false --dest-creds openshift:<token> dir:/home/pamoedo/mirror/etcd atomic:<docker-registry-default_fqdn>/imported/etcd:latest Getting image source signatures Checking if image destination supports signatures Copying blob sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 72.34 MB / 72.34 MB [======================================================] 6s Copying blob sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 1.23 KB / 1.23 KB [========================================================] 0s Copying blob sha256:96801660c3bbdb1cb819262b1967f67e80827d7ef939ae133ed45ab67faaae78 15.23 MB / 15.23 MB [======================================================] 0s Copying config sha256:d636cc8689ea7ebec89983e49330e2b7b30d7b24feae8944894a82539650e635 4.63 KB / 4.63 KB [========================================================] 0s Writing manifest to image destination Storing signatures [ocp-ocr]# oc get is NAME DOCKER REPO TAGS UPDATED etcd docker-registry.default.svc:5000/imported/etcd latest About a minute ago [ocp-ocr]# oc get istag NAME DOCKER REF UPDATED IMAGENAME etcd:latest docker-registry.default.svc:5000/imported/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec 2 minutes ago sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec [ocp-ocr]# oc describe istag etcd:latest | grep -A3 Signatures Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@e0b80d2b2f737b185e509f8fe2c26c1f Type: atomic Status: Unverified Image Signatures: Name: sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@7a66998677978f6b9985e86304bede59 Type: atomic Status: Unverified [...] [ocp-ocr]# oc adm verify-image-signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --expected-identity registry.access.redhat.com/rhel7/etcd@sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@7a66998677978f6b9985e86304bede59 for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22 is not accepted image "sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec" identity is now confirmed (signed by GPG key "199E2F91FD431D51") error verifying signature sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec@eb06e3c03f0b182972dbd9626500596a for image sha256:50a4f6067bd3564b627e568d3a6d5bcb1235879ca4b29c2f9ba0c7283a8b98ec (verification status will be removed): signature rejected: Signature for identity registry.redhat.io/rhel7/etcd:3.2.22-30 is not accepted [...] Expected results: To have the image signatures in place and be able to verify them, which, as you can as follows, is still working for previous image versions: # oc import-image registry.access.redhat.com/rhel7/etcd:3.2.22-28 --confirm [...] # oc get istag NAME DOCKER REF UPDATED IMAGENAME etcd:3.2.22-28 registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a About a minute ago sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a # oc describe istag etcd:3.2.22-28 | grep -A3 Signatures Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@79b327ec1b3b48744d8d7a4b7f88b1f3322af19bf06bf93f8842fbf59eaf148d Type: AtomicImageV1 Status: Unverified Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@f0550dd7edbeddcb36b60644a4668acb1862018e539319ecf7c69101a05bfdb5 Type: AtomicImageV1 Status: Unverified Image Signatures: Name: sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a@babd9cd80f0022e16fca51a90ee9a3ec0f8155321fa1d51477ae46081b0f9d34 Type: AtomicImageV1 Status: Unverified # oc adm verify-image-signature sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a --expected-identity registry.access.redhat.com/rhel7/etcd@sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") image "sha256:ccde0f53715b9243c8f607d2cd19c1dda90e39c5e093c75118eb4fcee159219a" identity is now confirmed (signed by GPG key "199E2F91FD431D51") References: [1] - https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html#importing-signatures-from-sigstore [2] - https://access.redhat.com/RegistryAuthentication#allowing-pods-to-reference-images-from-other-secured-registries-9 Additional info: There is also an error in the documentation related with "oc adm verify-image-signature" command, it states that "expected-identity" should be extracted from "Docker Pull Spec" value from imagestream description, but once imported, that value will always be the local registry and not the original identity that should be verified against the signature: $ oc adm verify-image-signature <image> --expected-identity=<pull_spec> [--save] [options]