Bug 1486272
| Summary: | 3.6 Images not found after upgrade (causing CrashLoopBackOff) | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Thom Carlin <tcarlin> |
| Component: | Release | Assignee: | Justin Pierce <jupierce> |
| Status: | CLOSED ERRATA | QA Contact: | liujia <jiajliu> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.6.0 | CC: | akostadi, aos-bugs, aweiteka, bparees, jhonce, jokerman, mitr, mmccomas, rhartman, rmcgover, sgraf, smunilla, ssampat, tcarlin |
| Target Milestone: | --- | ||
| Target Release: | 3.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: |
undefined
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-01-23 17:57:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1724792 | ||
|
Description
Thom Carlin
2017-08-29 10:40:45 UTC
Can you please provide `oc describe pod` on a pot that's failing? We need to understand which exact image it's looking for and not finding. Insecure Registries: Registries: registry.access.redhat.com (secure), registry.access.redhat.com (secure), docker.io (secure) Also collaborated with Scott through IRC: 1) System is fully subscribed 2) rpm -q docker is the same on all systems (docker-1.12.6-48.git0fdc778.el7.x86_64) 3) https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/openshift3/ose-sti-builder has deprecated flag # atomic trust show * (default) accept registry.access.redhat.com signed security,security Aaaron does signing affect the ability to pull this image in such a manner that it'd get authentication denied? The issue is you require a signature to be pulled to validate the image but that signature isn't there for the ose-sti-builder image. This seems like a release engineering bug: missing signature. It's interesting that this image shows up in docker search via CLI but I don't see it listed in our container catalog, https://access.redhat.com/containers/?count=50#/product/RedHatOpenshiftContainerPlatform. Workaround is to run 'atomic trust delete registry.access.redhat.com' I'm not able to reproduce the issue since I don't have the same version of atomic/skopeo in my RHEL 7.4 installation. Can we see atomic -v and skopeo -v? Rohan, could you verify we have a signature file for registry.access.redhat.com/openshift3/ose-sti-builder:latest? Aaron: Uncheck "Hide Deprecated" to see it in RHCC (as detailed in the See Also bz: https://bugzilla.redhat.com/show_bug.cgi?id=1479955) atomic -v 1.18.1 skopeo -v skopeo version 0.1.20 The workaround should be: atomic trust delete registry.access.redhat.com atomic pull registry.access.redhat.com/openshift3/ose-sti-builder atomic trust add \ --pubkeys /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release \ --sigstore https://access.redhat.com/webassets/docker/content/sigstore \ registry.access.redhat.com Deleting the trust is roughly analogous to disabling SELinux. So we have multiple issues: A) 3.6 images not appearing by default B) ose-sti-builder is marked deprecated (bz 1479955) C) ose-sti-builder does not have a valid signature (workaround is for this issue) According to above discussion, the issue has no related with upgrade, so just ensure that it can pull image trough atomic command now. atomic -v 1.17.1 skopeo -v skopeo version 0.1.20 # atomic trust show * (default) accept registry.access.redhat.com signed security,security After add registry.access.redhat.com, try to run "atomic pull" to pull ose-sti-builder image. # atomic pull registry.access.redhat.com/openshift3/ose-sti-builder:v3.6 Pulling registry.access.redhat.com/openshift3/ose-sti-builder:v3.6 ... Copying blob sha256:be1e04c6fbff244464b3deae14926bb737d67390a03a6c9004c2cb880b803a57 69.70 MB / ? [--------------------------------------------------------=------] Copying blob sha256:449ed11d916a0da0e334d9d7f3a31eb8c952ad1bcc7effeeb7fed478d347b4fd 0 B / ? [--------------------------------------------------------------------=] Copying blob sha256:ea7d7e15b2fc150b43295ca9bbc7f3fa148cad5e2d772f80c33bb156fc96c948 58.48 MB / ? [---------=-----------------------------------------------------] Copying blob sha256:843048ee27a7c9194e95bf4e23f4f001ee959cb4869a9033e0c34708fc96c471 113.22 MB / ? [------------------------------------------=-------------------] Copying blob sha256:a22dbd6b44ad9aa9d9aae4eca12b9964450e822c106f377ed983d45ee7e4d81f 0 B / ? [--------------------------------------------------------------------=] Copying config sha256:99ab8895d88a1f1154c0300e205e2cb31fabaaa16cc952d75990bba7a3640ab1 0 B / 4.60 KB [---------------------------------------------------------------] Writing manifest to image destination Storing signatures 4.60 KB / 4.60 KB [===========================================================] docker images|grep sti registry.access.redhat.com/openshift3/ose-sti-builder v3.6 99ab8895d88a 5 weeks ago 970.2 MB Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0113 |