Bug 1486272 - 3.6 Images not found after upgrade (causing CrashLoopBackOff)
Summary: 3.6 Images not found after upgrade (causing CrashLoopBackOff)
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.6.z
Assignee: Justin Pierce
QA Contact: liujia
Depends On:
Blocks: 1724792
TreeView+ depends on / blocked
Reported: 2017-08-29 10:40 UTC by Thom Carlin
Modified: 2019-06-28 16:09 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2018-01-23 17:57:29 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1479955 0 unspecified CLOSED Container ose-sti-builder is marked as deprecated 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2018:0113 0 normal SHIPPED_LIVE OpenShift Container Platform 3.7 and 3.6 bug fix and enhancement update 2018-01-23 22:55:59 UTC

Internal Links: 1479955

Description Thom Carlin 2017-08-29 10:40:45 UTC
Description of problem:

Following successful in-place automatic upgrade from OCP 3.5 to 3.6 (following our documented procedures), started receiving CrashLoopBackupOffs on pods.  Investigation revealed 3.6-tagged images were not found

Version-Release number of the following components:
rpm -q openshift-ansible

rpm -q ansible

ansible --version
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

How reproducible:

Believe 100%

Steps to Reproduce:
1. Start with OCP 3.5 system
2. Follow in-place automatic upgrade documentation
3. Check tags for ose, logging, and metrics images

Actual results:

No 3.6 images found

Expected results:

3.6 images found

Additional info:

Workaround: Section SYNCING IMAGES has information on downloading the 3.6 images

Comment 1 Scott Dodson 2017-08-29 15:38:41 UTC
Can you please provide `oc describe pod` on a pot that's failing? We need to understand which exact image it's looking for and not finding.

Comment 4 Thom Carlin 2017-08-29 17:32:20 UTC
Insecure Registries:
Registries: registry.access.redhat.com (secure), registry.access.redhat.com (secure), docker.io (secure)

Also collaborated with Scott through IRC:
1) System is fully subscribed
2) rpm -q docker is the same on all systems (docker-1.12.6-48.git0fdc778.el7.x86_64)

Comment 6 Scott Dodson 2017-08-29 17:56:26 UTC
# atomic trust show
* (default)                   accept
registry.access.redhat.com    signed security@redhat.com,security@redhat.com

Aaaron does signing affect the ability to pull this image in such a manner that it'd get authentication denied?

Comment 8 Aaron Weitekamp 2017-08-30 18:07:37 UTC
The issue is you require a signature to be pulled to validate the image but that signature isn't there for the ose-sti-builder image. This seems like a release engineering bug: missing signature. It's interesting that this image shows up in docker search via CLI but I don't see it listed in our container catalog, https://access.redhat.com/containers/?count=50#/product/RedHatOpenshiftContainerPlatform.

Workaround is to run 'atomic trust delete registry.access.redhat.com'

I'm not able to reproduce the issue since I don't have the same version of atomic/skopeo in my RHEL 7.4 installation. Can we see atomic -v and skopeo -v?

Rohan, could you verify we have a signature file for registry.access.redhat.com/openshift3/ose-sti-builder:latest?

Comment 9 Thom Carlin 2017-08-30 18:22:06 UTC
Aaron: Uncheck "Hide Deprecated" to see it in RHCC (as detailed in the See Also bz: https://bugzilla.redhat.com/show_bug.cgi?id=1479955)

atomic -v

skopeo -v
skopeo version 0.1.20

The workaround should be:
atomic trust delete registry.access.redhat.com
atomic pull registry.access.redhat.com/openshift3/ose-sti-builder
atomic trust add \
--pubkeys /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release \
--sigstore https://access.redhat.com/webassets/docker/content/sigstore \

Deleting the trust is roughly analogous to disabling SELinux.

So we have multiple issues:
A) 3.6 images not appearing by default
B) ose-sti-builder is marked deprecated (bz 1479955)
C) ose-sti-builder does not have a valid signature (workaround is for this issue)

Comment 16 liujia 2017-10-09 07:10:18 UTC
According to above discussion, the issue has no related with upgrade, so just ensure that it can pull image trough atomic command now.

atomic -v

skopeo -v
skopeo version 0.1.20

# atomic trust show
* (default)                         accept                               
registry.access.redhat.com          signed security@redhat.com,security@redhat.com 

After add registry.access.redhat.com, try to run "atomic pull" to pull ose-sti-builder image.

# atomic pull registry.access.redhat.com/openshift3/ose-sti-builder:v3.6
Pulling registry.access.redhat.com/openshift3/ose-sti-builder:v3.6 ...
Copying blob sha256:be1e04c6fbff244464b3deae14926bb737d67390a03a6c9004c2cb880b803a57
 69.70 MB / ? [--------------------------------------------------------=------] 
Copying blob sha256:449ed11d916a0da0e334d9d7f3a31eb8c952ad1bcc7effeeb7fed478d347b4fd
 0 B / ? [--------------------------------------------------------------------=]
Copying blob sha256:ea7d7e15b2fc150b43295ca9bbc7f3fa148cad5e2d772f80c33bb156fc96c948
 58.48 MB / ? [---------=-----------------------------------------------------] 
Copying blob sha256:843048ee27a7c9194e95bf4e23f4f001ee959cb4869a9033e0c34708fc96c471
 113.22 MB / ? [------------------------------------------=-------------------] 
Copying blob sha256:a22dbd6b44ad9aa9d9aae4eca12b9964450e822c106f377ed983d45ee7e4d81f
 0 B / ? [--------------------------------------------------------------------=]
Copying config sha256:99ab8895d88a1f1154c0300e205e2cb31fabaaa16cc952d75990bba7a3640ab1
 0 B / 4.60 KB [---------------------------------------------------------------]
Writing manifest to image destination
Storing signatures
 4.60 KB / 4.60 KB [===========================================================]

docker images|grep sti
registry.access.redhat.com/openshift3/ose-sti-builder   v3.6                99ab8895d88a        5 weeks ago         970.2 MB

Comment 19 errata-xmlrpc 2018-01-23 17:57:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.