Bug 1734627 - Build fails for PullBuilderImageFailed with EOL images when upgrade from 4.1 to 4.2
Summary: Build fails for PullBuilderImageFailed with EOL images when upgrade from 4.1 ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.2.0
Assignee: Gabe Montero
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-31 06:40 UTC by Wenjing Zheng
Modified: 2019-10-16 06:34 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:34:11 UTC
Target Upstream Version:


Attachments (Terms of Use)
Openshift ruby imagestream output (5.42 KB, text/plain)
2019-07-31 06:40 UTC, Wenjing Zheng
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-samples-operator pull 162 'None' 'closed' 'Bug 1734627: preserve deleted EOL ISTs on upgrade' 2019-12-03 20:12:35 UTC
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:34:23 UTC

Description Wenjing Zheng 2019-07-31 06:40:02 UTC
Created attachment 1594917 [details]
Openshift ruby imagestream output

Description of problem:
A new build is triggered and failed with error "PullBuilderImageFailed" if using EOL image in 4.1 and then upgrade to 4.2:
$ oc get builds
devexpupgrade   ruby-hello-world-3      Docker   Git@787f1be   Failed (PullBuilderImageFailed)   2 hours ago    47s
$ oc describe build/ruby-hello-world-2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Strategy:		Docker
URL:			https://github.com/openshift/ruby-hello-world
Commit:			787f1be (Merge pull request #78 from bparees/v22)
Author/Committer:	Ben Parees / GitHub
From Image:		DockerImage registry.redhat.io/rhscl/ruby-22-rhel7@sha256:2e83b9e07e85960060096b6aff7ee202a5f52e0e18447641b080b1f3879e0901
Output to:		ImageStreamTag ruby-hello-world:latest
Push Secret:		builder-dockercfg-pkct7

Build trigger cause:	Image change
Image ID:		registry.redhat.io/rhscl/ruby-22-rhel7@sha256:2e83b9e07e85960060096b6aff7ee202a5f52e0e18447641b080b1f3879e0901
Image Name/Kind:	ruby:2.2 / ImageStreamTag

Log Tail:	Pulling image registry.redhat.io/rhscl/ruby-22-rhel7@sha25...e85960060096b6aff7ee202a5f52e0e18447641b080b1f3879e0901 ...
		Warning: Pull failed, retrying in 5s ...
		Warning: Pull failed, retrying in 5s ...
		Warning: Pull failed, retrying in 5s ...
		error: build error: failed to pull image: After retrying 2...r: unable to retrieve auth token: invalid username/password
Events:		<none>


Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-07-30-073644

How reproducible:
Always

Steps to Reproduce:
1.Start a 4.1 cluster
2.Create a build with ruby:2.2 imagestream under Openshift
3.Upgrade to 4.2
4.A new build re-triggered, but failed with PullBuilderImageFailed 

Actual results:
Build failed

Expected results:
Should be no new build triggered, since no proper imagestream can be used.

Additional info:

Comment 1 Gabe Montero 2019-07-31 13:34:43 UTC
Adam - minus the EOL image noise, this is actually another instance of what we talked about yesterday wrt the installer pull credentials not getting pulled in by the build controller to handle TBR images.

But this most likely is a more reproducible case, vs. the e2e you mentioned only failed part of the time.

@Wenjing - you'll notice the image reference is properly resolved from ImageStream status:  Pulling image registry.redhat.io/rhscl/ruby-22-rhel7@sha25...e85960060096b6aff7ee202a5f52e0e18447641b080b1f3879e0901 ...

Closing this as a dup of the other bug.

*** This bug has been marked as a duplicate of bug 1721380 ***

Comment 2 Wenjing Zheng 2019-08-01 01:52:55 UTC
Yes, Gabe! seems pull credential is not used to builder image. Thanks for your analysis.

Comment 3 Gabe Montero 2019-08-01 20:15:54 UTC
Have some additional analysis here 

One of the items that is lost when the imagestream spec no longer has the EOL image, even if the EOL images is present in the imagestream status, is the registry pullthrough logic ... i.e. when the IST referencePolicy is "Local" on the IST in the image stream spec.

Rather that doing pull through, it defaults to type:  Source, and attempts to pull from registry.redhat.io directly vs. using the copy in the internal image registry.

The relevant code is at https://github.com/openshift/library-go/blob/master/pkg/image/imageutil/helpers.go#L310-L355

Ben / Adam - are we interested in modifying that code to allow for a tag to only be in status, and enable pull through in that case?

Comment 4 Ben Parees 2019-08-01 20:34:45 UTC
probably not a dupe per discussion.

I think this needs to work or we need to revisit removing those EOL images from our imagestreams.

Comment 5 Gabe Montero 2019-08-01 20:59:06 UTC
not advocating this as a long term solution, but a workaround on 4.2 is to re-import the tag so it shows up in the spec:

oc import-image openshift/ruby:2.2 --from="registry.redhat.io/rhscl/ruby-22-rhel7:latest" --confirm=true --insecure=true  --reference-policy='local' -n openshift

for example ... the local ref policy is key ... users can play around with insecure, specific docker tags, as desired.

Comment 6 Gabe Montero 2019-08-02 17:48:22 UTC
I'm going to look at augmenting samples opearator to not totally replace the imagestream.spec, and leave existing EOL specs if they have upgraded.

If that proves untenable, we will undo the removal of EOL images.

Comment 7 Gabe Montero 2019-08-02 18:59:50 UTC
OK I have a change that is working with local testing, both clean 4.2 install and 4.1 to 4.2 upgrade

PR up shortly

Comment 9 wewang 2019-08-06 09:52:29 UTC
Cluster openshift-samples updated 4.2.0-0.nightly-2019-08-06-062019 and no new build triggered because ruby:2.2 image has no changed in upgraded env, old build still complete, so verified it.

[wewang@Desktop work]$ oc get is ruby -o yaml -n openshift
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/display-name: Ruby
    samples.operator.openshift.io/version: 4.2.0-0.nightly-2019-08-06-062019
  creationTimestamp: "2019-08-06T06:56:25Z"
  generation: 3
  labels:
    samples.operator.openshift.io/managed: "true"
  name: ruby
  namespace: openshift
  resourceVersion: "65016"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/ruby
  uid: 4e5e5360-b817-11e9-ac6e-0a580a810009
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      description: Build and run Ruby 2.0 applications on RHEL 7. For more information
        about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.0/README.md.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby 2.0
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby:2.0,ruby
      tags: hidden,builder,ruby
      version: "2.0"
    from:
      kind: DockerImage
      name: registry.redhat.io/openshift3/ruby-20-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "2.0"
    referencePolicy:
      type: Local
  - annotations:
      description: Build and run Ruby 2.2 applications on RHEL 7. For more information
        about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.2/README.md.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby 2.2
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby:2.2,ruby
      tags: hidden,builder,ruby
      version: "2.2"
    from:
      kind: DockerImage
      name: registry.redhat.io/rhscl/ruby-22-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "2.2"
    referencePolicy:
      type: Local
  - annotations:
      description: Build and run Ruby 2.3 applications on RHEL 7. For more information
        about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.3/README.md.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby 2.3
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby:2.3,ruby
      tags: hidden,builder,ruby
      version: "2.3"
    from:
      kind: DockerImage
      name: registry.redhat.io/rhscl/ruby-23-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "2.3"
    referencePolicy:
      type: Local
  - annotations:
      description: Build and run Ruby 2.4 applications on RHEL 7. For more information
        about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.4/README.md.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby 2.4
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby:2.4,ruby
      tags: builder,ruby
      version: "2.4"
    from:
      kind: DockerImage
      name: registry.redhat.io/rhscl/ruby-24-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "2.4"
    referencePolicy:
      type: Local
  - annotations:
      description: Build and run Ruby 2.5 applications on RHEL 7. For more information
        about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.5/README.md.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby 2.5
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby:2.5,ruby
      tags: builder,ruby
      version: "2.5"
    from:
      kind: DockerImage
      name: registry.redhat.io/rhscl/ruby-25-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "2.5"
    referencePolicy:
      type: Local
  - annotations:
      description: |-
        Build and run Ruby applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.5/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of Ruby available on OpenShift, including major version updates.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby
      tags: builder,ruby
    from:
      kind: ImageStreamTag
      name: "2.5"
    generation: 1
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/ruby
  tags:
  - items:
    - created: "2019-08-06T06:57:00Z"
      dockerImageReference: registry.redhat.io/openshift3/ruby-20-rhel7@sha256:9cfdf4b811ace13d4c555335b249ab831832a384113035512abc9d4d5cc59716
      generation: 2
      image: sha256:9cfdf4b811ace13d4c555335b249ab831832a384113035512abc9d4d5cc59716
    tag: "2.0"
  - items:
    - created: "2019-08-06T06:57:00Z"
      dockerImageReference: registry.redhat.io/rhscl/ruby-22-rhel7@sha256:2e83b9e07e85960060096b6aff7ee202a5f52e0e18447641b080b1f3879e0901
      generation: 2
      image: sha256:2e83b9e07e85960060096b6aff7ee202a5f52e0e18447641b080b1f3879e0901
    tag: "2.2"
  - items:
    - created: "2019-08-06T06:57:00Z"
      dockerImageReference: registry.redhat.io/rhscl/ruby-23-rhel7@sha256:3f205876e1e6d05d693c8fc94e7abf7137f20767c66c636431774f1ea37094d6
      generation: 2
      image: sha256:3f205876e1e6d05d693c8fc94e7abf7137f20767c66c636431774f1ea37094d6
    tag: "2.3"
  - items:
    - created: "2019-08-06T06:57:00Z"
      dockerImageReference: registry.redhat.io/rhscl/ruby-24-rhel7@sha256:f3faf0a8255db207acaba98aa00293e4d2e890b5b63009fc95fafb46f035f94f
      generation: 2
      image: sha256:f3faf0a8255db207acaba98aa00293e4d2e890b5b63009fc95fafb46f035f94f
    tag: "2.4"
  - items:
    - created: "2019-08-06T06:57:00Z"
      dockerImageReference: registry.redhat.io/rhscl/ruby-25-rhel7@sha256:a05e92bbe06aafa013a2b6f5b85dec8b2f546195f17f365d99cc92aa917b7d64
      generation: 2
      image: sha256:a05e92bbe06aafa013a2b6f5b85dec8b2f546195f17f365d99cc92aa917b7d64
    tag: "2.5"
  - items:
    - created: "2019-08-06T06:57:00Z"
      dockerImageReference: registry.redhat.io/rhscl/ruby-25-rhel7@sha256:a05e92bbe06aafa013a2b6f5b85dec8b2f546195f17f365d99cc92aa917b7d64
      generation: 2
      image: sha256:a05e92bbe06aafa013a2b6f5b85dec8b2f546195f17f365d99cc92aa917b7d64
    tag: latest

Comment 10 errata-xmlrpc 2019-10-16 06:34:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.