Bug 1788700 - need up-to-date messages from imagestreaimport InternalErrors surface in imagestream status
Summary: need up-to-date messages from imagestreaimport InternalErrors surface in imag...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.4.0
Assignee: Gabe Montero
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks: 1833019
TreeView+ depends on / blocked
 
Reported: 2020-01-07 20:49 UTC by Gabe Montero
Modified: 2020-05-07 16:35 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: stale error messages would exist on failed imagestreamimports Consequence: users would be mislead on what current imagestreamimport problems were because errors from prior import failures were still the only ones surfaced Fix: the logic for updating imagestreamimport error messages was enhanced to better determine that successive errors were in fact from different root causes, and hence update the error messages when appropriate Result: users get better guidance from imagestreamimport errors after successive failed attempts on what is needed to fix problems
Clone Of:
Environment:
Last Closed: 2020-05-04 11:23:07 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift openshift-apiserver pull 69 None closed Bug 1788700: include ISI error message changes when determining new generation need 2020-05-18 03:10:11 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:23:40 UTC

Description Gabe Montero 2020-01-07 20:49:29 UTC
From Oleg's and my recent debug foray with a customer disconnected install 

1) On an import error, the bump of the condition does not take into account the message as the comment notes at 

https://github.com/openshift/openshift-apiserver/blob/master/pkg/image/apiserver/internalimageutil/helpers.go#L470-L479

Namely

// HasTagCondition returns true if the specified image stream tag has a condition with the same type, status, and
// reason (does not check generation, date, or message).
func HasTagCondition(stream *imageapi.ImageStream, tag string, condition imageapi.TagEventCondition) bool {
	for _, existing := range stream.Status.Tags[tag].Conditions {
		if condition.Type == existing.Type && condition.Status == existing.Status && condition.Reason == existing.Reason {
			return true
		}
	}
	return false
}

So we could do things like change the image registry used, and if it fails in the same way, we won't bump the condition from the prior image registry used.

The above check should be updated to look at condition.Message as well.

Also, Oleg found that the error logging at https://github.com/openshift/openshift-controller-manager/blob/master/pkg/image/controller/imagestream_controller.go#L316

Namely

		klog.V(4).Infof("Import stream %s/%s partial=%t error: %v", stream.Namespace, stream.Name, partial, err)

masks error by default.

Perhaps that was done in a case where imports were failing for a while, and the log got chatty ... but we should revisit bumping down the log level here.

Comment 2 XiuJuan Wang 2020-02-03 06:57:06 UTC
Test with 4.4.0-0.nightly-2020-02-02-225006
The failed importing imagestreamtag is as following, could I mark the bug as verified?

$oc get is ruby  -n openshift -o yaml 
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/ruby
  tags:
  - conditions:
    - generation: 4
      lastTransitionTime: "2020-02-03T06:55:19Z"
      message: dockerimage.image.openshift.io "ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-23-rhel7:latest"
        not found
      reason: NotFound
      status: "False"
      type: ImportSuccess
    items: null
    tag: "2.3"
  - conditions:
    - generation: 5
      lastTransitionTime: "2020-02-03T06:55:19Z"
      message: dockerimage.image.openshift.io "ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-24-rhel7:latest"
        not found
      reason: NotFound
      status: "False"
      type: ImportSuccess
    items: null
    tag: "2.4"
  - items:
    - created: "2020-02-03T06:55:19Z"
      dockerImageReference: ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-25-rhel7@sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
      generation: 5
      image: sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
    tag: "2.5"
  - items:
    - created: "2020-02-03T06:55:19Z"
      dockerImageReference: ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-25-rhel7@sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
      generation: 5
      image: sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
    tag: latest

Comment 3 Gabe Montero 2020-02-03 15:29:59 UTC
I'm going to need some clarification on the steps you took XiuJuan to end up with the imagestream yaml in #Comment 2

Did you:

1) configure the samplesRegistry to an initial registry that did not have the ruby images

2) then change samplesRegistry to ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000 , where the ruby images are not there as well?

If so, then yes, this looks like a verified bug.

I will note that with generations of 5 for those tags, it looks like you went though such multiple configuration attempts.

But please clarify and we can go from there.

Comment 4 XiuJuan Wang 2020-02-04 01:42:14 UTC
@Gabe,
Yes, I did 1) and 2) as you said.
So I will mark this as verified.

Thanks

Comment 6 errata-xmlrpc 2020-05-04 11:23:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.