Bug 1788700

Summary: need up-to-date messages from imagestreaimport InternalErrors surface in imagestream status
Product: OpenShift Container Platform Reporter: Gabe Montero <gmontero>
Component: ImageStreamsAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.4CC: aos-bugs, jokerman, wzheng, xiuwang
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: stale error messages would exist on failed imagestreamimports Consequence: users would be mislead on what current imagestreamimport problems were because errors from prior import failures were still the only ones surfaced Fix: the logic for updating imagestreamimport error messages was enhanced to better determine that successive errors were in fact from different root causes, and hence update the error messages when appropriate Result: users get better guidance from imagestreamimport errors after successive failed attempts on what is needed to fix problems
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:23:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1833019    

Description Gabe Montero 2020-01-07 20:49:29 UTC 
From Oleg's and my recent debug foray with a customer disconnected install 

1) On an import error, the bump of the condition does not take into account the message as the comment notes at 

https://github.com/openshift/openshift-apiserver/blob/master/pkg/image/apiserver/internalimageutil/helpers.go#L470-L479

Namely

// HasTagCondition returns true if the specified image stream tag has a condition with the same type, status, and
// reason (does not check generation, date, or message).
func HasTagCondition(stream *imageapi.ImageStream, tag string, condition imageapi.TagEventCondition) bool {
	for _, existing := range stream.Status.Tags[tag].Conditions {
		if condition.Type == existing.Type && condition.Status == existing.Status && condition.Reason == existing.Reason {
			return true
		}
	}
	return false
}

So we could do things like change the image registry used, and if it fails in the same way, we won't bump the condition from the prior image registry used.

The above check should be updated to look at condition.Message as well.

Also, Oleg found that the error logging at https://github.com/openshift/openshift-controller-manager/blob/master/pkg/image/controller/imagestream_controller.go#L316

Namely

		klog.V(4).Infof("Import stream %s/%s partial=%t error: %v", stream.Namespace, stream.Name, partial, err)

masks error by default.

Perhaps that was done in a case where imports were failing for a while, and the log got chatty ... but we should revisit bumping down the log level here.
Comment 2 XiuJuan Wang 2020-02-03 06:57:06 UTC 
Test with 4.4.0-0.nightly-2020-02-02-225006
The failed importing imagestreamtag is as following, could I mark the bug as verified?

$oc get is ruby  -n openshift -o yaml 
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/ruby
  tags:
  - conditions:
    - generation: 4
      lastTransitionTime: "2020-02-03T06:55:19Z"
      message: dockerimage.image.openshift.io "ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-23-rhel7:latest"
        not found
      reason: NotFound
      status: "False"
      type: ImportSuccess
    items: null
    tag: "2.3"
  - conditions:
    - generation: 5
      lastTransitionTime: "2020-02-03T06:55:19Z"
      message: dockerimage.image.openshift.io "ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-24-rhel7:latest"
        not found
      reason: NotFound
      status: "False"
      type: ImportSuccess
    items: null
    tag: "2.4"
  - items:
    - created: "2020-02-03T06:55:19Z"
      dockerImageReference: ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-25-rhel7@sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
      generation: 5
      image: sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
    tag: "2.5"
  - items:
    - created: "2020-02-03T06:55:19Z"
      dockerImageReference: ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000/rhscl/ruby-25-rhel7@sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
      generation: 5
      image: sha256:339ae7056c25d075e746040903e2fc1d9f349066b064c9f166cea0f6c47da54c
    tag: latest
Comment 3 Gabe Montero 2020-02-03 15:29:59 UTC 
I'm going to need some clarification on the steps you took XiuJuan to end up with the imagestream yaml in #Comment 2

Did you:

1) configure the samplesRegistry to an initial registry that did not have the ruby images

2) then change samplesRegistry to ec2-3-12-153-248.us-east-2.compute.amazonaws.com:5000 , where the ruby images are not there as well?

If so, then yes, this looks like a verified bug.

I will note that with generations of 5 for those tags, it looks like you went though such multiple configuration attempts.

But please clarify and we can go from there.
Comment 4 XiuJuan Wang 2020-02-04 01:42:14 UTC 
@Gabe,
Yes, I did 1) and 2) as you said.
So I will mark this as verified.

Thanks
Comment 6 errata-xmlrpc 2020-05-04 11:23:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581