Bug 2064408

Summary: operator-sdk 1.10-ocp: scorecard impacted by docker rate limit; switch to using ubi instead of busybox
Product: OpenShift Container Platform Reporter: Jesus M. Rodriguez <jesusr>
Component: Operator SDKAssignee: Jesus M. Rodriguez <jesusr>
Status: CLOSED ERRATA QA Contact: xzha
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.9CC: aos-bugs, diflores, jesusr, jfan, scuppett
Target Milestone: ---Keywords: Reopened
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Hardcoded busybox image from docker was used for scorecard. Consequence: Docker's new rate limit would cause periodic failures when running scorecard. Fix: Use ubi for scorecard: "registry.access.redhat.com/ubi8/ubi:8.4" Result: No more failures due to rate limiting.
Story Points: ---
Clone Of: 2058044
: 2064413 (view as bug list) Environment:
Last Closed: 2022-04-20 14:49:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2064413    
Bug Blocks: 2058044    

Description Jesus M. Rodriguez 2022-03-15 18:43:21 UTC
This is the 4.9 clone to fix the untar image in operator-sdk 1.10-ocp for OCP 4.9

+++ This bug was initially created as a clone of Bug #2058044 +++

Description of problem:
operator-sdk version 1.8-ocp does not contain option to --untar-image and --storage-image that is harcoded to docker.io/busybox
CU would like to override this setting and be able to use image from private repository instead of pulling from docker.io

Version-Release number of selected component (if applicable):
OCP 4.8
operator-sdk 1-8-ocp

How reproducible:
Pulling an image from docker.io fails with You have reached your pull rate limit

Steps to Reproduce:
1. operator-sdk scorecard <bundle-dir> -o text --selector=suite=olm -w 600 -x
2. scorecard pod events log:

Failed to pull image "docker.io/busybox:1.33.0": rpc error: code = Unknown desc = Error reading manifest 1.33.0 in docker.io/library/busybox: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Actual results:


Expected results:
Use scorecard without the need to pull images from docker.io to avoid pull rate limit issue

Comment 1 Jesus M. Rodriguez 2022-03-15 19:11:42 UTC
So the final solution for this bug was to use UBI as we did in a newer release of operator-sdk. Adding a new flag is effectively adding a new feature which for such an old release did not make sense. In OCP 4.10 with operator-sdk 1.16-ocp there is a flag that can solve this if you still have a problem.

Comment 2 Jesus M. Rodriguez 2022-04-07 02:46:25 UTC
this is currently blocked by a repo configuration that is causing e2e tests to fail. A PR to fix the test infrastructure has been posted. https://github.com/openshift/release/pull/27626

Comment 4 Jesus M. Rodriguez 2022-04-12 18:17:23 UTC
We replaced the busybox image from docker with "registry.access.redhat.com/ubi8/ubi:8.4".

Comment 5 xzha 2022-04-13 06:00:43 UTC
verify:

zhaoxia@xzha-mac memcached-operator % ./operator-sdk version
operator-sdk version: "v1.10.1-ocp", commit: "8efa92dcb512196f37bdccedeadadae22f7d8fdb", kubernetes version: "v1.21", go version: "go1.17.6", GOOS: "darwin", GOARCH: "amd64"

zhaoxia@xzha-mac memcached-operator % ./operator-sdk scorecard ./bundle -o text --selector=suite=olm -w 600s -x
--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:v1.10.1
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
	"suite":"olm"
	"test":"olm-crds-have-validation-test"
Results:
	Name: olm-crds-have-validation
	State: fail

	Suggestions:
		Add CRD validation for spec field `foo` in Memcached/v1alpha1
	Log:
....

zhaoxia@xzha-mac memcached-operator % oc get pod
NAME                  READY   STATUS      RESTARTS   AGE
scorecard-test-5t94   0/1     Completed   0          6m41s
scorecard-test-bqqs   0/1     Completed   0          47s
scorecard-test-bs6x   0/1     Completed   0          47s
scorecard-test-cwsf   0/1     Completed   0          47s
scorecard-test-lhjg   0/1     Completed   0          6m41s
scorecard-test-ssv7   0/1     Completed   0          47s
scorecard-test-tpkl   0/1     Completed   0          6m41s
scorecard-test-vkmz   0/1     Completed   0          6m41s
scorecard-test-vwxg   0/1     Completed   0          6m41s
scorecard-test-wtmq   0/1     Completed   0          47s
zhaoxia@xzha-mac memcached-operator % oc get pod scorecard-test-wtmq -o yaml
apiVersion: v1
kind: Pod
...
initContainers:
  - args:
    - tar
    - xvzf
    - /scorecard/bundle.tar.gz
    - -C
    - /scorecard-bundle
    image: registry.access.redhat.com/ubi8/ubi:8.4
    imagePullPolicy: IfNotPresent
    name: scorecard-untar
    resources: {}
    securityContext:
      capabilities:
        drop:
        - MKNOD
    terminationMessagePath: /dev/termination-log
...


zhaoxia@xzha-mac memcached-operator % oc get events | grep "Pulling image"  
9m28s       Normal    Pulling                  pod/scorecard-test-5t94   Pulling image "registry.access.redhat.com/ubi8/ubi:8.4"
9m14s       Normal    Pulling                  pod/scorecard-test-5t94   Pulling image "quay.io/operator-framework/scorecard-test:v1.10.1"
9m28s       Normal    Pulling                  pod/scorecard-test-lhjg   Pulling image "registry.access.redhat.com/ubi8/ubi:8.4"
9m18s       Normal    Pulling                  pod/scorecard-test-lhjg   Pulling image "quay.io/operator-framework/scorecard-test:v1.10.1"
9m28s       Normal    Pulling                  pod/scorecard-test-tpkl   Pulling image "registry.access.redhat.com/ubi8/ubi:8.4"
9m14s       Normal    Pulling                  pod/scorecard-test-tpkl   Pulling image "quay.io/operator-framework/scorecard-test:v1.10.1"
9m28s       Normal    Pulling                  pod/scorecard-test-vkmz   Pulling image "registry.access.redhat.com/ubi8/ubi:8.4"
9m7s        Normal    Pulling                  pod/scorecard-test-vkmz   Pulling image "quay.io/operator-framework/scorecard-test:v1.10.1"
9m28s       Normal    Pulling                  pod/scorecard-test-vwxg   Pulling image "registry.access.redhat.com/ubi8/ubi:8.4"
9m6s        Normal    Pulling                  pod/scorecard-test-vwxg   Pulling image "quay.io/operator-framework/scorecard-test:v1.10.1"


LGTM, verified

Comment 10 errata-xmlrpc 2022-04-20 14:49:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.29 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1363