Bug 1865791 - operator-sdk scorecard fails when executing on disconnected cluster
Summary: operator-sdk scorecard fails when executing on disconnected cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Operator SDK
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.6.0
Assignee: Jeff McCormick
QA Contact: kuiwang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-04 08:03 UTC by kuiwang
Modified: 2020-10-27 16:23 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:23:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:23:27 UTC

Description kuiwang 2020-08-04 08:03:08 UTC
Description of problem:
when executing it on disconnected cluster (we have to setup http_proxy to access cluster, for example ipi-on-vsphere/versioned-installer-6_7-http_proxy-vsphere_slave-ci), it report error "error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found" as the following, but actually it exists

--
[root@preserve-olm-env memcached-operator-scorecard]# operator-sdk scorecard  --kubeconfig /root/kubeconf/kubeconfig -o text --verbose deploy/olm-catalog/memcached-operator
DEBU[0000] Debug logging is set                         
Error: error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found
Usage:
  operator-sdk scorecard [flags]

Flags:
  -c, --config string            path to scorecard config file
  -h, --help                     help for scorecard
      --kubeconfig string        kubeconfig path
  -L, --list                     Option to enable listing which tests are run
  -n, --namespace string         namespace to run the test images in
  -o, --output string            Output format for results. Valid values: text, json (default "text")
  -l, --selector string          label selector to determine which tests are run
  -s, --service-account string   Service account to use for tests (default "default")
  -x, --skip-cleanup             Disable resource cleanup after tests are run
  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)

Global Flags:
      --verbose   Enable verbose logging

FATA[0001] error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found 
--

Version-Release number of selected component (if applicable):
[root@preserve-olm-env memcached-operator-scorecard]# operator-sdk version
operator-sdk version: "v1.0.0-alpha.2+git", commit: "unknown", kubernetes version: "unknown", go version: "go1.14 linux/amd64", GOOS: "linux", GOARCH: "amd64"

it is compiled based on the following
--
[root@preserve-olm-env operator-sdk]# git log -n 1
commit 8e5d03bd118715516f97287e3485a3262fe4adcb
Author: Eric Stroczynski <estroczy>
Date:   Mon Aug 3 17:00:39 2020 -0400

    internal/plugins/manifests: inject operator image using `IMG=<tag>` (#3634)
    
    This PR adds `cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)` to the manifests
    phase 2 plugin's Makefile modifier, such that a user can specify operator image using `make bundle
    IMG=<tag>`. Our e2e tests were passing because `make deploy IMG=<tag>` was run beforehand;
    in a newly scaffolded project that only uses bundles/package manifests,
    that will likely not be the case.
    
    internal/plugins/manifests: add `IMG` variable injection to `bundle` recipe
    
    test/: update tests to use `IMG=<test image tag>`
    
    website/: update OLM integration docs with this change

--

How reproducible:
always

Steps to Reproduce:
[root@preserve-olm-env memcached-operator-scorecard]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128
[root@preserve-olm-env memcached-operator-scorecard]# export https_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128
[root@preserve-olm-env memcached-operator-scorecard]# 
[root@preserve-olm-env memcached-operator-scorecard]# oc get cluster version
error: the server doesn't have a resource type "cluster"
[root@preserve-olm-env memcached-operator-scorecard]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-08-02-091622   True        False         23h     Cluster version is 4.6.0-0.nightly-2020-08-02-091622


[root@preserve-olm-env memcached-operator-scorecard]# operator-sdk scorecard  --kubeconfig /root/kubeconf/kubeconfig -o text --verbose deploy/olm-catalog/memcached-operator
DEBU[0000] Debug logging is set                         
Error: error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found
Usage:
  operator-sdk scorecard [flags]

Flags:
  -c, --config string            path to scorecard config file
  -h, --help                     help for scorecard
      --kubeconfig string        kubeconfig path
  -L, --list                     Option to enable listing which tests are run
  -n, --namespace string         namespace to run the test images in
  -o, --output string            Output format for results. Valid values: text, json (default "text")
  -l, --selector string          label selector to determine which tests are run
  -s, --service-account string   Service account to use for tests (default "default")
  -x, --skip-cleanup             Disable resource cleanup after tests are run
  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)

Global Flags:
      --verbose   Enable verbose logging

FATA[0001] error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found 

[root@preserve-olm-env memcached-operator-scorecard]# oc get cm
NAME                  DATA   AGE
scorecard-test-jks6   1      7s
scorecard-test-kz52   1      33m

[root@preserve-olm-env memcached-operator-scorecard]# oc get cm scorecard-test-jks6 -o yaml
apiVersion: v1
binaryData:
  bundle.tar.gz: H4sIAAAAAAAA/+xaW2/ruBHOs38F4ZcF2kqWHMcp9FSfJM0GmxuSNEVRFAFNjW2uKZIlKed4t/vfC1IXS7IcO+nGp4v1vNgezgznmyE5vDjBnE5AG330eRQEQXB6cuI+gyBofrrv4fHwdDDsn4TH4VEQ9k9Ph0fo5BN9KinVBqujIFBCmLfktrU3wf1GqMx/j2AyAx++4kQy8IlIXhJIHDPWL0TF/hIn7EN92HgMB4NN+Q/D/kk9//1+Pzw+QnsJ4u88/1jSZ1CaCh4hLCl8NcDtL+3P/6x9KnqLcAwGh5055XGEzlJtRPIAWqSKwDlMKKeGCt5JwOAYGxx1ECIKsGU+0QS0wYmMEE8Z6yDEcQIRWo0rf23QdbQEYo1MlUhlhNYFMivayiCUeXVTGHQ8RrX5oc6/ptq4NslShVnVBcfWlE9ThlWloYOQJkJChG5tdxKTjJeOVY4+d0EbbFIdoZ9/6SC0wIzGDnzWKCTw0f3V8/EjmUGCMyZCZmkNi/GPQEzO+urN0zEoDga0JxVoUAvwUj7n4pV7Ewos1hEyKgXbTZYy54GXR3URYiZnOMycstpxKW+9FApPIefkPncQwoSANBDfNmPa7dYC5n4SwWOXbl1m1NqF+LnwJ2N/60F9oJ1ptf6XI98TEhQ2QvmEpdqAsoOJEsgH3fvrwNvrfxgEg0b97w/CQf+w/u+Dqut/kXftE6FA2I+kVy4reQHIhsRjNiRy1drqjzkXBptieUIIs8TLF3Adof94+YL3z/wToZ/Lbwh1Vw51I9RdW/5Lf7p/qmpZ56x8ueLXmwv/ulGtM4S6du20inkXXjkLuhW5X2rGbIVaN6TpT9bQcVWt0/z2L/dJsMRjyqihNiBfsKYEXXFtcFuNXM3GReAHfljUP1uQIiQZJjATLAZVVk4saT5j47I+F/WJuPpdVLBqe1auXjnE2VevtbbuVMIzwUUxrCqFKQZNFJWuPqI5nmPLo1oyvLx1dnMeJVkB9dAYaxgO3NgqSlICMcVZBXUcmoUur8Z5ECzFIJlYJsCNLljexvCWuapasKRAMkqwjlBY4WpgQIxQUW0UJNiQ2TUeA9P1hs1prdo0ChuYLvNcFWQgkQwbaHRVmXJVYq2979T/OnY3WgU3mHJQa0Y9BHyx3lMR47+Pns6+f7kd3Vw83o/OLtbk3G4phb8qkawbseQ2PQ8waW/N2++xmUVlNPxybmx06/7u3Dm1P382unJ3f/Ewerp7eNOfrUmrWhzdPl59ub54uRw9fX/xcHV7udkqfLXjmpqttp4vHr7cPV49/WOzre6gu9XM+cWXv12+XN9dPr5h50mlsG6KJm7r+nBxfz06u3i5uhldrofLCd2njN0LRskyQiP2ipf6YxMBuYlf7PPr8zH3WLA0gRuRVlaXKvjEtmSDoWcS2cNc0zGDsrueSjmH9V4LB1ub83V9RIi1frsbkszTttmbSLM8p6oVX4sXElRCtV6VC2tEpaxq27Pl59Ke3WodesXi3QxtTUaKWNcYOd52Zm9COWb0J1D1duCxFJSbOlfacqQNcJPFgzBMk4biAppaRPAJnSZYNl0gCmqiC1DjBhp3DIYaKwYGDdYUTO03K46qpeO2ptQ4qYybhl9rQhuTgGUNyIY0VOpmnY8hEVxDg10UyCbfHjJhkjL9W4xUIjg1QlE+rWyGt4cuH5u59hbYTUCNMLw3i5UjfInirbVhe/bbZ1gLkkacf81FYEvc/sehvmnstk+Bj/qyaYv8hmPf/eG7/9c5844itNrRrgK62rHfiBjyKyydSimUqV1ZZVv8u1d+W9nTvSX6SPmUwUbpCWa6Kn6TMkN3tT1irBS1Q2IOy1eh4tz9/NjCKJ/X7uTKwxO6q4YlVSxCM2Okjnpt9y6xSDDlHXueoNXNty3ZmLLsmPSXqf1ejiZePT8l2KSKmmWE3NGrg5BUYkFjyM8rNeHcn/xXeWzLTpvf+n7iQJ9LxUHlM/v4wPvfyfDk8P63Dyry36tc2338pa+dtrz/BcP+cSP/w2A4PNz/7oMat7WrK+Di20ThBF6FmvtU+OOUx3YHM8OcA/NjmOCUGX8RrgrNuyzoD6iWLxZOd/V+sbuB4v7QGVAwpdqo5R8X4Tss5Lc7zoNiBu2sLjGZ4ynk2msbp73m33zyfz+OPrb+h4PTw/q/D3L572l7xiVYxZ/Sx275Pw37/eD0dDg4CsJBGB7q/16okf9eduH16+4AttT/Qfb+W83/8Lh/+P/PXsjlP+oUB9Zu9kBouaF7W8tu3v+d4qVPRe9HSAgRKulVxKLyXRK4UUt39Rohd1rNpDwr1qm/TOmUGohygeyYDdoUDGe3+VyIObp6fM4lnDgyM2xQLED7vt8God/d7tV2hP0VwvdB6H8YwhhrSjwyAzL33HPzuqPFdsErdxirWeywRQnWxt3dr0WgLli8s9Z6bAfrpCpYm1qFvRpq14zMDNzzotNcIRUs8bJtkbf679Tnw23tth2zYNXstiq2ws6bwSHPVFqwExVrb4YX+4ff1vOuEWjTbQ3C2cO5Rlaw8s+4TQEob0L3jb/s+P3wS9Ut6Fdd1MC7OVOoCLUn5M1ed4Xd1GvFfHd9gx7tPD+vdFAH7f58+A1gr/W7M/A1zc3QnWg7+HlqDHOa70bsVKMYFu0uu+amOz9YpkOh26K08uatCH3r/cmBDnSgAx3oc+i/AQAA//+KQNTgADYAAA==
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-04T07:47:05Z"
  labels:
    app: scorecard-test
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:binaryData:
        .: {}
        f:bundle.tar.gz: {}
      f:metadata:
        f:labels:
          .: {}
          f:app: {}
    manager: operator-sdk
    operation: Update
    time: "2020-08-04T07:47:05Z"
  name: scorecard-test-jks6
  namespace: default
  resourceVersion: "1352312"
  selfLink: /api/v1/namespaces/default/configmaps/scorecard-test-jks6
  uid: 8b747fa2-686b-4289-b670-90241e49126e



[root@preserve-olm-env memcached-operator-scorecard]# cat .osdk-scorecard.yaml 
scorecard:
  # Setting a global scorecard option
  output: json
  plugins:
    # `basic` tests configured to test 2 CRs
    - basic:
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
        proxy-port: 9001
    # `olm` tests configured to test 2 CRs
    - olm:
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
        csv-path: "deploy/olm-catalog/memcached-operator/manifests/memcached-operator.clusterserviceversion.yaml"
        proxy-port: 9000
[root@preserve-olm-env memcached-operator-scorecard]# cat deploy/olm-catalog/memcached-operator/tests/scorecard/config.yaml 
tests:
- name: "customtest1"
  image: quay.io/jemccorm/customtest1:v0.0.1
  entrypoint: 
  - custom-test
  labels:
    suite: custom
    test: customtest1
  description: an ISV custom test that does...
- name: "customtest2"
  entrypoint: 
  - custom-test
  image: quay.io/jemccorm/customtest2:v0.0.1
  labels:
    suite: custom
    test: customtest2
  description: an ISV custom test that does...
- name: "basic-check-spec"
  image: quay.io/operator-framework/scorecard-test:master
  entrypoint: 
  - scorecard-test
  - basic-check-spec
  labels:
    suite: basic
    test: basic-check-spec-test
  description: check the spec test
- name: "olm-bundle-validation"
  image: quay.io/operator-framework/scorecard-test:master
  entrypoint: 
  - scorecard-test
  - olm-bundle-validation
  labels:
    suite: olm
    test: olm-bundle-validation-test
  description: validate the bundle test
- name: "olm-crds-have-validation"
  image: quay.io/operator-framework/scorecard-test:master
  entrypoint: 
  - scorecard-test
  - olm-crds-have-validation
  labels:
    suite: olm
    test: olm-crds-have-validation-test
  description: CRDs have validation
- name: "olm-crds-have-resources"
  image: quay.io/operator-framework/scorecard-test:master
  entrypoint: 
  - scorecard-test
  - olm-crds-have-resources
  labels:
    suite: olm
    test: olm-crds-have-resources-test
  description: CRDs have resources
- name: "olm-spec-descriptors"
  image: quay.io/operator-framework/scorecard-test:master
  entrypoint: 
  - scorecard-test
  - olm-spec-descriptors
  labels:
    suite: olm
    test: olm-spec-descriptors-test
  description: OLM Spec Descriptors
- name: "olm-status-descriptors"
  image: quay.io/operator-framework/scorecard-test:master
  entrypoint: 
  - scorecard-test
  - olm-status-descriptors
  labels:
    suite: olm
    test: olm-status-descriptors-test
  description: OLM Status Descriptors
- name: "kuttl-tests"
  image: quay.io/operator-framework/scorecard-kuttl:dev
  labels:
    suite: kuttl
  description: Kuttl tests
  entrypoint: 
  - kuttl-test
  - olm-status-descriptors



Actual results:
the operator-sdk scorecard fails

Expected results:
the operator-sdk scorecard succeed


Additional info:

Comment 1 Jeff McCormick 2020-08-04 16:16:13 UTC
the scorecard configuration (config.yaml) format appears to not be correct, we recently
added a parallel/group into that config which when missing will cause scorecard to not
work.  You will want to update that config.yaml to the latest format and retry.

Also, with the new scorecard, the older scorecard config file (.osdk_scorecard.yaml) is not used any
longer and can be removed from this test.

Comment 3 Jeff McCormick 2020-08-05 13:55:25 UTC
Hi Kuiwang!

the scorecard has been updated by moving the 'alpha' version into the standard version, so when you run 'operator-sdk scorecard' now you will be running the 'new' version essentially.

You can use the config.yaml file that is included in the testdata, but you might have to modify it's image tags from 'dev' to 'latest'.  We do not push the 'dev' tags to quay.

Check out the new docs for scorecard here:
https://master.sdk.operatorframework.io/docs/advanced-topics/scorecard/scorecard/
https://master.sdk.operatorframework.io/docs/advanced-topics/scorecard/custom-tests/
https://master.sdk.operatorframework.io/docs/advanced-topics/scorecard/kuttl-tests/

regards,
Jeff Mc

Comment 4 kuiwang 2020-08-07 03:19:16 UTC
Verify it on 4.6. LGTM

--
[root@preserve-olm-env memcached-operator]# ck https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/105720/artifact/workdir/install-dir/auth/kubeconfig/*view*/
try to get kubeconfig https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/105720/artifact/workdir/install-dir/auth/kubeconfig/*view*/
the new kubeconfig file is kubeconfig-2020-08-06-23:10:07
Warning: wildcards not supported in HTTP.
--2020-08-06 23:10:07--  https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/105720/artifact/workdir/install-dir/auth/kubeconfig/*view*/
Resolving mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com (mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com)... 10.0.19.25, 10.0.19.22, 10.0.19.23
Connecting to mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com (mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com)|10.0.19.25|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12119 (12K) [text/plain]
Saving to: 'kubeconfig-2020-08-06-23:10:07'

100%[=============================================================================================================================================================>] 12,119      --.-K/s   in 0s      

2020-08-06 23:10:07 (125 MB/s) - 'kubeconfig-2020-08-06-23:10:07' saved [12119/12119]

succeed to get kubeconfig-2020-08-06-23:10:07
the new kubeconfig file kubeconfig-2020-08-06-23:10:07 take effective
[root@preserve-olm-env memcached-operator]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128
[root@preserve-olm-env memcached-operator]# export https_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128

[root@preserve-olm-env memcached-operator]# operator-sdk scorecard ./ --skip-cleanup -w 90s --selector=test=olm-crds-have-resources-test
--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test:master
Entrypoint: [scorecard-test olm-crds-have-resources]
Labels:
	"suite":"olm"
	"test":"olm-crds-have-resources-test"
Results:
	Name: olm-crds-have-resources
	State: pass

	Log:
		Loaded ClusterServiceVersion: memcached-operator.v0.0.1


[root@preserve-olm-env memcached-operator]# cat tests/scorecard/config.yaml 
kind: Configuration
apiversion: scorecard.operatorframework.io/v1alpha3
metadata:
  name: config
stages:
- parallel: true
  tests:
  - image: quay.io/operator-framework/scorecard-test:master
    entrypoint:
    - scorecard-test
    - basic-check-spec
    labels:
      suite: basic
      test: basic-check-spec-test
  - image: quay.io/operator-framework/scorecard-test:master
    entrypoint:
    - scorecard-test
    - olm-bundle-validation
    labels:
      suite: olm
      test: olm-bundle-validation-test
  - image: quay.io/operator-framework/scorecard-test:master
    entrypoint:
    - scorecard-test
    - olm-crds-have-validation
    labels:
      suite: olm
      test: olm-crds-have-validation-test
  - image: quay.io/operator-framework/scorecard-test:master
    entrypoint:
    - scorecard-test
    - olm-crds-have-resources
    labels:
      suite: olm
      test: olm-crds-have-resources-test
  - image: quay.io/operator-framework/scorecard-test:master
    entrypoint:
    - scorecard-test
    - olm-spec-descriptors
    labels:
      suite: olm
      test: olm-spec-descriptors-test
  - image: quay.io/operator-framework/scorecard-test:master
    entrypoint:
    - scorecard-test
    - olm-status-descriptors
    labels:
      suite: olm
      test: olm-status-descriptors-test

--

Comment 7 errata-xmlrpc 2020-10-27 16:23:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.