Description of problem: when executing it on disconnected cluster (we have to setup http_proxy to access cluster, for example ipi-on-vsphere/versioned-installer-6_7-http_proxy-vsphere_slave-ci), it report error "error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found" as the following, but actually it exists -- [root@preserve-olm-env memcached-operator-scorecard]# operator-sdk scorecard --kubeconfig /root/kubeconf/kubeconfig -o text --verbose deploy/olm-catalog/memcached-operator DEBU[0000] Debug logging is set Error: error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found Usage: operator-sdk scorecard [flags] Flags: -c, --config string path to scorecard config file -h, --help help for scorecard --kubeconfig string kubeconfig path -L, --list Option to enable listing which tests are run -n, --namespace string namespace to run the test images in -o, --output string Output format for results. Valid values: text, json (default "text") -l, --selector string label selector to determine which tests are run -s, --service-account string Service account to use for tests (default "default") -x, --skip-cleanup Disable resource cleanup after tests are run -w, --wait-time duration seconds to wait for tests to complete. Example: 35s (default 30s) Global Flags: --verbose Enable verbose logging FATA[0001] error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found -- Version-Release number of selected component (if applicable): [root@preserve-olm-env memcached-operator-scorecard]# operator-sdk version operator-sdk version: "v1.0.0-alpha.2+git", commit: "unknown", kubernetes version: "unknown", go version: "go1.14 linux/amd64", GOOS: "linux", GOARCH: "amd64" it is compiled based on the following -- [root@preserve-olm-env operator-sdk]# git log -n 1 commit 8e5d03bd118715516f97287e3485a3262fe4adcb Author: Eric Stroczynski <estroczy> Date: Mon Aug 3 17:00:39 2020 -0400 internal/plugins/manifests: inject operator image using `IMG=<tag>` (#3634) This PR adds `cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)` to the manifests phase 2 plugin's Makefile modifier, such that a user can specify operator image using `make bundle IMG=<tag>`. Our e2e tests were passing because `make deploy IMG=<tag>` was run beforehand; in a newly scaffolded project that only uses bundles/package manifests, that will likely not be the case. internal/plugins/manifests: add `IMG` variable injection to `bundle` recipe test/: update tests to use `IMG=<test image tag>` website/: update OLM integration docs with this change -- How reproducible: always Steps to Reproduce: [root@preserve-olm-env memcached-operator-scorecard]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128 [root@preserve-olm-env memcached-operator-scorecard]# export https_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128 [root@preserve-olm-env memcached-operator-scorecard]# [root@preserve-olm-env memcached-operator-scorecard]# oc get cluster version error: the server doesn't have a resource type "cluster" [root@preserve-olm-env memcached-operator-scorecard]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2020-08-02-091622 True False 23h Cluster version is 4.6.0-0.nightly-2020-08-02-091622 [root@preserve-olm-env memcached-operator-scorecard]# operator-sdk scorecard --kubeconfig /root/kubeconf/kubeconfig -o text --verbose deploy/olm-catalog/memcached-operator DEBU[0000] Debug logging is set Error: error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found Usage: operator-sdk scorecard [flags] Flags: -c, --config string path to scorecard config file -h, --help help for scorecard --kubeconfig string kubeconfig path -L, --list Option to enable listing which tests are run -n, --namespace string namespace to run the test images in -o, --output string Output format for results. Valid values: text, json (default "text") -l, --selector string label selector to determine which tests are run -s, --service-account string Service account to use for tests (default "default") -x, --skip-cleanup Disable resource cleanup after tests are run -w, --wait-time duration seconds to wait for tests to complete. Example: 35s (default 30s) Global Flags: --verbose Enable verbose logging FATA[0001] error running tests error deleting configMap scorecard-test-jks6 configmaps "scorecard-test-jks6" not found [root@preserve-olm-env memcached-operator-scorecard]# oc get cm NAME DATA AGE scorecard-test-jks6 1 7s scorecard-test-kz52 1 33m [root@preserve-olm-env memcached-operator-scorecard]# oc get cm scorecard-test-jks6 -o yaml apiVersion: v1 binaryData: bundle.tar.gz: H4sIAAAAAAAA/+xaW2/ruBHOs38F4ZcF2kqWHMcp9FSfJM0GmxuSNEVRFAFNjW2uKZIlKed4t/vfC1IXS7IcO+nGp4v1vNgezgznmyE5vDjBnE5AG330eRQEQXB6cuI+gyBofrrv4fHwdDDsn4TH4VEQ9k9Ph0fo5BN9KinVBqujIFBCmLfktrU3wf1GqMx/j2AyAx++4kQy8IlIXhJIHDPWL0TF/hIn7EN92HgMB4NN+Q/D/kk9//1+Pzw+QnsJ4u88/1jSZ1CaCh4hLCl8NcDtL+3P/6x9KnqLcAwGh5055XGEzlJtRPIAWqSKwDlMKKeGCt5JwOAYGxx1ECIKsGU+0QS0wYmMEE8Z6yDEcQIRWo0rf23QdbQEYo1MlUhlhNYFMivayiCUeXVTGHQ8RrX5oc6/ptq4NslShVnVBcfWlE9ThlWloYOQJkJChG5tdxKTjJeOVY4+d0EbbFIdoZ9/6SC0wIzGDnzWKCTw0f3V8/EjmUGCMyZCZmkNi/GPQEzO+urN0zEoDga0JxVoUAvwUj7n4pV7Ewos1hEyKgXbTZYy54GXR3URYiZnOMycstpxKW+9FApPIefkPncQwoSANBDfNmPa7dYC5n4SwWOXbl1m1NqF+LnwJ2N/60F9oJ1ptf6XI98TEhQ2QvmEpdqAsoOJEsgH3fvrwNvrfxgEg0b97w/CQf+w/u+Dqut/kXftE6FA2I+kVy4reQHIhsRjNiRy1drqjzkXBptieUIIs8TLF3Adof94+YL3z/wToZ/Lbwh1Vw51I9RdW/5Lf7p/qmpZ56x8ueLXmwv/ulGtM4S6du20inkXXjkLuhW5X2rGbIVaN6TpT9bQcVWt0/z2L/dJsMRjyqihNiBfsKYEXXFtcFuNXM3GReAHfljUP1uQIiQZJjATLAZVVk4saT5j47I+F/WJuPpdVLBqe1auXjnE2VevtbbuVMIzwUUxrCqFKQZNFJWuPqI5nmPLo1oyvLx1dnMeJVkB9dAYaxgO3NgqSlICMcVZBXUcmoUur8Z5ECzFIJlYJsCNLljexvCWuapasKRAMkqwjlBY4WpgQIxQUW0UJNiQ2TUeA9P1hs1prdo0ChuYLvNcFWQgkQwbaHRVmXJVYq2979T/OnY3WgU3mHJQa0Y9BHyx3lMR47+Pns6+f7kd3Vw83o/OLtbk3G4phb8qkawbseQ2PQ8waW/N2++xmUVlNPxybmx06/7u3Dm1P382unJ3f/Ewerp7eNOfrUmrWhzdPl59ub54uRw9fX/xcHV7udkqfLXjmpqttp4vHr7cPV49/WOzre6gu9XM+cWXv12+XN9dPr5h50mlsG6KJm7r+nBxfz06u3i5uhldrofLCd2njN0LRskyQiP2ipf6YxMBuYlf7PPr8zH3WLA0gRuRVlaXKvjEtmSDoWcS2cNc0zGDsrueSjmH9V4LB1ub83V9RIi1frsbkszTttmbSLM8p6oVX4sXElRCtV6VC2tEpaxq27Pl59Ke3WodesXi3QxtTUaKWNcYOd52Zm9COWb0J1D1duCxFJSbOlfacqQNcJPFgzBMk4biAppaRPAJnSZYNl0gCmqiC1DjBhp3DIYaKwYGDdYUTO03K46qpeO2ptQ4qYybhl9rQhuTgGUNyIY0VOpmnY8hEVxDg10UyCbfHjJhkjL9W4xUIjg1QlE+rWyGt4cuH5u59hbYTUCNMLw3i5UjfInirbVhe/bbZ1gLkkacf81FYEvc/sehvmnstk+Bj/qyaYv8hmPf/eG7/9c5844itNrRrgK62rHfiBjyKyydSimUqV1ZZVv8u1d+W9nTvSX6SPmUwUbpCWa6Kn6TMkN3tT1irBS1Q2IOy1eh4tz9/NjCKJ/X7uTKwxO6q4YlVSxCM2Okjnpt9y6xSDDlHXueoNXNty3ZmLLsmPSXqf1ejiZePT8l2KSKmmWE3NGrg5BUYkFjyM8rNeHcn/xXeWzLTpvf+n7iQJ9LxUHlM/v4wPvfyfDk8P63Dyry36tc2338pa+dtrz/BcP+cSP/w2A4PNz/7oMat7WrK+Di20ThBF6FmvtU+OOUx3YHM8OcA/NjmOCUGX8RrgrNuyzoD6iWLxZOd/V+sbuB4v7QGVAwpdqo5R8X4Tss5Lc7zoNiBu2sLjGZ4ynk2msbp73m33zyfz+OPrb+h4PTw/q/D3L572l7xiVYxZ/Sx275Pw37/eD0dDg4CsJBGB7q/16okf9eduH16+4AttT/Qfb+W83/8Lh/+P/PXsjlP+oUB9Zu9kBouaF7W8tu3v+d4qVPRe9HSAgRKulVxKLyXRK4UUt39Rohd1rNpDwr1qm/TOmUGohygeyYDdoUDGe3+VyIObp6fM4lnDgyM2xQLED7vt8God/d7tV2hP0VwvdB6H8YwhhrSjwyAzL33HPzuqPFdsErdxirWeywRQnWxt3dr0WgLli8s9Z6bAfrpCpYm1qFvRpq14zMDNzzotNcIRUs8bJtkbf679Tnw23tth2zYNXstiq2ws6bwSHPVFqwExVrb4YX+4ff1vOuEWjTbQ3C2cO5Rlaw8s+4TQEob0L3jb/s+P3wS9Ut6Fdd1MC7OVOoCLUn5M1ed4Xd1GvFfHd9gx7tPD+vdFAH7f58+A1gr/W7M/A1zc3QnWg7+HlqDHOa70bsVKMYFu0uu+amOz9YpkOh26K08uatCH3r/cmBDnSgAx3oc+i/AQAA//+KQNTgADYAAA== kind: ConfigMap metadata: creationTimestamp: "2020-08-04T07:47:05Z" labels: app: scorecard-test managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:binaryData: .: {} f:bundle.tar.gz: {} f:metadata: f:labels: .: {} f:app: {} manager: operator-sdk operation: Update time: "2020-08-04T07:47:05Z" name: scorecard-test-jks6 namespace: default resourceVersion: "1352312" selfLink: /api/v1/namespaces/default/configmaps/scorecard-test-jks6 uid: 8b747fa2-686b-4289-b670-90241e49126e [root@preserve-olm-env memcached-operator-scorecard]# cat .osdk-scorecard.yaml scorecard: # Setting a global scorecard option output: json plugins: # `basic` tests configured to test 2 CRs - basic: cr-manifest: - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml" proxy-port: 9001 # `olm` tests configured to test 2 CRs - olm: cr-manifest: - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml" csv-path: "deploy/olm-catalog/memcached-operator/manifests/memcached-operator.clusterserviceversion.yaml" proxy-port: 9000 [root@preserve-olm-env memcached-operator-scorecard]# cat deploy/olm-catalog/memcached-operator/tests/scorecard/config.yaml tests: - name: "customtest1" image: quay.io/jemccorm/customtest1:v0.0.1 entrypoint: - custom-test labels: suite: custom test: customtest1 description: an ISV custom test that does... - name: "customtest2" entrypoint: - custom-test image: quay.io/jemccorm/customtest2:v0.0.1 labels: suite: custom test: customtest2 description: an ISV custom test that does... - name: "basic-check-spec" image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - basic-check-spec labels: suite: basic test: basic-check-spec-test description: check the spec test - name: "olm-bundle-validation" image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-bundle-validation labels: suite: olm test: olm-bundle-validation-test description: validate the bundle test - name: "olm-crds-have-validation" image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-crds-have-validation labels: suite: olm test: olm-crds-have-validation-test description: CRDs have validation - name: "olm-crds-have-resources" image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-crds-have-resources labels: suite: olm test: olm-crds-have-resources-test description: CRDs have resources - name: "olm-spec-descriptors" image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-spec-descriptors labels: suite: olm test: olm-spec-descriptors-test description: OLM Spec Descriptors - name: "olm-status-descriptors" image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-status-descriptors labels: suite: olm test: olm-status-descriptors-test description: OLM Status Descriptors - name: "kuttl-tests" image: quay.io/operator-framework/scorecard-kuttl:dev labels: suite: kuttl description: Kuttl tests entrypoint: - kuttl-test - olm-status-descriptors Actual results: the operator-sdk scorecard fails Expected results: the operator-sdk scorecard succeed Additional info:
the scorecard configuration (config.yaml) format appears to not be correct, we recently added a parallel/group into that config which when missing will cause scorecard to not work. You will want to update that config.yaml to the latest format and retry. Also, with the new scorecard, the older scorecard config file (.osdk_scorecard.yaml) is not used any longer and can be removed from this test.
Hi Kuiwang! the scorecard has been updated by moving the 'alpha' version into the standard version, so when you run 'operator-sdk scorecard' now you will be running the 'new' version essentially. You can use the config.yaml file that is included in the testdata, but you might have to modify it's image tags from 'dev' to 'latest'. We do not push the 'dev' tags to quay. Check out the new docs for scorecard here: https://master.sdk.operatorframework.io/docs/advanced-topics/scorecard/scorecard/ https://master.sdk.operatorframework.io/docs/advanced-topics/scorecard/custom-tests/ https://master.sdk.operatorframework.io/docs/advanced-topics/scorecard/kuttl-tests/ regards, Jeff Mc
Verify it on 4.6. LGTM -- [root@preserve-olm-env memcached-operator]# ck https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/105720/artifact/workdir/install-dir/auth/kubeconfig/*view*/ try to get kubeconfig https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/105720/artifact/workdir/install-dir/auth/kubeconfig/*view*/ the new kubeconfig file is kubeconfig-2020-08-06-23:10:07 Warning: wildcards not supported in HTTP. --2020-08-06 23:10:07-- https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/105720/artifact/workdir/install-dir/auth/kubeconfig/*view*/ Resolving mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com (mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com)... 10.0.19.25, 10.0.19.22, 10.0.19.23 Connecting to mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com (mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com)|10.0.19.25|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 12119 (12K) [text/plain] Saving to: 'kubeconfig-2020-08-06-23:10:07' 100%[=============================================================================================================================================================>] 12,119 --.-K/s in 0s 2020-08-06 23:10:07 (125 MB/s) - 'kubeconfig-2020-08-06-23:10:07' saved [12119/12119] succeed to get kubeconfig-2020-08-06-23:10:07 the new kubeconfig file kubeconfig-2020-08-06-23:10:07 take effective [root@preserve-olm-env memcached-operator]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128 [root@preserve-olm-env memcached-operator]# export https_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@136.144.52.200:3128 [root@preserve-olm-env memcached-operator]# operator-sdk scorecard ./ --skip-cleanup -w 90s --selector=test=olm-crds-have-resources-test -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:master Entrypoint: [scorecard-test olm-crds-have-resources] Labels: "suite":"olm" "test":"olm-crds-have-resources-test" Results: Name: olm-crds-have-resources State: pass Log: Loaded ClusterServiceVersion: memcached-operator.v0.0.1 [root@preserve-olm-env memcached-operator]# cat tests/scorecard/config.yaml kind: Configuration apiversion: scorecard.operatorframework.io/v1alpha3 metadata: name: config stages: - parallel: true tests: - image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - basic-check-spec labels: suite: basic test: basic-check-spec-test - image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-bundle-validation labels: suite: olm test: olm-bundle-validation-test - image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-crds-have-validation labels: suite: olm test: olm-crds-have-validation-test - image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-crds-have-resources labels: suite: olm test: olm-crds-have-resources-test - image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-spec-descriptors labels: suite: olm test: olm-spec-descriptors-test - image: quay.io/operator-framework/scorecard-test:master entrypoint: - scorecard-test - olm-status-descriptors labels: suite: olm test: olm-status-descriptors-test --
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196