Hide Forgot
In general we are leaning more heavily into iteration, perform a number of cleanups on the test code (for porting to 4.6 as well) that unify some temporary hacks into a more principled structure. --fail-fast should terminate cleanly when a test fails (and highlight the test htat failed) [Early] and [Late] should not be run multiple times with --count [Early] and [Late] differ between jobs and should not overlap SCC pre-test checks were extremely slow (every 15s) and should take no more than 30s with check intervals much faster Cloud provider initialization should not occur unless needed Disruptive tests should skip [Late] but not [Early]
1, `--count` works as expected, if it is `-1`, it will run forever. [root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "OLM Report Upgradeable"|./openshift-tests run --count -1 -f - openshift-tests version: v4.1.0-3659-g7c51c89 started: (0/1/1) "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" passed: (3.8s) 2021-03-04T09:45:44 "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" started: (0/2/2) "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" passed: (2.8s) 2021-03-04T09:45:47 "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" started: (0/3/3) "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" ... But, if --count = -5 , it runs 1 time, it should run forever if --count < 0. [root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "OLM Report Upgradeable"|./openshift-tests run --count -5 -f - openshift-tests version: v4.1.0-3659-g7c51c89 started: (0/1/1) "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" passed: (2.8s) 2021-03-04T09:48:33 "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" Timeline: Mar 04 09:48:31.559 I ns/e2e-test-olm-23440-ksd65 namespace/e2e-test-olm-23440-ksd65 reason/CreatedSCCRanges created SCC ranges Mar 04 09:48:31.699 - 1s W ns/openshift-marketplace pod/qe-app-registry-r2ztw node/ip-10-0-179-41.us-east-2.compute.internal pod has been pending longer than a minute Mar 04 09:48:31.699 - 1s W ns/e2e-runtimeclass-7115 pod/test-runtimeclass-e2e-runtimeclass-7115-non-conflict-runti882v2 node/ip-10-0-140-208.us-east-2.compute.internal pod has been pending longer than a minute Mar 04 09:48:31.699 - 1s W ns/openshift-marketplace pod/qe-app-registry-2h64f node/ip-10-0-140-208.us-east-2.compute.internal pod has been pending longer than a minute 1 pass, 0 skip (2.8s) if --count = 0 , it runs 1 time, it should not run. [root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "OLM Report Upgradeable"|./openshift-tests run --count 0 -f - openshift-tests version: v4.1.0-3659-g7c51c89 started: (0/1/1) "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" passed: (2.9s) 2021-03-04T09:47:51 "[sig-operator] an end user can use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" Timeline: Mar 04 09:47:49.026 I ns/e2e-test-olm-23440-hwk9h namespace/e2e-test-olm-23440-hwk9h reason/CreatedSCCRanges created SCC ranges Mar 04 09:47:49.134 - 999ms W ns/openshift-marketplace pod/qe-app-registry-2h64f node/ip-10-0-140-208.us-east-2.compute.internal pod has been pending longer than a minute Mar 04 09:47:49.134 - 999ms W ns/e2e-runtimeclass-7115 pod/test-runtimeclass-e2e-runtimeclass-7115-non-conflict-runti882v2 node/ip-10-0-140-208.us-east-2.compute.internal pod has been pending longer than a minute Mar 04 09:47:49.134 - 999ms W ns/openshift-marketplace pod/qe-app-registry-r2ztw node/ip-10-0-179-41.us-east-2.compute.internal pod has been pending longer than a minute 1 pass, 0 skip (2.9s) `--fail-fast` works as expected, it exited the suite when any test fails [root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "\[sig-node\] RuntimeClass" "[sig-node] RuntimeClass should support RuntimeClasses API operations [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]" "[sig-node] RuntimeClass should reject a Pod requesting a RuntimeClass with an unconfigured handler [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" "[sig-node] RuntimeClass should reject a Pod requesting a RuntimeClass with conflicting node selector [Disabled:Broken] [Suite:k8s]" "[sig-node] RuntimeClass should reject a Pod requesting a deleted RuntimeClass [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" "[sig-node] RuntimeClass should reject a Pod requesting a non-existent RuntimeClass [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling with taints [Serial] [Disabled:Broken] [Suite:k8s]" "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taints [Disabled:Broken] [Suite:k8s]" [root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "\[sig-node\] RuntimeClass"|./openshift-tests run --fail-fast -f - openshift-tests version: v4.1.0-3659-g7c51c89 started: (0/1/8) "[sig-node] RuntimeClass should reject a Pod requesting a RuntimeClass with conflicting node selector [Disabled:Broken] [Suite:k8s]" started: (0/2/8) "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taints [Disabled:Broken] [Suite:k8s]" started: (0/3/8) "[sig-node] RuntimeClass should reject a Pod requesting a non-existent RuntimeClass [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" started: (0/4/8) "[sig-node] RuntimeClass should support RuntimeClasses API operations [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]" started: (0/5/8) "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" started: (0/6/8) "[sig-node] RuntimeClass should reject a Pod requesting a deleted RuntimeClass [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" started: (0/7/8) "[sig-node] RuntimeClass should reject a Pod requesting a RuntimeClass with an unconfigured handler [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" passed: (1.4s) 2021-03-04T09:20:59 "[sig-node] RuntimeClass should reject a Pod requesting a RuntimeClass with conflicting node selector [Disabled:Broken] [Suite:k8s]" ... failed: (5m4s) 2021-03-04T09:26:01 "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s]" skipped: (5m4s) 2021-03-04T09:26:01 "[sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taints [Disabled:Broken] [Suite:k8s]" Timeline: ... Failing tests: [sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler [NodeFeature:RuntimeHandler] [Disabled:Broken] [Suite:k8s] error: 1 fail, 5 pass, 1 skip (5m4s) But, it should have 2 skip test cases, the results summary is incorrect.
@clayton what is left here?