Bug 1741138 - Cluster loader not working as expected
Summary: Cluster loader not working as expected
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Loader
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.z
Assignee: Sebastian Jug
QA Contact: Simon
URL:
Whiteboard:
Depends On: 1765249 1824828
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-14 11:03 UTC by Shivkumar Ople
Modified: 2020-07-27 13:49 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-27 13:49:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24382 0 None closed Bug 1741138: Add mandatory ifExists field to CL config, and add ignore-params flag 2020-12-28 08:18:07 UTC
Red Hat Product Errata RHBA-2020:2990 0 None None None 2020-07-27 13:49:23 UTC

Description Shivkumar Ople 2019-08-14 11:03:56 UTC
Description of problem:

Cluster loader is not working as expected. Tested on 2 to 3 different OCP clusters. But it is failing on the same task.
Followed [1] in order to configure.

The script/loader is failing at the same point in many clusters when tested. Need to know why it's failing on the same task in the clusters. Seems like a problem from tool (cluster loader) side.

https://docs.openshift.com/container-platform/3.11/scaling_performance/using_cluster_loader.html



Same failure msgs on running the test, 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

• Failure in Spec Setup (BeforeEach) [7.898 seconds]
[Feature:Performance][Serial][Slow] Load cluster
/builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:30
  should load the cluster [Suite:openshift] [BeforeEach]
  /builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:61

  Aug 14 06:32:59.839: Error parsing config: Unsupported Config Type ""
  

  /builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:57
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSAug 14 06:33:07.178: INFO: Running AfterSuite actions on all node
Aug 14 06:33:07.178: INFO: Running AfterSuite actions on node 1


Summarizing 1 Failure:

[Fail] [Feature:Performance][Serial][Slow] Load cluster [BeforeEach] should load the cluster [Suite:openshift] 
/builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:57

Ran 1 of 1270 Specs in 8.030 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 1269 Skipped --- FAIL: TestExtended (8.51s)
FAIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Version-Release number of selected component (if applicable):

v3.11.129


How reproducible:

Always 


Steps to Reproduce:
1. Follow [1]
2. Configure cluster loader and run the extended.sh script to test
3. Run the test and observe the behaviour 

Actual results:

Script is failing at the same point in many clusters when tested.


Expected results:

Cluster loader should run properly on healthy cluster and check the load by creating resoure


Additional info:

Comment 10 Justin Pierce 2019-10-16 14:39:21 UTC
The error is resulting from an inability to parse a viper configuration file. Are you specifying one on the command line or trying to use the default / internal?
If you are specifying one, can you attach it to the BZ? I'll try to reproduce with the internal one in the meantime.

Comment 23 kirby.shabaga 2020-01-17 19:43:34 UTC
I am unable to run cluster loader too.  Following 3.11 documentation, I am trying:

./extended.test --ginkgo.focus="Load cluster"


The error returned is:

Jan 17 11:28:21.989: INFO: 
Latency metrics for node **********************
[AfterEach] [Feature:Performance][Serial][Slow] Load cluster
  /builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:142
STEP: Dumping a list of prepulled images on each node...
Jan 17 11:28:22.009: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready

• Failure in Spec Setup (BeforeEach) [2.131 seconds]
[Feature:Performance][Serial][Slow] Load cluster
/builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:30
  should load the cluster [Suite:openshift] [BeforeEach]
  /builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:61

  Expected error:
      <*errors.StatusError | 0xc420fbe5a0>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""},
              Status: "Failure",
              Message: "projectrequests.project.openshift.io is forbidden: User \"e2e-test-cl-54bzb-user\" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched",
              Reason: "Forbidden",
              Details: {
                  Name: "",
                  Group: "project.openshift.io",
                  Kind: "projectrequests",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 403,
          },
      }
      projectrequests.project.openshift.io is forbidden: User "e2e-test-cl-54bzb-user" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched
  not to have occurred

  /builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/util/cli.go:193
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSJan 17 11:28:22.069: INFO: Running AfterSuite actions on all node
Jan 17 11:28:22.069: INFO: Running AfterSuite actions on node 1


Summarizing 1 Failure:

[Fail] [Feature:Performance][Serial][Slow] Load cluster [BeforeEach] should load the cluster [Suite:openshift] 
/builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/util/cli.go:193

Ran 1 of 1270 Specs in 2.213 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 1269 Skipped --- FAIL: TestExtended (2.42s)

Comment 39 Simon 2020-07-22 18:33:45 UTC
PASS.
I was not able to reproduce the issue.

$ oc version
oc v3.11.235
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-31-31-73.us-west-2.compute.internal:8443
openshift v3.11.248
kubernetes v1.11.0+d4cacc0

Test
$ ./extended.test --ginkgo.focus="Load cluster"
without problems.
Logs:
http://file.rdu.redhat.com/skordas/logs/20200722/cluster_loader.log

Comment 41 errata-xmlrpc 2020-07-27 13:49:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2990


Note You need to log in before you can comment on or make changes to this bug.