Bug 1741138

Summary: Cluster loader not working as expected
Product: OpenShift Container Platform Reporter: Shivkumar Ople <sople>
Component: Cluster LoaderAssignee: Sebastian Jug <sejug>
Status: CLOSED ERRATA QA Contact: Simon <skordas>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: aabhishe, agawand, akhaire, aos-bugs, dahernan, jokerman, jupierce, kirby.shabaga, malonso, mifiedle, mtleilia, pweil, rdave, sejug, skuznets, sponnaga, ssadhale, vjaypurk, vlaad
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-27 13:49:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1765249, 1824828    
Bug Blocks:    

Description Shivkumar Ople 2019-08-14 11:03:56 UTC
Description of problem:

Cluster loader is not working as expected. Tested on 2 to 3 different OCP clusters. But it is failing on the same task.
Followed [1] in order to configure.

The script/loader is failing at the same point in many clusters when tested. Need to know why it's failing on the same task in the clusters. Seems like a problem from tool (cluster loader) side.

https://docs.openshift.com/container-platform/3.11/scaling_performance/using_cluster_loader.html



Same failure msgs on running the test, 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

• Failure in Spec Setup (BeforeEach) [7.898 seconds]
[Feature:Performance][Serial][Slow] Load cluster
/builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:30
  should load the cluster [Suite:openshift] [BeforeEach]
  /builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:61

  Aug 14 06:32:59.839: Error parsing config: Unsupported Config Type ""
  

  /builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:57
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSAug 14 06:33:07.178: INFO: Running AfterSuite actions on all node
Aug 14 06:33:07.178: INFO: Running AfterSuite actions on node 1


Summarizing 1 Failure:

[Fail] [Feature:Performance][Serial][Slow] Load cluster [BeforeEach] should load the cluster [Suite:openshift] 
/builddir/build/BUILD/atomic-openshift-git-0.14e54a3/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:57

Ran 1 of 1270 Specs in 8.030 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 1269 Skipped --- FAIL: TestExtended (8.51s)
FAIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Version-Release number of selected component (if applicable):

v3.11.129


How reproducible:

Always 


Steps to Reproduce:
1. Follow [1]
2. Configure cluster loader and run the extended.sh script to test
3. Run the test and observe the behaviour 

Actual results:

Script is failing at the same point in many clusters when tested.


Expected results:

Cluster loader should run properly on healthy cluster and check the load by creating resoure


Additional info:

Comment 10 Justin Pierce 2019-10-16 14:39:21 UTC
The error is resulting from an inability to parse a viper configuration file. Are you specifying one on the command line or trying to use the default / internal?
If you are specifying one, can you attach it to the BZ? I'll try to reproduce with the internal one in the meantime.

Comment 23 kirby.shabaga 2020-01-17 19:43:34 UTC
I am unable to run cluster loader too.  Following 3.11 documentation, I am trying:

./extended.test --ginkgo.focus="Load cluster"


The error returned is:

Jan 17 11:28:21.989: INFO: 
Latency metrics for node **********************
[AfterEach] [Feature:Performance][Serial][Slow] Load cluster
  /builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:142
STEP: Dumping a list of prepulled images on each node...
Jan 17 11:28:22.009: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready

• Failure in Spec Setup (BeforeEach) [2.131 seconds]
[Feature:Performance][Serial][Slow] Load cluster
/builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:30
  should load the cluster [Suite:openshift] [BeforeEach]
  /builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/cluster/cl.go:61

  Expected error:
      <*errors.StatusError | 0xc420fbe5a0>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""},
              Status: "Failure",
              Message: "projectrequests.project.openshift.io is forbidden: User \"e2e-test-cl-54bzb-user\" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched",
              Reason: "Forbidden",
              Details: {
                  Name: "",
                  Group: "project.openshift.io",
                  Kind: "projectrequests",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 403,
          },
      }
      projectrequests.project.openshift.io is forbidden: User "e2e-test-cl-54bzb-user" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched
  not to have occurred

  /builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/util/cli.go:193
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSJan 17 11:28:22.069: INFO: Running AfterSuite actions on all node
Jan 17 11:28:22.069: INFO: Running AfterSuite actions on node 1


Summarizing 1 Failure:

[Fail] [Feature:Performance][Serial][Slow] Load cluster [BeforeEach] should load the cluster [Suite:openshift] 
/builddir/build/BUILD/atomic-openshift-git-0.bd4f2d5/_output/local/go/src/github.com/openshift/origin/test/extended/util/cli.go:193

Ran 1 of 1270 Specs in 2.213 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 1269 Skipped --- FAIL: TestExtended (2.42s)

Comment 39 Simon 2020-07-22 18:33:45 UTC
PASS.
I was not able to reproduce the issue.

$ oc version
oc v3.11.235
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-31-31-73.us-west-2.compute.internal:8443
openshift v3.11.248
kubernetes v1.11.0+d4cacc0

Test
$ ./extended.test --ginkgo.focus="Load cluster"
without problems.
Logs:
http://file.rdu.redhat.com/skordas/logs/20200722/cluster_loader.log

Comment 41 errata-xmlrpc 2020-07-27 13:49:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2990