Bug 2006980

@miabbott how frequent is frequently, is it all the time, 50%, 25%. Can you give me a sense. It will make a difference on how I approach this.

Pulling some stats from search.ci.openshift.org:

periodic-ci-openshift-release-master-nightly-4.10-e2e-azure (all) - 33 runs, 42% failed, 21% of failures match = 9% impact
periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-aws-ovn-upgrade (all) - 519 runs, 75% failed, 7% of failures match = 5% impact
periodic-ci-openshift-release-master-ci-4.10-e2e-aws-ovn-upgrade (all) - 470 runs, 75% failed, 7% of failures match = 5% impact
periodic-ci-openshift-release-master-nightly-4.10-e2e-vsphere (all) - 49 runs, 51% failed, 12% of failures match = 6% impact
periodic-ci-openshift-release-master-ci-4.10-e2e-azure-ovn-upgrade (all) - 329 runs, 85% failed, 4% of failures match = 3% impact
periodic-ci-openshift-release-master-nightly-4.10-e2e-aws-upgrade (all) - 361 runs, 49% failed, 7% of failures match = 3% impact
periodic-ci-openshift-multiarch-master-nightly-4.10-ocp-e2e-remote-libvirt-s390x (all) - 14 runs, 100% failed, 21% of failures match = 21% impact
periodic-ci-openshift-release-master-nightly-4.10-e2e-aws (all) - 38 runs, 16% failed, 83% of failures match = 13% impact
periodic-ci-openshift-release-master-ci-4.10-e2e-gcp-upgrade (all) - 519 runs, 46% failed, 8% of failures match = 3% impact

"Frequently" might have been generous from the boilerplate text.  :)

Since `e2e-aws` is probably the most used test in this list, I'd suggest using those numbers (38 runs, 16% failed, 83% of failures match = 13% impact) to decide how to approach this problem

Unable to recreate this on 4.9

dperaza@dperaza-mac origin % oc version
Client Version: 4.7.5
Server Version: 4.9.0-rc.7
Kubernetes Version: v1.22.0-rc.0+894a78b
dperaza@dperaza-mac origin % ./openshift-tests run openshift/templates --run "templateinstance impersonation tests"
openshift-tests version: v4.1.0-4436-gd5783a4
started: (0/1/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

passed: (13.3s) 2021-10-14T21:19:56 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

started: (0/2/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

passed: (17.3s) 2021-10-14T21:20:14 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

started: (0/3/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

passed: (13.8s) 2021-10-14T21:20:28 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

Trying to get me a 4.10 environment to test there, let you know what I find out.

Unable to recreate this on 4.9

dperaza@dperaza-mac origin % oc version
Client Version: 4.7.5
Server Version: 4.9.0-rc.7
Kubernetes Version: v1.22.0-rc.0+894a78b
dperaza@dperaza-mac origin % ./openshift-tests run openshift/templates --run "templateinstance impersonation tests"
openshift-tests version: v4.1.0-4436-gd5783a4
started: (0/1/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

passed: (13.3s) 2021-10-14T21:19:56 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

started: (0/2/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

passed: (17.3s) 2021-10-14T21:20:14 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

started: (0/3/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

passed: (13.8s) 2021-10-14T21:20:28 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

Trying to get me a 4.10 environment to test there, let you know what I find out.

On 4.10 google cluster is failing... Investigating, not sure is the same issue we see in our CI pipeline.

dperaza@dperaza-mac origin % oc version                                                                            Client Version: 4.7.5
Server Version: 4.10.0-0.nightly-2021-10-15-025303
Kubernetes Version: v1.22.1+9312243
dperaza@dperaza-mac origin % ./openshift-tests run openshift/templates --run "templateinstance impersonation tests"
openshift-tests version: v4.1.0-4436-gd5783a4
started: (0/1/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

Oct 15 10:25:45.865: INFO: Fetching cloud provider for "gce"
E1015 10:25:45.877361   32702 test_context.go:500] Failed to setup provider config for "gce": Error building GCE/GKE provider: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

failed: (100ms) 2021-10-15T14:25:45 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

started: (1/2/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

Oct 15 10:25:46.007: INFO: Fetching cloud provider for "gce"
E1015 10:25:46.010936   32703 test_context.go:500] Failed to setup provider config for "gce": Error building GCE/GKE provider: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

failed: (100ms) 2021-10-15T14:25:46 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

started: (2/3/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

Oct 15 10:25:46.141: INFO: Fetching cloud provider for "gce"
E1015 10:25:46.144614   32704 test_context.go:500] Failed to setup provider config for "gce": Error building GCE/GKE provider: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

failed: (100ms) 2021-10-15T14:25:46 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

Unable to recreate on AWS 4.10 cluster:

Logged into "https://api.ci-ln-g327fs2-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443" as "kube:admin" using the token provided.

You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
dperaza@dperaza-mac origin % oc version                                                                            Client Version: 4.7.5
Server Version: 4.10.0-0.nightly-2021-10-15-101546
Kubernetes Version: v1.22.1+9312243
dperaza@dperaza-mac origin % ./openshift-tests run openshift/templates --run "templateinstance impersonation tests"openshift-tests version: v4.1.0-4436-gd5783a4
started: (0/1/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

passed: (22s) 2021-10-15T18:54:11 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel]"

started: (0/2/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

passed: (15.8s) 2021-10-15T18:54:27 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation creation tests [Suite:openshift/conformance/parallel]"

started: (0/3/3) "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

passed: (16.4s) 2021-10-15T18:54:43 "[sig-devex][Feature:Templates] templateinstance impersonation tests should pass impersonation deletion tests [Suite:openshift/conformance/parallel]"

This test is also failing frequently for e2e-metal-ipi jobs;

https://search.ci.openshift.org/?search=templateinstance+impersonation+tests+should+pass+impersonation+update+tests&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

For example, for this job https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.10-e2e-metal-ipi-ovn-ipv6/1452523669183008768, it panicked with the same logs mentioned above.

After investigating the must-gather logs, there is no clue about what causing this.

Because it looked like there are two test one the passes and one that fails, my first though was that the previous test leaves an unexpected state making the second test fail. I tried running multiple test, sequentially and in parallel pointing to a and OCP 4.10 cluster and I can not replicate this test. Is there a way we can leave one of these clusters where the test is failing running for me to reproduced and trie to fix? I think the panic occurs as part of a mechanism from Ginkgo to stop that test bucket. The key of the unexpected error is the error printed here:

STEP: Destroying namespace "e2e-test-templates-bbz62" for this suite.
[AfterEach] [sig-devex][Feature:Templates] templateinstance impersonation tests
  github.com/openshift/origin/test/extended/templates/templateinstance_impersonation.go:221
fail [github.com/openshift/origin/test/extended/templates/templateinstance_impersonation.go:271]: Unexpected error:
    <*errors.StatusError | 0xc0002cdb80>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "templateinstances.template.openshift.io \"test\" is forbidden: User \"e2e-test-templates-bbz62-impersonateuser\" cannot get resource \"templateinstances\" in API group \"template.openshift.io\" in the namespace \"e2e-test-templates-bbz62\"",
            Reason: "Forbidden",
            Details: {
                Name: "test",
                Group: "template.openshift.io",
                Kind: "templateinstances",
                UID: "",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 403,
        },
    }
    templateinstances.template.openshift.io "test" is forbidden: User "e2e-test-templates-bbz62-impersonateuser" cannot get resource "templateinstances" in API group "template.openshift.io" in the namespace "e2e-test-templates-bbz62"
occurred

Which points to an access issue. My cluster RBAC roles and policies are must likely different than the cluster where the fails test and that would explain why it does not fail consistently, and why it started to fail without any changes to Templates code and test cases. At this point my hope is that I can get access to a cluster where the test fails consistently. Notice last time the template origin test was changed it was July 16 of 2020 and last time the template controller was changed was on October 1, 2020:
https://github.com/openshift/openshift-controller-manager/commits/master/pkg/template/controller
https://github.com/openshift/origin/commits/master/test/extended/templates

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-ovn-ipv6/1461272559717715968#1:build-log.txt%3A13995



Full Stack Trace
github.com/openshift/origin/test/extended/util.(*CLI).AdminConfig(0xc001b7d680, 0x83123dd)
	github.com/openshift/origin/test/extended/util/client.go:524 +0x6b
github.com/openshift/origin/test/extended/util.(*CLI).GetClientConfigForUser(0xc001b7d680, 0xc001a66510, 0x22, 0x0)
	github.com/spf13/cobra.3/command.go:897
main.main.func1(0xc001676c80, 0x0, 0x0)
	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:84 +0x94
main.main()
	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:85 +0x42c
E1118 12:18:44.610399   57149 runtime.go:78] Observed a panic: ginkgowrapper.FailurePanic{Message:"Nov 18 12:18:44.609: invalid configuration: [context was not found for specified context: policy-can-i/api-ostest-test-metalkube-org:6443/storage-adm2, cluster has no server defined]", Filename:"github.com/openshift/origin/test/extended/util/client.go", Line:524, FullStackTrace:"github.com/openshift/origin/test/extended/util.(*CLI).AdminConfig(0xc001b7d680, 0x83123dd)\n\tgithub.com/openshift/origin/test/extended/util/client.go:524 +0x6b\ngithub.com/openshift/origin/test/extended/util.(*CLI).GetClientConfigForUser(0xc001b7d680, 0xc001a66510, 0x22, 0x0)\n\tgithub.com/openshift/origin/test/extended/util/client.go:753 +0x82c\ngithub.com/openshift/origin/test/extended/util.(*CLI).ChangeUser(0xc001b7d680, 0xc001a66510, 0x22, 0xc002452500)\n\tgithub.com/openshift/origin/test/extended/util/client.go:168 +0x6a\ngithub.com/openshift/origin/test/extended/templates.glob..func5.1.1(0x0, 0x8f52888, 0x10)\n\tgithub.com/openshift/origin/test/extended/templates/templateinstance_impersonation.go:109 +0x115\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1(0x8fc45e0, 0xc00007a0c8, 0x7f0b20d4ca68, 0x18, 0xc00091b7a0)\n\tk8s.io/apimachinery.1/pkg/util/wait/wait.go:217 +0x26\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext(0x8fc45e0, 0xc00007a0c8, 0xc001d582b0, 0x3136d00, 0x0, 0x0)\n\tk8s.io/apimachinery.1/pkg/util/wait/wait.go:230 +0x7f\nk8s.io/apimachinery/pkg/util/wait.poll(0x8fc45e0, 0xc00007a0c8, 0xc00091b701, 0xc00091b7a0, 0xc001d582b0, 0x7988400, 0xc001d582b0)\n\tk8s.io/apimachinery.1/pkg/util/wait/wait.go:577 +0xe5\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext(0x8fc45e0, 0xc00007a0c8, 0x3b9aca00, 0x6fc23ac00, 0xc001d582b0, 0x7f8e0a0, 0x1)\n\tk8s.io/apimachinery.1/pkg/util/wait/wait.go:523 +0x66\nk8s.io/apimachinery/pkg/util/wait.PollImmediate(0x3b9aca00, 0x6fc23ac00, 0xc000ef8fc0, 0x0, 0x0)\n\tk8s.io/apimachinery.1/pkg/util/wait/wait.go:509 +0x6f\ngithub.com/openshift/origin/test/extended/templates.glob..func5.1()\n\tgithub.com/openshift/origin/test/extended/templates/templateinstance_impersonation.go:107 +0x9fb\ngithub.com/openshift/origin/pkg/test/ginkgo.(*TestOptions).Run(0xc001663ad0, 0xc000ab2f10, 0x1, 0x1, 0x831e5f9, 0x4a4de80)\n\tgithub.com/openshift/origin/pkg/test/ginkgo/cmd_runtest.go:61 +0x418\nmain.newRunTestCommand.func1.1()\n\tgithub.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:426 +0x4e\ngithub.com/openshift/origin/test/extended/util.WithCleanup(0xc001bbfc18)\n\tgithub.com/openshift/origin/test/extended/util/test.go:168 +0x5f\nmain.newRunTestCommand.func1(0xc001677680, 0xc000ab2f10, 0x1, 0x1, 0x0, 0x0)\n\tgithub.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:426 +0x333\ngithub.com/spf13/cobra.(*Command).execute(0xc001677680, 0xc000ab2d20, 0x1, 0x1, 0xc001677680, 0xc000ab2d20)\n\tgithub.com/spf13/cobra.3/command.go:852 +0x472\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xc001676c80, 0x0, 0x8f27f80, 0xbfcd620)\n\tgithub.com/spf13/cobra.3/command.go:960 +0x375\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\tgithub.com/spf13/cobra.3/command.go:897\nmain.main.func1(0xc001676c80, 0x0, 0x0)\n\tgithub.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:84 +0x94\nmain.main()\n\tgithub.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:85 +0x42c"} (

Hi Since we cannot Reproduce this issue and no update on the bug, we would be closing this bug