**What happened**: Submariner E2E for Service Discovery has 2 failed tests: [discovery] Test Service Discovery Across Clusters [discovery] Test Stateful Sets Discovery Across Clusters Due to: Failed to retrieve EndpointSlices. 1 EndpointSlices found when expected 2 **What you expected to happen**: All E2E tests should pass. **How to reproduce it (as minimally and precisely as possible)**: 1. Install ACM 2.4 with Submariner 0.11 2. Run Service Discovery E2E tests https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-2.4-Submariner-0.11-AWSx2-SDN/lastBuild/Test-Report/ **Anything else we need to know?**: • Failure [228.128 seconds] [discovery] Test Service Discovery Across Clusters /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:41 when a pod tries to resolve a service in a specific remote cluster by its cluster name /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:74 should resolve the service on the specified cluster [It] /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:75 Failed to retrieve EndpointSlices. 1 EndpointSlices found when expected 2 Unexpected error: <*errors.errorString | 0xc0002d81b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /home/jenkins/go/pkg/mod/github.com/submariner-io/shipyard.0-rc2/test/e2e/framework/framework.go:490 • Failure [226.242 seconds] [discovery] Test Stateful Sets Discovery Across Clusters /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:33 when a pod tries to resolve a podname from stateful set in a local cluster /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:42 should resolve the pod IP from the local cluster [It] /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:43 Failed to retrieve EndpointSlices. 1 EndpointSlices found when expected 2 Unexpected error: <*errors.errorString | 0xc0002d81b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /home/jenkins/go/pkg/mod/github.com/submariner-io/shipyard.0-rc2/test/e2e/framework/framework.go:490 Full Stack Trace github.com/submariner-io/shipyard/test/e2e/framework.AwaitUntil(0x162cfa6, 0x17, 0xc0005eacb8, 0xc0005ead08, 0x0, 0xc000060930) /home/jenkins/go/pkg/mod/github.com/submariner-io/shipyard.0-rc2/test/e2e/framework/framework.go:490 +0x172 github.com/submariner-io/lighthouse/test/e2e/framework.(*Framework).AwaitEndpointSlices(0xc00021a0c8, 0x1, 0xc000b82180, 0x8, 0xc000c08080, 0x19, 0x2, 0x2, 0xc00071ff10) /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/framework/framework.go:422 +0x3c8 github.com/submariner-io/lighthouse/test/e2e/discovery.RunSSDiscoveryLocalTest(0xc00021a0c8) /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:122 +0x70a github.com/submariner-io/lighthouse/test/e2e/discovery.glob..func3.2.1() /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:44 +0x2a github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc000215200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/leafnodes/runner.go:113 +0xa3 github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc000215200, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/leafnodes/runner.go:64 +0x15c github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00049d280, 0x17ca0c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/leafnodes/it_node.go:26 +0x87 github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc00002e4b0, 0x0, 0x17ca0c0, 0xc0001a4900) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/spec/spec.go:215 +0x72f github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc00002e4b0, 0x17ca0c0, 0xc0001a4900) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/spec/spec.go:138 +0xf2 github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc00050a000, 0xc00002e4b0, 0x0) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/specrunner/spec_runner.go:200 +0x111 github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc00050a000, 0x1) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/specrunner/spec_runner.go:170 +0x147 github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc00050a000, 0xc0004cb298) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/specrunner/spec_runner.go:66 +0x117 github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0001fa9a0, 0x7fce788e51b8, 0xc000702480, 0x1629612, 0x14, 0xc00070c1a0, 0x2, 0x2, 0x17fbea0, 0xc0001a4900, ...) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/internal/suite/suite.go:79 +0x546 github.com/onsi/ginkgo.runSpecsWithCustomReporters(0x17cb360, 0xc000702480, 0x1629612, 0x14, 0xc00070c180, 0x2, 0x2, 0x2) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/ginkgo_dsl.go:245 +0x218 github.com/onsi/ginkgo.RunSpecs(0x17cb360, 0xc000702480, 0x1629612, 0x14, 0xc000758770) /home/jenkins/go/pkg/mod/github.com/onsi/ginkgo.5/ginkgo_dsl.go:220 +0xa7 github.com/submariner-io/shipyard/test/e2e.RunE2ETests(0xc000702480, 0xc0000d68f0) /home/jenkins/go/pkg/mod/github.com/submariner-io/shipyard.0-rc2/test/e2e/e2e.go:95 +0x115 github.com/submariner-io/lighthouse/test/e2e.TestE2E(0xc000702480) /home/jenkins/go/src/github.com/submariner-io/lighthouse/test/e2e/e2e_test.go:28 +0x2b testing.tRunner(0xc000702480, 0x16c32b0) /mnt/skynet-data/skynet-env-2/miniconda/envs/conda-test/go/src/testing/testing.go:1193 +0xef created by testing.(*T).Run /mnt/skynet-data/skynet-env-2/miniconda/envs/conda-test/go/src/testing/testing.go:1238 +0x2b3 **Environment**: ### OCP Cluster api-nmanos-aws-devcluster-a-devcluster-openshift-com:6443 ### Client Version: 4.9.0 Server Version: 4.9.0 Kubernetes Version: v1.22.0-rc.0+894a78b ### OCP Cluster api-nmanos-aws-devcluster-c-devcluster-openshift-com:6443 ### Client Version: 4.9.0 Server Version: 4.9.0 Kubernetes Version: v1.22.0-rc.0+894a78b ### Images of Pods (in namespace submariner-operator) ### ### submariner-operator-bundle-index Image ### id=image-registry.openshift-image-registry.svc:5000/submariner-operator/submariner-operator-bundle-index@sha256:615da1ed8923495c05283d3da1caf20c525b4513a48ae8e59aeeabf4abec42b3 name=openshift/ose-operator-registry release=202110200713.p0.git.32eb259.assembly.art3171 url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.9.0-202110200713.p0.git.32eb259.assembly.art3171 version=v4.9.0 ### ocp-v4.0-art-dev Image ### id=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee2b825aa8ecc5711b914e6e98ceb324b1960f34ab56a97e0b817e91c291c6c ### ocp-v4.0-art-dev Image ### id=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb7321686220855dfc3f435e0f653dbce87699e0e019c531a1629ec38c227984 ### rhacm2-tech-preview-submariner-operator-bundle Image ### id=registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-tech-preview-submariner-operator-bundle@sha256:0eb8b71e6af202f94842de5df3a6d63a77e9dd680e534f0141c5b359fd68b158 name=rhacm2-tech-preview/submariner-operator-bundle release=43 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-operator-bundle/images/v0.11.0-43 version=v0.11.0 ### lighthouse-agent-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/lighthouse-agent-rhel8@sha256:e648ece136dc940de4656925f572c125039dfc7f315b3b91d300bf7f99caeb88 name=rhacm2-tech-preview/lighthouse-agent-rhel8 release=18 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.11.0-18 version=v0.11.0 ### lighthouse-coredns-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/lighthouse-coredns-rhel8@sha256:3f7dfafc721c7f3add511749b0dc4fdfa1a2578f87dab8e912772c7ec0e19759 name=rhacm2-tech-preview/lighthouse-coredns-rhel8 release=18 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.11.0-18 version=v0.11.0 ### submariner-gateway-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-gateway-rhel8@sha256:80a55047a6017c7b5d014fcc3c92bffe674697a1dd23cbec38ec2d946816e66d name=rhacm2-tech-preview/submariner-gateway-rhel8 release=17 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.11.0-17 version=v0.11.0 ### submariner-rhel8-operator Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-rhel8-operator@sha256:08d185eea8ee48a87660554ece1c9e03fe003c7d33e2ad238d61e06ab9f2fecf ### submariner-route-agent-rhel8 Image ### id=registry.redhat.io/rhacm2-tech-preview/submariner-route-agent-rhel8@sha256:bd180069f6562cb02bb0a7c22073e6377b6576cf6066f33d9ab2e6d5e5cb8982 name=rhacm2-tech-preview/submariner-route-agent-rhel8 release=17 url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.11.0-17 version=v0.11.0 ### submariner-addon-rhel8 Image ### id=registry.redhat.io/rhacm2/submariner-addon-rhel8@sha256:21633c4d0eeaa29f90c5052a7b868729c9cc00f6aa8bd4da4baee6d40db7d3d7
https://github.com/open-cluster-management/submariner-addon/pull/216
The fix will be shipped in v2.4.1
Verified on ACM 2.4.1 with: https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-2.4.1-Submariner-0.11.0-AWSx2-SDN/1/Test-Report/ Ran 12 of 15 Specs in 389.599 seconds SUCCESS! -- 12 Passed | 0 Failed | 0 Pending | 3 Skipped --- PASS: TestE2E (389.60s) PASS ok github.com/submariner-io/lighthouse/test/e2e 389.627s Submariner images: https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-operator-registry/images/v4.9.0-202111020858.p0.git.6a8cbc2.assembly.stream https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-operator-bundle/images/v0.11.0-52 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-agent-rhel8/images/v0.11.0-21 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/lighthouse-coredns-rhel8/images/v0.11.0-21 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-gateway-rhel8/images/v0.11.0-20 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-rhel8-operator/images/v0.11.0-50 https://access.redhat.com/containers/#/registry.access.redhat.com/rhacm2-tech-preview/submariner-route-agent-rhel8/images/v0.11.0-20 Note: When upgrading from ACM 2.4.0 to 2.4.1, the same Submariner E2E tests will fail. It is fixed when installing ACM 2.4.1 from scratch.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Advanced Cluster Management 2.4.1 bug fixes and container updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4970