Bug 1847313
| Summary: | Tests are failing due to constant etcd leader elections changes | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Maysa Macedo <mdemaced> | ||||||
| Component: | Networking | Assignee: | Maysa Macedo <mdemaced> | ||||||
| Networking sub component: | kuryr | QA Contact: | GenadiC <gcheresh> | ||||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||||
| Severity: | high | ||||||||
| Priority: | urgent | CC: | ltomasbo, rlobillo | ||||||
| Version: | 4.5 | ||||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 4.6.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-10-27 16:07:13 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1849051 | ||||||||
| Attachments: |
|
||||||||
Created attachment 1702042 [details]
NP test results
Created attachment 1702043 [details]
ETCD metrics during test execution
Verified on OCP4.6.0-0.nightly-2020-07-21-004949 with OSP16.1 (RHOS-16.1-RHEL-8-20200714.n.0) with OVN. NP tests run with parallelism set to 2 with expected results. - No etcd leader change observed during test execution (on day 2020-07-22): [stack@undercloud-0 ~]$ date && for i in $(oc get pods -n openshift-etcd -l app=etcd -o NAME); do echo "# $i"; oc logs $i -n openshift-etcd -c etcd |grep 'became leader'; done Wed Jul 22 04:41:33 EDT 2020 # pod/etcd-ostest-tzdfc-master-0 # pod/etcd-ostest-tzdfc-master-1 raft2020/07/21 16:00:39 INFO: f56b8ef5cf671236 became leader at term 8 # pod/etcd-ostest-tzdfc-master-2 - 4 timeouts on port 2380 during test execution on master-1 but recovered succesfully (on day 2020-07-22):: [stack@undercloud-0 ~]$ date && for i in $(oc get pods -n openshift-etcd -l app=etcd -o NAME); do echo "# $i"; oc logs $i -n openshift-etcd -c etcd |grep 'timeout'; done Wed Jul 22 04:42:04 EDT 2020 # pod/etcd-ostest-tzdfc-master-0 # pod/etcd-ostest-tzdfc-master-1 2020-07-22 05:46:36.727875 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 2.559533ms, to fbb05cfa50510a87) 2020-07-22 05:46:36.727982 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 2.697912ms, to c0e6832f3d3c32b7) 2020-07-22 07:51:15.350022 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 3.185218ms, to fbb05cfa50510a87) 2020-07-22 07:51:15.350080 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 3.255695ms, to c0e6832f3d3c32b7) # pod/etcd-ostest-tzdfc-master-2 Furthermore, etcd metrics show an stable behaviour. Attached test logs and metrics: attachment 1702042 [details] & attachment 1702043 [details]. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Description of problem: Depending on the load running on the cluster, etcd leader change is happening more constantly causing Network Policy and Tempests tests to fail. Some tests failed in different stages, but with the following errors: should enforce multiple, stacked policies with overlapping podSelectors [Feature:NetworkPolicy-10] [BeforeEach] /home/stack/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:488 Jun 1 22:12:08.856: Pod did not finish as expected. Unexpected error: <*errors.StatusError | 0xc0013c4c80>: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "rpc error: code = Unavailable desc = etcdserver: leader changed", Reason: "", Details: nil, Code: 500, }, } rpc error: code = Unavailable desc = etcdserver: leader changed occurred should enforce policy to allow traffic only from a pod in a different namespace based on PodSelector and NamespaceSelector [Feature:NetworkPolicy-08] [BeforeEach] /home/stack/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:382 Jun 1 22:16:30.619: Pod did not finish as expected. Unexpected error: <*url.Error | 0xc002f52360>: { Op: "Get", URL: "https://api.ostest.shiftstack.com:6443/api/v1/namespaces/network-policy-7642/pods/client-can-connect-80-4gp4f", Err: {s: "EOF"}, } Get https://api.ostest.shiftstack.com:6443/api/v1/namespaces/network-policy-7642/pods/client-can-connect-80-4gp4f: EOF occurred Version-Release number of selected component (if applicable): How reproducible: Red Hat OpenStack Platform release 16.1.0 Beta (Train) OVN Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: