Bug 2008214
| Summary: | DU node gets rebooted while running cnf-tests with DISCOVERY_MODE=true | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Marius Cornea <mcornea> | ||||||
| Component: | CNF Platform Validation | Assignee: | Federico Paolinelli <fpaoline> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Nikita <nkononov> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 4.8 | CC: | achernet, aos-bugs, elevin, fpaoline, keyoung | ||||||
| Target Milestone: | --- | Keywords: | AutomationBlocker | ||||||
| Target Release: | 4.10.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | cnf-tests-container-v4.10.0-11 | Doc Type: | No Doc Update | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2022-08-24 12:52:49 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 2012962 | ||||||||
| Attachments: |
|
||||||||
Can you please provide test console logs with the -v and -progress ginkgo flags and also must-gather from the environment. Created attachment 1828353 [details]
cnf_e2e_console.log
Attaching the console output log.
Created attachment 1828354 [details]
must-gather
(In reply to Marius Cornea from comment #5) > Created attachment 1828354 [details] > must-gather Attaching must-gather. By watching the machine configs and performance profiles created during the test run I see that the reboot time matches the creation of 2 new performance profile(v1 and without-numa) and 2 new machine configs(50-performance-without-numa and 99-master-generated-kubelet-1): Sat Oct 2 09:43:15 EDT 2021 NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE machineconfig.machineconfiguration.openshift.io/00-master d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/00-worker d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-master-container-runtime d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-master-kubelet d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-worker-kubelet d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/02-master-workload-partitioning 3.2.0 19h machineconfig.machineconfiguration.openshift.io/50-masters-chrony-configuration 2.2.0 19h machineconfig.machineconfiguration.openshift.io/50-nto-master 3.2.0 18h machineconfig.machineconfiguration.openshift.io/50-performance-openshift-node-performance-profile 3.2.0 18h machineconfig.machineconfiguration.openshift.io/50-performance-without-numa 3.2.0 1s machineconfig.machineconfiguration.openshift.io/50-workers-chrony-configuration 2.2.0 19h machineconfig.machineconfiguration.openshift.io/99-master-dnsmasq-configuration 2.2.0 19h machineconfig.machineconfiguration.openshift.io/99-master-generated-kubelet d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/99-master-generated-kubelet-1 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 0s machineconfig.machineconfiguration.openshift.io/99-master-generated-registries d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/99-master-ssh 3.2.0 19h machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/99-worker-ssh 3.2.0 19h machineconfig.machineconfiguration.openshift.io/container-mount-namespace 3.1.0 19h machineconfig.machineconfiguration.openshift.io/disable-chronyd 2.2.0 18h machineconfig.machineconfiguration.openshift.io/load-sctp-module 2.2.0 19h machineconfig.machineconfiguration.openshift.io/rendered-master-47b0ee5721bc9b8dd13cfbca0ccb4eb3 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/rendered-master-5c2cf31acb762521ffb00447d541493c d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 17h machineconfig.machineconfiguration.openshift.io/rendered-master-5f5c3643f66fb1d2460624d419ec9415 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/rendered-master-6f2666bc5d8c95298fd43a8e67e2096f d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/rendered-worker-b838de50a038a10348fa1c4ad5a59ca8 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h NAME AGE performanceprofile.performance.openshift.io/openshift-node-performance-profile 18h performanceprofile.performance.openshift.io/v1 1s performanceprofile.performance.openshift.io/without-numa 1s Sat Oct 2 09:43:16 EDT 2021 NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE machineconfig.machineconfiguration.openshift.io/00-master d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/00-worker d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-master-container-runtime d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-master-kubelet d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/01-worker-kubelet d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/02-master-workload-partitioning 3.2.0 19h machineconfig.machineconfiguration.openshift.io/50-masters-chrony-configuration 2.2.0 19h machineconfig.machineconfiguration.openshift.io/50-nto-master 3.2.0 18h machineconfig.machineconfiguration.openshift.io/50-performance-openshift-node-performance-profile 3.2.0 18h machineconfig.machineconfiguration.openshift.io/50-performance-without-numa 3.2.0 2s machineconfig.machineconfiguration.openshift.io/50-workers-chrony-configuration 2.2.0 19h machineconfig.machineconfiguration.openshift.io/99-master-dnsmasq-configuration 2.2.0 19h machineconfig.machineconfiguration.openshift.io/99-master-generated-registries d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/99-master-ssh 3.2.0 19h machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/99-worker-ssh 3.2.0 19h machineconfig.machineconfiguration.openshift.io/container-mount-namespace 3.1.0 19h machineconfig.machineconfiguration.openshift.io/disable-chronyd 2.2.0 18h machineconfig.machineconfiguration.openshift.io/load-sctp-module 2.2.0 19h machineconfig.machineconfiguration.openshift.io/rendered-master-22adb9ab079b8efa2b5554b2b38aa9ba d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 0s machineconfig.machineconfiguration.openshift.io/rendered-master-47b0ee5721bc9b8dd13cfbca0ccb4eb3 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/rendered-master-5c2cf31acb762521ffb00447d541493c d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 17h machineconfig.machineconfiguration.openshift.io/rendered-master-5f5c3643f66fb1d2460624d419ec9415 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/rendered-master-6f2666bc5d8c95298fd43a8e67e2096f d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h machineconfig.machineconfiguration.openshift.io/rendered-worker-b838de50a038a10348fa1c4ad5a59ca8 d86cd9058d31182045d9f58061d7faffc610b209 3.2.0 18h NAME AGE performanceprofile.performance.openshift.io/openshift-node-performance-profile 18h performanceprofile.performance.openshift.io/without-numa 2s This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1975356 P{lease ignore the comment above. I pasted this in the incorrect bug.
Verified on cnf-tests:4.10.0 Bulk closing of all "CNF Platform Validation" component BZs assigned to CNF Network team members and in VERIFIED status for longer than 1 month. |
Description of problem: DU node gets rebooted while running cnf-tests with DISCOVERY_MODE=true Version-Release number of selected component (if applicable): quay.io/openshift-kni/cnf-tests:4.8 How reproducible: 100% Steps to Reproduce: 1. Run cnf-tests via podman run --name cnf-container-tests \ --net=host \ -v /home/kni/cnf_tests_mcornea:/kubeconfig:Z \ -v /home/kni/cnf_tests_mcornea/junit:/junit:Z \ -v /home/kni/cnf_tests_mcornea/report:/report:Z \ -e KUBECONFIG=/kubeconfig/kubeconfig \ -e IMAGE_REGISTRY=registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/ \ -e CNF_TESTS_IMAGE=cnf-tests:4.8 \ -e DPDK_TESTS_IMAGE=dpdk:4.8 \ -e ROLE_WORKER_CNF=master \ -e PERF_TEST_PROFILE=openshift-node-performance-profile \ -e DISCOVERY_MODE=true \ quay.io/openshift-kni/cnf-tests:4.8 \ /usr/bin/test-run.sh \ -ginkgo.focus="performance|sriov|ptp|sctp|dpdk" \ --junit /junit -ginkgo.v Actual results: During the test run the node gets rebooted thus causing some of the other tests failing due to timeout or unable to reach the API while the node comes back up. Expected results: The node doesn't get rebooted while running the tests. Additional info: The first test which failed is [sriov] SCTP integration Test Connectivity Connectivity between client and server • Failure in Spec Setup (BeforeEach) [82.277 seconds] [sriov] SCTP integration /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/sctp/sctp_sriov.go:33 Test Connectivity [BeforeEach] /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/sctp/sctp_sriov.go:112 Connectivity between client and server /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/sctp/sctp_sriov.go:113 Should work over a SR-IOV device /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/sctp/sctp_sriov.go:131 Unexpected error: <*errors.StatusError | 0xc000e91ea0>: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server was unable to return a response in the time allotted, but may still be processing the request (get pods testpod-hjp6c)", Reason: "Timeout", Details: { Name: "testpod-hjp6c", Group: "", Kind: "pods", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "{\"metadata\":{},\"status\":\"Failure\",\"message\":\"Timeout: request did not complete within the allotted timeout\",\"reason\":\"Timeout\",\"details\ ":{},\"code\":504}", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 504, }, } the server was unable to return a response in the time allotted, but may still be processing the request (get pods testpod-hjp6c) occurred