Bug 2236344
Summary: | Unable to perform EUS to EUS upgrade between 4.12 and 4.14 with workloads | ||||||
---|---|---|---|---|---|---|---|
Product: | Container Native Virtualization (CNV) | Reporter: | Debarati Basu-Nag <dbasunag> | ||||
Component: | Virtualization | Assignee: | lpivarc | ||||
Status: | CLOSED ERRATA | QA Contact: | Debarati Basu-Nag <dbasunag> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 4.14.0 | CC: | dshchedr, fdeutsch, lpivarc | ||||
Target Milestone: | --- | ||||||
Target Release: | 4.14.0 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | v4.14.0.rhel9-1911 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2023-11-08 14:06:16 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Debarati Basu-Nag
2023-08-31 01:48:07 UTC
During EUS to EUS upgrade from 4.12.z to 4.14.0, as seen below we see that: CNV 4.14 virt-launcher pods are trying to run on OCP 4.12. 1) OCP 4.12 [kbidarka@kbidarka-thinkpadt14sgen2i ocp-cnv-scripts]$ oc debug node/cnv-qe-infra-32.cnvqe2.lab.eng.rdu2.redhat.com Temporary namespace openshift-debug-ss9m5 is created for debugging node... Starting pod/cnv-qe-infra-32cnvqe2labengrdu2redhatcom-debug-9m7tx ... To use host binaries, run `chroot /host` Pod IP: 10.1.156.40 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# semodule -l | grep virt_launcher virt_launcher sh-4.4# cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.12 2) The failed virt-launcher pod version ./fetch_pod_version_info_icsp.sh virt-launcher-always-run-strategy-vm-1693420081-0430818-7lsfj test-upgrade-namespace "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/virt-launcher-rhel9/images/v4.14.0-379" Also looking at the above comment in description section, it appears we are hitting SELinux issue. --------------------------------------------------------------------------------------- To provide more explicit info from the cluster: [kbidarka@kbidarka-thinkpadt14sgen2i auth]$ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME cnv-qe-infra-29.cnvqe2.lab.eng.rdu2.redhat.com Ready control-plane,master 2d14h v1.27.4+d424288 x.x.x.x <none> Red Hat Enterprise Linux CoreOS 414.92.202308281054-0 (Plow) 5.14.0-284.28.1.el9_2.x86_64 cri-o://1.27.1-6.rhaos4.14.gitc2c9f36.el9 cnv-qe-infra-30.cnvqe2.lab.eng.rdu2.redhat.com Ready control-plane,master 2d14h v1.27.4+d424288 x.x.x.x <none> Red Hat Enterprise Linux CoreOS 414.92.202308281054-0 (Plow) 5.14.0-284.28.1.el9_2.x86_64 cri-o://1.27.1-6.rhaos4.14.gitc2c9f36.el9 cnv-qe-infra-31.cnvqe2.lab.eng.rdu2.redhat.com Ready control-plane,master 2d14h v1.27.4+d424288 x.x.x.x <none> Red Hat Enterprise Linux CoreOS 414.92.202308281054-0 (Plow) 5.14.0-284.28.1.el9_2.x86_64 cri-o://1.27.1-6.rhaos4.14.gitc2c9f36.el9 cnv-qe-infra-32.cnvqe2.lab.eng.rdu2.redhat.com Ready worker 2d13h v1.25.12+26bab08 x.x.x.x <none> Red Hat Enterprise Linux CoreOS 412.86.202308260032-0 (Ootpa) 4.18.0-372.70.1.el8_6.x86_64 cri-o://1.25.4-4.rhaos4.12.gitb9319a2.el8 cnv-qe-infra-33.cnvqe2.lab.eng.rdu2.redhat.com Ready,SchedulingDisabled worker 2d13h v1.25.12+26bab08 x.x.x.x <none> Red Hat Enterprise Linux CoreOS 412.86.202308260032-0 (Ootpa) 4.18.0-372.70.1.el8_6.x86_64 cri-o://1.25.4-4.rhaos4.12.gitb9319a2.el8 cnv-qe-infra-34.cnvqe2.lab.eng.rdu2.redhat.com Ready worker 2d13h v1.25.12+26bab08 x.x.x.x <none> Red Hat Enterprise Linux CoreOS 412.86.202308260032-0 (Ootpa) 4.18.0-372.70.1.el8_6.x86_64 cri-o://1.25.4-4.rhaos4.12.gitb9319a2.el8 [kbidarka@kbidarka-thinkpadt14sgen2i auth]$ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v4.14.0 OpenShift Virtualization 4.14.0 kubevirt-hyperconverged-operator.v4.13.4 Succeeded [kbidarka@kbidarka-thinkpadt14sgen2i auth]$ oc get ip -A NAMESPACE NAME CSV APPROVAL APPROVED nvidia-gpu-operator install-f57pp gpu-operator-certified.v23.6.0 Automatic true openshift-cnv install-4wn76 kubevirt-hyperconverged-operator.v4.13.2 Manual true openshift-cnv install-9bsl4 kubevirt-hyperconverged-operator.v4.13.4 Manual true openshift-cnv install-srm28 kubevirt-hyperconverged-operator.v4.13.3 Manual true openshift-cnv install-sz25x kubevirt-hyperconverged-operator.v4.13.1 Manual true openshift-cnv install-xrwpm kubevirt-hyperconverged-operator.v4.14.0 Manual true openshift-local-storage install-452wk local-storage-operator.v4.12.0-202307182142 Automatic true From 4.12.6 cluster, the VM's created by default are nonRoot VM's. ]$ oc get hco kubevirt-hyperconverged -n openshift-cnv -o yaml apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: certConfig: ca: duration: 48h0m0s renewBefore: 24h0m0s server: duration: 24h0m0s renewBefore: 12h0m0s featureGates: deployTektonTaskResources: false disableMDevConfiguration: false enableCommonBootImageImport: true nonRoot: true withHostPassthroughCPU: false I will need to collect logs for all our components. Cluster access would be preferred. Performed EUS->EUS upgrade from 4.12.6->4.14.0 successfully. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.14.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6817 |