Bug 2022895
| Summary: | Post upgrade (4.9.0 -> 4.9.1) hostpath provisioner pods are not restarted | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Debarati Basu-Nag <dbasunag> |
| Component: | Storage | Assignee: | Alex Kalenyuk <akalenyu> |
| Status: | CLOSED ERRATA | QA Contact: | Jenia Peimer <jpeimer> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.9.1 | CC: | cnv-qe-bugs, ibesso, ngavrilo, stirabos, yadu, ycui |
| Target Milestone: | --- | ||
| Target Release: | 4.9.2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | hostpath-provisioner-rhel8-operator v4.9.2-2, CNV v4.9.2-4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-01-19 17:49:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Debarati Basu-Nag
2021-11-12 21:53:56 UTC
Associated logs from the pods: ================================== cnv-qe-jenkins@c01-dbnp-490-hl62r-executor ~]$ kubectl logs hostpath-provisioner-gb5b4 -n openshift-cnv I1115 20:06:22.461624 1 hostpath-provisioner.go:82] initiating kubevirt/hostpath-provisioner on node: c01-dbnp-490-hl62r-worker-0-9sp8h I1115 20:06:22.461690 1 hostpath-provisioner.go:277] creating provisioner controller with name: kubevirt.io/hostpath-provisioner I1115 20:06:22.462010 1 controller.go:772] Starting provisioner controller kubevirt.io/hostpath-provisioner_hostpath-provisioner-gb5b4_8a7d3118-350f-418f-8b81-b54e9b7b0c3f! I1115 20:06:22.562204 1 controller.go:821] Started provisioner controller kubevirt.io/hostpath-provisioner_hostpath-provisioner-gb5b4_8a7d3118-350f-418f-8b81-b54e9b7b0c3f! I1115 20:09:50.406102 1 hostpath-provisioner.go:95] isCorrectNodeByBindingMode mode: WaitForFirstConsumer I1115 20:09:50.406131 1 hostpath-provisioner.go:118] missing volume.kubernetes.io/selected-node annotation, skipping operations for pvc I1115 20:09:50.411439 1 hostpath-provisioner.go:95] isCorrectNodeByBindingMode mode: WaitForFirstConsumer I1115 20:09:50.411455 1 hostpath-provisioner.go:118] missing volume.kubernetes.io/selected-node annotation, skipping operations for pvc I1115 20:09:50.420885 1 hostpath-provisioner.go:95] isCorrectNodeByBindingMode mode: WaitForFirstConsumer I1115 20:09:50.420902 1 hostpath-provisioner.go:118] missing volume.kubernetes.io/selected-node annotation, skipping operations for pvc [cnv-qe-jenkins@c01-dbnp-490-hl62r-executor ~]$ =================================== Log from HPP operator pod: http://pastebin.test.redhat.com/1008652 On HPP operator logs we see:
{"level":"error","ts":1637081879.5886474,"logger":"controller_hostpathprovisioner","msg":"Unable to create DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"DaemonSet.apps "hostpath-provisioner" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app.kubernetes.io/component":"storage", "app.kubernetes.io/managed-by":"hostpath-provisioner-operator", "app.kubernetes.io/part-of":"hyperconverged-cluster", "app.kubernetes.io/version":"v4.9.1", "k8s-app":"hostpath-provisioner"}: `selector` does not match template `labels`","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error
/remote-source/app/vendor/github.com/go-logr/zapr/zapr.go:128
kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).reconcileUpdate
/remote-source/app/pkg/controller/hostpathprovisioner/controller.go:342
kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile
while on the previous object from CNV 4.9.0:
$ oc get daemonsets -n openshift-cnv hostpath-provisioner -o json | jq ".metadata.labels"
{
"app.kubernetes.io/component": "storage",
"app.kubernetes.io/managed-by": "hostpath-provisioner-operator",
"app.kubernetes.io/part-of": "hyperconverged-cluster",
"app.kubernetes.io/version": "v4.9.0",
"k8s-app": "hostpath-provisioner"
}
So the issue is in the operator trying to update hostpath-provisioner daemonset passing a template with a label ("app.kubernetes.io/version") that doesn't match the existing object ("v4.9.0" vs "v4.9.1").
Verified for CNV v4.9.2-22, HCO v4.9.2-6, hostpath-provisioner-operator v4.9.2-2 Upgraded CNV v4.8.4 -> 4.9.0 -> 4.9.1 -> 4.9.2-22 $ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v4.9.2 OpenShift Virtualization 4.9.2 kubevirt-hyperconverged-operator.v4.9.1 Succeeded $ oc get ip -n openshift-cnv NAME CSV APPROVAL APPROVED install-6f2x9 kubevirt-hyperconverged-operator.v4.9.1 Manual true install-pcjnw kubevirt-hyperconverged-operator.v4.9.0 Manual true install-ws4fk kubevirt-hyperconverged-operator.v4.8.4 Manual true install-x4h9g kubevirt-hyperconverged-operator.v4.9.2 Manual true hostpath-provisioner pods are restarted: $ oc get pods -n openshift-cnv | grep hostpath hostpath-provisioner-8qm4q 1/1 Running 0 17h hostpath-provisioner-nskhg 1/1 Running 0 17h hostpath-provisioner-operator-85984bf8f4-2qwj5 1/1 Running 0 17h hostpath-provisioner-wxd2k 1/1 Running 0 17h Upgraded CNV v4.9.0 -> 4.9.1 -> 4.9.2-22 [cnv-qe-jenkins@c01-jen49-up2-vkns4-executor ~]$ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v4.9.2 OpenShift Virtualization 4.9.2 kubevirt-hyperconverged-operator.v4.9.1 Succeeded [cnv-qe-jenkins@c01-jen49-up2-vkns4-executor ~]$ oc get ip -n openshift-cnv NAME CSV APPROVAL APPROVED install-2h54g kubevirt-hyperconverged-operator.v4.9.0 Manual true install-7cmmn kubevirt-hyperconverged-operator.v4.9.1 Manual true install-csmg7 kubevirt-hyperconverged-operator.v4.9.2 Manual true [cnv-qe-jenkins@c01-jen49-up2-vkns4-executor ~]$ oc get pods -n openshift-cnv | grep hostpath hostpath-provisioner-8hxxc 1/1 Running 0 162m hostpath-provisioner-knwfv 1/1 Running 0 162m hostpath-provisioner-operator-85984bf8f4-c5xnk 1/1 Running 0 163m hostpath-provisioner-tlwt2 1/1 Running 0 162m Upgraded CNV 4.9.1 -> 4.9.2-22 $ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v4.9.2 OpenShift Virtualization 4.9.2 kubevirt-hyperconverged-operator.v4.9.1 Succeeded $ oc get ip -n openshift-cnv NAME CSV APPROVAL APPROVED install-5g5x8 kubevirt-hyperconverged-operator.v4.9.2 Manual true install-75zzm kubevirt-hyperconverged-operator.v4.9.1 Manual true $ oc get pods -n openshift-cnv | grep hostpath hostpath-provisioner-5xn4c 1/1 Running 0 10m hostpath-provisioner-brpd6 1/1 Running 0 10m hostpath-provisioner-kvdp6 1/1 Running 0 10m hostpath-provisioner-operator-85984bf8f4-qpgh9 1/1 Running 0 10m Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.9.2 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0191 |