Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: I am trying to deploy HPP, using HPP operator which comes with HCO for CNV-2.2 . I am using latest HCO bundle 163.0.0. I deploy CNV-2.2 using market place, and I see hostpath-provisioner-operator pod running. [cnv-qe-jenkins@cnv-executor-lbednar2 ~]$ oc get pods --all-namespaces | grep hostpa openshift-cnv hostpath-provisioner-operator-74dc75578d-75mmt 1/1 Running 0 11m Then I create HPP-CR [1] and then I see it in error state [1] https://raw.githubusercontent.com/kubevirt/hostpath-provisioner-operator/master/deploy/hostpathprovisioner_cr.yaml [cnv-qe-jenkins@cnv-executor-lbednar2 ~]$ oc get pods --all-namespaces | grep hostpa openshift-cnv hostpath-provisioner-operator-74dc75578d-75mmt 0/1 Error 4 14m When I take a look at the logs (full log in additional-info) I see: /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/asm_amd64.s:1337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x12bb60d] Version-Release number of selected component (if applicable): cnv-2.2.0 HCO bundle 163.0.0 How reproducible: Steps to Reproduce: 1. Deploy CNV from market place 2. create HPP CR [1] Actual results: HPP operator failing Expected results: HPP is deployed by HPP operator Additional info: [cnv-qe-jenkins@cnv-executor-lbednar2 ~]$ oc logs -n openshift-cnv hostpath-provisioner-operator-74dc75578d-75mmt {"level":"info","ts":1575469059.5784547,"logger":"cmd","msg":"Go Version: go1.12.12"} {"level":"info","ts":1575469059.5785656,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1575469059.5785706,"logger":"cmd","msg":"Version of operator-sdk: v0.11.0"} {"level":"info","ts":1575469059.5794225,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1575469062.418997,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."} {"level":"info","ts":1575469062.4190483,"logger":"leader","msg":"Continuing as the leader."} {"level":"info","ts":1575469065.2271235,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1575469065.2276523,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.2278988,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.2280571,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.2282245,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.2284064,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.2289288,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.229043,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"hostpathprovisioner-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575469065.2293298,"logger":"cmd","msg":"Starting the Cmd."} E1204 14:17:45.255169 1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io) {"level":"info","ts":1575469065.329994,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"hostpathprovisioner-controller"} {"level":"info","ts":1575469065.4303026,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"hostpathprovisioner-controller","worker count":1} {"level":"info","ts":1575469065.4304929,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"hostpath-provisioner"} E1204 14:17:45.430814 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:522 /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:82 /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/signal_unix.go:390 /go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:202 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/asm_amd64.s:1337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x12bb60d] goroutine 662 [running]: kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105 panic(0x1488ae0, 0x268bae0) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:522 +0x1b5 kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile(0xc0004af000, 0x0, 0x0, 0xc0002453e0, 0x14, 0x26a69e0, 0xc0006ee240, 0xc000670d88, 0xc000670db8) /go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:202 +0x5fd kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000015720, 0x14e5e00, 0xc0004bbf00, 0x14e5e00) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216 +0x146 kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000015720, 0xc0001ade00) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192 +0xb5 kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000015720) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171 +0x2b kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000397940) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54 kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000397940, 0x3b9aca00, 0x0, 0xc000000001, 0xc0009b60c0) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8 kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000397940, 0x3b9aca00, 0xc0009b60c0) /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start /go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x311
The problem is the version.txt file is missing in the operator container, it reads that file to determine which version it is (needed for upgrades for instance). I suspect the file is not being created during the build of the container.
Both Alexander and I were able to deploy successfully using the following images which are currently attached to the errata and specified in the HCO bundle: hco-bundle-registry-container-v2.2.0-70 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1030906 registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hostpath-provisioner-rhel8-operator:v2.2.0-7 registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hostpath-provisioner-rhel8:v2.2.0-5 Therefore, I think your issue must be due to the use of stale images. Additionally, the segfault from your logs (/go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:202) references the following code in broken versions: cr.Status.OperatorVersion = *versionString and in fixed versions that line is: MarkCrDeploying(cr, "DeployStarted", "Started Deployment") which cannot segfault.
verified. We are able to activate the hpp.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0307