Description of problem: cnv is not deployed successfully Version-Release number of selected component (if applicable): "registry-proxy.engineering.redhat.com/rh-osbs/iib:11025" # hco-bundle-registry:v2.5.0-178 How reproducible: always Steps to Reproduce: 1. deploy cnv 2. 3. Additional info: cdi-apiserver-6dbc9f557c-vcngc 0/1 CrashLoopBackOff 11 36m cdi-deployment-5578456969-t8q79 0/1 ContainerCreating 0 36m cdi-operator-8656697c-lj5kb 1/1 Running 0 37m cdi-uploadproxy-79fbd9b747-kz245 0/1 CreateContainerConfigError 0 36m oc logs -nopenshift-cnv cdi-apiserver-6dbc9f557c-vcngc /usr/bin/virt-cdi-apiserver: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory oc describe pod -nopenshift-cnv cdi-apiserver-6dbc9f557c-vcngc Warning FailedMount 41m (x3 over 41m) kubelet MountVolume.SetUp failed for volume "ca-bundle" : configmap "cdi-apiserver-signer-bundle" not found Warning FailedMount 41m kubelet MountVolume.SetUp failed for volume "server-cert" : references non-existent secret key: tls.crt Warning FailedMount 41m kubelet MountVolume.SetUp failed for volume "ca-bundle" : configmap references non-existent config key: ca-bundle.crt Warning FailedMount 41m (x4 over 41m) kubelet MountVolume.SetUp failed for volume "server-cert" : secret "cdi-apiserver-server-cert" not found Normal AddedInterface 40m multus Add eth0 [10.128.2.16/23] Normal Pulling 40m kubelet Pulling image "registry.redhat.io/container-native-virtualization/virt-cdi-apiserver@sha256:01dfc2265a8ef701d1cb7d2070b4d9e15d5f6581f578c945af6f91421ae946cf" Normal Pulled 40m kubelet Successfully pulled image "registry.redhat.io/container-native-virtualization/virt-cdi-apiserver@sha256:01dfc2265a8ef701d1cb7d2070b4d9e15d5f6581f578c945af6f91421ae946cf" in 7.93148258s oc describe pod cdi-uploadproxy-79fbd9b747-kz245 -nopenshift-cnv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 42m default-scheduler Successfully assigned openshift-cnv/cdi-uploadproxy-79fbd9b747-kz245 to cdi-night46-h9qkn-worker-0-pvbtb Warning FailedMount 42m (x4 over 42m) kubelet MountVolume.SetUp failed for volume "server-cert" : references non-existent secret key: tls.crt Warning FailedMount 42m (x5 over 42m) kubelet MountVolume.SetUp failed for volume "client-cert" : references non-existent secret key: tls.crt Normal AddedInterface 42m multus Add eth0 [10.128.2.17/23] Normal Pulling 42m kubelet Pulling image "registry.redhat.io/container-native-virtualization/virt-cdi-uploadproxy@sha256:e3f04555f20029c6cbde357d495d11e27430126dd3f47a743532fcaf94f59613" Normal Pulled 41m kubelet Successfully pulled image "registry.redhat.io/container-native-virtualization/virt-cdi-uploadproxy@sha256:e3f04555f20029c6cbde357d495d11e27430126dd3f47a743532fcaf94f59613" in 24.706506541s Warning Failed 32m (x45 over 41m) kubelet Error: secret "cdi-api-signing-key" not found Normal Pulled 2m19s (x184 over 41m) kubelet Container image "registry.redhat.io/container-native-virtualization/virt-cdi-uploadproxy@sha256:e3f04555f20029c6cbde357d495d11e27430126dd3f47a743532fcaf94f59613" already present on machine oc describe pod cdi-deployment-5578456969-t8q79 -nopenshift-cnv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43m default-scheduler Successfully assigned openshift-cnv/cdi-deployment-5578456969-t8q79 to cdi-night46-h9qkn-worker-0-pvbtb Warning FailedMount 43m (x2 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-ca-cert" : secret "cdi-uploadserver-signer" not found Warning FailedMount 43m (x2 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-ca-bundle" : configmap "cdi-uploadserver-signer-bundle" not found Warning FailedMount 43m (x3 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-client-ca-cert" : secret "cdi-uploadserver-client-signer" not found Warning FailedMount 43m (x3 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-client-ca-bundle" : configmap "cdi-uploadserver-client-signer-bundle" not found Warning FailedMount 43m (x3 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-ca-cert" : references non-existent secret key: tls.crt Warning FailedMount 43m (x3 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-ca-bundle" : configmap references non-existent config key: ca-bundle.crt Warning FailedMount 43m (x2 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-client-ca-bundle" : configmap references non-existent config key: ca-bundle.crt Warning FailedMount 43m (x2 over 43m) kubelet MountVolume.SetUp failed for volume "uploadserver-client-ca-cert" : references non-existent secret key: tls.crt Warning FailedMount 12m (x23 over 43m) kubelet MountVolume.SetUp failed for volume "cdi-api-signing-key" : secret "cdi-api-signing-key" not found Warning FailedMount 7m4s (x2 over 9m21s) kubelet Unable to attach or mount volumes: unmounted volumes=[cdi-api-signing-key], unattached volumes=[uploadserver-client-ca-bundle cdi-sa-token-rlfpd cdi-api-signing-key uploadserver-ca-cert uploadserver-client-ca-cert uploadserver-ca-bundle]: timed out waiting for the condition Warning FailedMount 2m35s (x10 over 41m) kubelet (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[cdi-api-signing-key], unattached volumes=[cdi-sa-token-rlfpd cdi-api-signing-key uploadserver-ca-cert uploadserver-client-ca-cert uploadserver-ca-bundle uploadserver-client-ca-bundle]: timed out waiting for the condition
The root cause is: [cloud-user@ocp-psi-executor ~]$ oc logs -n openshift-cnv cdi-apiserver-6dbc9f557c-hkjk8 /usr/bin/virt-cdi-apiserver: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory
Please notice that CDI operator in upstream release 1.23.1 is not able to start due to: {"level":"error","ts":1600241122.6585944,"logger":"cmd","msg":"","error":"cdis.cdi.kubevirt.io \"cdi\" not found","stacktrace":"kubevirt.io/containerized-data-importer/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\nmain.main\n\tcmd/cdi-operator/operator.go:109\nruntime.main\n\tGOROOT/src/runtime/proc.go:203"} https://github.com/kubevirt/containerized-data-importer/pull/1374 should address it. We fear that the issue is still not visible downstream just because it's shadowed by this one.
Multiple people are working on this (waiting on a pull request from mhenriks+awels and alitke to update downstream.)
Still reproducible with HCO-v2.5.0-286 and IIB-12395
HCO-v2.5.0-286 & IIB-12395 pulled following images: registry.redhat.io/container-native-virtualization/virt-cdi-apiserver@sha256:01dfc2265a8ef701d1cb7d2070b4d9e15d5f6581f578c945af6f91421ae946cf registry.redhat.io/container-native-virtualization/virt-cdi-controller@sha256:e29711fd8733e6641b7ea0b1afff3fc9078833271d0cbdaff6d27fd160e28cfe registry.redhat.io/container-native-virtualization/virt-cdi-operator@sha256:3668f59d1ff7ed7c1f5f88cc72138d7766d7b4784b671c30da7cfc8644f83f92 registry.redhat.io/container-native-virtualization/virt-cdi-uploadproxy@sha256:e3f04555f20029c6cbde357d495d11e27430126dd3f47a743532fcaf94f59613 Note that these addresses are redirected by imageContentSourcePolicy to: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-*
We determined what was pulling in libdevmapper, and figured out a way to not have it included at all.
In the comments on this bugzilla, I see two different issues, for which we have PRs to fix them. 1. dev mapper being needed. 2. Comment #3 where the operator fails to start when deployed through OLM. The problem is use looking for the CR when it is not created yet. These are two separate issues unrelated to each other. We have fixed both, just going through the motions to create a release with them included.
CDI upstream v1.23.2 contains the fix. Still need downstream builds.
working in HCO-v2.5.0-193 cdi-apiserver-f5fb5fc84-jh6qq 1/1 Running 0 23m cdi-deployment-845766b7f5-5kqtv 1/1 Running 0 23m cdi-operator-f96b47b57-qx9bg 1/1 Running 0 23m cdi-uploadproxy-687c79ff74-4qkpm 1/1 Running 0 23m
oc logs cdi-apiserver-7c8b77b8c5-6v5nz -nopenshift-cnv /usr/bin/virt-cdi-apiserver: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory We still see this issue deploying from hco-bundle-registry-container-v2.6.0-5 which points to: - image: registry.redhat.io/container-native-virtualization/virt-cdi-apiserver@sha256:19ade5c8e0954870fb456b31738096ae6567f94c37b08589b5583a3a1bdd2a22 name: registry.redhat.io/container-native-virtualization/virt-cdi-apiserver:v2.6.0-2 ... - image: registry.redhat.io/container-native-virtualization/virt-cdi-operator@sha256:0e4dea18e66838ab4603382415047a9e293080ddf40ebc06164fcaea0913c5eb name: registry.redhat.io/container-native-virtualization/virt-cdi-operator:v2.6.0-2 Adam, can you please trigger a rebuild for 2.6?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5127