Bug 1879072
| Summary: | Deployment with encryption at rest is failing to bring up OSD pods | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | Petr Balogh <pbalogh> | |
| Component: | rook | Assignee: | Sébastien Han <shan> | |
| Status: | CLOSED ERRATA | QA Contact: | Petr Balogh <pbalogh> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.6 | CC: | assingh, bniver, madam, muagarwa, ocs-bugs, shan, sostapov, vavuthu | |
| Target Milestone: | --- | Keywords: | Automation | |
| Target Release: | OCS 4.6.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | 4.6.0-102.ci | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1883927 (view as bug list) | Environment: | ||
| Last Closed: | 2020-12-17 06:24:14 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1883927 | |||
|
Comment 4
Sébastien Han
2020-09-15 14:28:50 UTC
Copying must gather I've collected locally here: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-1879072-logs.tar.gz As I see that the logs from the job wasn't collected well Petr, the reason why the deployment failed is because the Ceph container image does not have the encryption code: [root@13ceeafe1c5f /]# ceph -v ceph version 14.2.8-91.el8cp (75b4845da7d469665bd48d1a49badcc3677bf5cd) nautilus (stable) ceph-volume is missing the encryption flag. The downstream backport was done as per https://bugzilla.redhat.com/show_bug.cgi?id=1845622#c9 Are we using RHCS 4.1z2 rc build? Hey Sebastien. We've discussed this yesterday on program call and in this case we need to get OCS 4.6 DS build which will have new image. I think it's now on Christina's team so moving this to build component now. This can be moved to ON_QA now with the latest 4.6 build. Failed QE. OCP Version: 4.6.0-0.nightly-2020-09-30-052433 Tried with build: 4.6.0-102.ci which suppose to have proper RHCS image: The build should have RHCS 4-33 - confirmed from Boris. Logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr2696-b2972/jnk-pr2696-b2972_20200930T112803/logs/ Job: https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/12913/ OSD Pods: rook-ceph-osd-0-557f59b57d-9tswd 0/1 Init:CrashLoopBackOff 6 7m52s rook-ceph-osd-1-6cc7896876-xjlwk 0/1 Init:CrashLoopBackOff 6 7m52s rook-ceph-osd-2-7d5577c5df-rmfmb 0/1 Init:CrashLoopBackOff 6 7m53s Describe on one of OSD pod: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned openshift-storage/rook-ceph-osd-0-557f59b57d-9tswd to compute-2 Normal SuccessfulMountVolume 11m kubelet, compute-2 MapVolume.MapPodDevice succeeded for volume "pvc-0fb386f8-b2a4-46b4-bbac-16300e41d740" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/volumeDevices/[vsanDatastore] 3c242d5f-6405-9967-713e-e4434bd7dee0/jnk-pr2696-b2972-rg9sv-dynamic-pvc-0fb386f8-b2a4-46b4-bbac-16300e41d740.vmdk" Normal SuccessfulMountVolume 11m kubelet, compute-2 MapVolume.MapPodDevice succeeded for volume "pvc-0fb386f8-b2a4-46b4-bbac-16300e41d740" volumeMapPath "/var/lib/kubelet/pods/32c15db1-cc03-41e4-adfb-44bac2fd587b/volumeDevices/kubernetes.io~vsphere-volume" Normal Started 11m kubelet, compute-2 Started container encryption-open Normal AddedInterface 11m multus Add eth0 [10.128.2.25/23] Normal Pulled 11m kubelet, compute-2 Container image "quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9" already present on machine Normal Created 11m kubelet, compute-2 Created container encryption-open Normal Started 11m kubelet, compute-2 Started container blkdevmapper-encryption Normal Pulled 11m kubelet, compute-2 Container image "quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9" already present on machine Normal Created 11m kubelet, compute-2 Created container blkdevmapper-encryption Normal Pulled 11m kubelet, compute-2 Container image "quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9" already present on machine Normal Created 11m kubelet, compute-2 Created container encrypted-block-status Normal Started 11m kubelet, compute-2 Started container encrypted-block-status Normal Started 11m (x3 over 11m) kubelet, compute-2 Started container expand-encrypted-bluefs Normal Pulled 11m (x4 over 11m) kubelet, compute-2 Container image "quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9" already present on machine Normal Created 11m (x4 over 11m) kubelet, compute-2 Created container expand-encrypted-bluefs Warning BackOff 95s (x46 over 11m) kubelet, compute-2 Back-off restarting failed container Kubeconfig shared with Sebastien: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr2696-b2972/jnk-pr2696-b2972_20200930T112803/openshift-cluster-dir/auth/kubeconfig If the build has encryption enabled then the the component needs to be changed to rook. Actually Sebastien told me to create the clone of this to rook component so I did here: https://bugzilla.redhat.com/show_bug.cgi?id=1883927 So we can close this one now as the build should contain the proper RHCS image. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.6.0 security, bug fix, enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5605 |