Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): OCS meta operator appears to expect PVCs from LocalVolume objects provisioned first. LocalVolume objects don't have a clear way to create a set of storageclass on multiple nodes with different device names. Then it appears the Rook operator uses these LocalVolume PVCs to create a "PVC-based Cluster" of Ceph backing volumes for its dynamic PVC provisioner. https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#pvc-based-cluster The problem we have with LocalVolume objects is that they do not allow us to specify differing device names per node: https://github.com/openshift/local-storage-operator/blob/master/docs/deploy-with-olm.md#cr-using-volumemode---filesystem This object assumes that all nodes have exact same names (nodeSelector matchExpressions) for devicePaths. But with bare metal deployments, we have no way of ensuring the order of devices is the same on each physical host, unless we first use some sort of multipath WWID or device mapper workaround before applying the LocalVolume. Version of all relevant components (if applicable): OCS 4.2 Is there any workaround available to the best of your knowledge? Workaround is a Rook deploy using "host-based CephCluster" for now. https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#host-based-cluster
This is certainly something we should tackle, but is out of scope for OCS 4.3. Moving to 4.4.
It appears "official" documentation is now available: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.3/html/deploying_openshift_container_storage/deploying-openshift-container-storage#creating-openshift-container-storage-cluster-on-bare-metal_rhocs The following example LocalVolume config worked perfectly, picking up the specific devicePath on each hostname where it was available on that hostname. Clearly this approach would not work with traditional device paths such as "/dev/sda" apiVersion: local.storage.openshift.io/v1 kind: LocalVolume metadata: name: ocs-backing namespace: local-storage spec: logLevel: Normal managementState: Managed nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - 10.194.150.196 - 10.194.150.201 - 10.194.150.204 storageClassDevices: - devicePaths: - /dev/disk/by-id/scsi-3600605b000ac67d02642f5990340a0e6 - /dev/disk/by-id/scsi-3600605b00b5982502642ac600341c0bb - /dev/disk/by-id/scsi-3600605b00b597c202642ac890342e043 storageClassName: local-ocs-backing volumeMode: Block
sounds we're waiting for the customer to confirm that the documented procedure works for them. Moving the BZ to 4.5 while we wait for the confirmation.
(In reply to Brian Ward from comment #7) > It appears "official" documentation is now available: > > https://access.redhat.com/documentation/en-us/ > red_hat_openshift_container_storage/4.3/html/ > deploying_openshift_container_storage/deploying-openshift-container- > storage#creating-openshift-container-storage-cluster-on-bare-metal_rhocs > > The following example LocalVolume config worked perfectly, picking up the > specific devicePath on each hostname where it was available on that > hostname. Clearly this approach would not work with traditional device > paths such as "/dev/sda" Those traditional devices are not safe. They may change after boot. You also have very little guarantee they are the same across nodes. I'm closing this for the time being. Please re-open if this needs further discussion.