Bug 1796980

Summary:	[GSS] [RFE] Provide a way to provision rook/ceph on bare metal nodes with differing physical device paths
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Brian Ward <bward>
Component:	installation	Assignee:	Sébastien Han <shan>
Status:	CLOSED WORKSFORME	QA Contact:	Petr Balogh <pbalogh>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.2	CC:	assingh, bkunal, hnallurv, madam, ocs-bugs, ratamir, sabose, sostapov
Target Milestone:	---	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-06-02 14:58:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1841422

Description Brian Ward 2020-01-31 16:38:45 UTC

Description of problem (please be detailed as possible and provide log
snippests):

OCS meta operator appears to expect PVCs from LocalVolume objects provisioned first.  LocalVolume objects don't have a clear way to create a set of storageclass on multiple nodes with different device names.  

Then it appears the Rook operator uses these LocalVolume PVCs to create a "PVC-based Cluster" of Ceph backing volumes for its dynamic PVC provisioner.
https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#pvc-based-cluster

The problem we have with LocalVolume objects is that they do not allow us to specify differing device names per node:
https://github.com/openshift/local-storage-operator/blob/master/docs/deploy-with-olm.md#cr-using-volumemode---filesystem
This object assumes that all nodes have exact same names (nodeSelector matchExpressions) for devicePaths.  But with bare metal deployments, we have no way of ensuring the order of devices is the same on each physical host, unless we first use some sort of multipath WWID or device mapper workaround before applying the LocalVolume.

Version of all relevant components (if applicable):

OCS 4.2

Is there any workaround available to the best of your knowledge?
 
Workaround is a Rook deploy using "host-based CephCluster" for now.
https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#host-based-cluster

Comment 4 Jose A. Rivera 2020-02-03 09:20:27 UTC

This is certainly something we should tackle, but is out of scope for OCS 4.3. Moving to 4.4.

Comment 7 Brian Ward 2020-05-06 15:26:38 UTC

It appears "official" documentation is now available:

https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.3/html/deploying_openshift_container_storage/deploying-openshift-container-storage#creating-openshift-container-storage-cluster-on-bare-metal_rhocs

The following example LocalVolume config worked perfectly, picking up the specific devicePath on each hostname where it was available on that hostname.  Clearly this approach would not work with traditional device paths such as "/dev/sda"

apiVersion: local.storage.openshift.io/v1
kind: LocalVolume
metadata:
  name: ocs-backing
  namespace: local-storage
spec:
  logLevel: Normal
  managementState: Managed
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - 10.194.150.196
        - 10.194.150.201
        - 10.194.150.204
  storageClassDevices:
  - devicePaths:
    - /dev/disk/by-id/scsi-3600605b000ac67d02642f5990340a0e6
    - /dev/disk/by-id/scsi-3600605b00b5982502642ac600341c0bb
    - /dev/disk/by-id/scsi-3600605b00b597c202642ac890342e043
    storageClassName: local-ocs-backing
    volumeMode: Block

Comment 9 Michael Adam 2020-05-06 21:28:37 UTC

sounds we're waiting for the customer to confirm that the documented procedure works for them.
Moving the BZ to 4.5 while we wait for the confirmation.

Comment 10 Yaniv Kaul 2020-06-02 14:58:30 UTC

(In reply to Brian Ward from comment #7)
> It appears "official" documentation is now available:
> 
> https://access.redhat.com/documentation/en-us/
> red_hat_openshift_container_storage/4.3/html/
> deploying_openshift_container_storage/deploying-openshift-container-
> storage#creating-openshift-container-storage-cluster-on-bare-metal_rhocs
> 
> The following example LocalVolume config worked perfectly, picking up the
> specific devicePath on each hostname where it was available on that
> hostname.  Clearly this approach would not work with traditional device
> paths such as "/dev/sda"

Those traditional devices are not safe. They may change after boot.
You also have very little guarantee they are the same across nodes.

I'm closing this for the time being. Please re-open if this needs further discussion.