Bug 2109455

Summary:	Provide documentation for enabling rxbounce option for krbd when using ODF
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Orit Wasserman <owasserm>
Component:	documentation	Assignee:	Olive Lakra <olakra>
Status:	NEW ---	QA Contact:	Neha Berry <nberry>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.12	CC:	alitke, apinnick, asriram, bbenshab, bkunal, fdeutsch, idryomov, jhopper, muagarwa, ndevos, odf-bz-bot, olakra, owasserm, pelauter
Target Milestone:	---	Flags:	ndevos: needinfo? (olakra)
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1973317, 2051525
Bug Blocks:

Description Orit Wasserman 2022-07-21 09:43:50 UTC

Describe the issue:
When running windows VMs with krbd the user may get bad crc/signature errors, see more: https://bugzilla.redhat.com/show_bug.cgi?id=1973317

We introduced a new flag for krbd rxbounce to solve this issue. The users will need to update the StorageClass they are using to enable it.

Describe the task you were trying to accomplish:

Document the new rxbounce options and how to enable it in ODF

Suggestions for improvement:

Document URL:

Chapter/Section Number and Title:

Product Version:

Environment Details:

Any other versions of this document that also needs this update:

Additional information:

Comment 2 Niels de Vos 2022-07-21 12:03:26 UTC

An example of a new StorageClass that has the "rxbounce" option set:

$ oc get sc/windows-vms -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: windows-vms
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  imageFeatures: layering
  imageFormat: "2"
  pool: ocs-storagecluster-cephblockpool
  mapOptions: "krbd:rxbounce"
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate


Note that some options were stripped. The important part is the added "mapOptions:" value. The rest of the StorageClass is just a copy of the default "ocs-storagecluster-ceph-rbd" one.

Comment 3 Niels de Vos 2022-07-22 10:21:04 UTC

Correcting myself, as the StorageClass in comment #2 is not functional for ODF-4.10, it needs a "mounter: rbd" option as well. The following StorageClass has successfully been tested:

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: windows-vms
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  imageFeatures: layering
  imageFormat: "2"
  pool: ocs-storagecluster-cephblockpool
  mounter: rbd
  mapOptions: "krbd:rxbounce"
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---

Validation can be done by checking the logs from the csi-rbdplugin pod that runs on the node where a pod uses the PVC:

$ oc -n openshift-storage logs -c csi-rbdplugin csi-rbdplugin-b4zh6
...
I0722 10:01:56.798742  198188 utils.go:191] ID: 10 Req-ID: 0001-0011-openshift-storage-0000000000000001-4cd4fad2-09a5-11ed-b174-0a580a800216 GRPC call: /csi.v1.Node/NodeStageVolume
...
I0722 10:01:57.060675  198188 cephcmds.go:63] ID: 10 Req-ID: 0001-0011-openshift-storage-0000000000000001-4cd4fad2-09a5-11ed-b174-0a580a800216 command succeeded: rbd [--id csi-rbd-node -m 172.30.83.204:6789,172.30.129.151:6789,172.30.134.46:6789 --keyfile=***stripped*** map ocs-storagecluster-cephblockpool/csi-vol-4cd4fad2-09a5-11ed-b174-0a580a800216 --device-type krbd --options noudev --options rxbounce]
I0722 10:01:57.060749  198188 nodeserver.go:397] ID: 10 Req-ID: 0001-0011-openshift-storage-0000000000000001-4cd4fad2-09a5-11ed-b174-0a580a800216 rbd image: ocs-storagecluster-cephblockpool/csi-vol-4cd4fad2-09a5-11ed-b174-0a580a800216 was successfully mapped at /dev/rbd0
...

This shows that the node has /dev/rbd0 mapped. It should also have the "rxbounce" option set:

$ oc -n openshift-storage exec -c csi-rbdplugin csi-rbdplugin-b4zh6 -- cat /sys/block/rbd0/device/config_info
172.30.129.151:6789,172.30.134.46:6789,172.30.83.204:6789 name=csi-rbd-node,key=client.csi-rbd-node,rxbounce ocs-storagecluster-cephblockpool csi-vol-4cd4fad2-09a5-11ed-b174-0a580a800216 -

Comment 36 Peter Lauterbach 2023-03-10 19:15:38 UTC

to further enhance this, the validation steps in comment 3, to make sure you've turned this on correctly, is a horrible and error prone user experience.