Bug 2094829

Summary: node-volume-attach-limit not considered
Product: OpenShift Container Platform Reporter: rlobillo
Component: StorageAssignee: ShiftStack Bugwatcher <shiftstack-bugwatcher>
Storage sub component: OpenStack CSI Drivers QA Contact: Jon Uriarte <juriarte>
Status: CLOSED DEFERRED Docs Contact:
Severity: low    
Priority: low CC: lmurthy, pprinett
Version: 4.11Keywords: Triaged
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: CSINode's count isn't updated by kubelet when CSIDriver provides a new value. Consequence: This causes node-volume-attach-limit cloud provider option modifications to be ignored. Workaround (if any): After modification delete CSINode objects and restart kubelet on all the nodes. You might also just delete all CSINode objects before the modification of cloud provider config. Result:
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-03-09 01:21:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description rlobillo 2022-06-08 11:39:17 UTC 
Description of problem:

Thanks to the work included on https://bugzilla.redhat.com/show_bug.cgi?id=2065597, now it is possible to configure cinder-csi by editing cloud-provider-config.

One of the parameters included is 'node-volume-attach-limit' [1]. Changing the param to 10 and running 11 pods connected to 11 different PVCs in the same worker is accepted:

	$ oc get cm -n openshift-cluster-csi-drivers cloud-conf -o yaml
	apiVersion: v1
	data:
	  cloud.conf: |+
		[Global]
		region      = regionOne
		ca-file     = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
		use-clouds  = true
		clouds-file = /etc/kubernetes/secret/clouds.yaml
		cloud       = openstack

		[LoadBalancer]
		use-octavia = True

		[BlockStorage]
		ignore-volume-az         = yes
		node-volume-attach-limit = 10

		[Metadata]
		search-order = metadataService

	$ oc get pods
	NAME                       READY   STATUS    RESTARTS   AGE
	demo-0-8466fc476-mbll5     1/1     Running   0          2m17s
	demo-1-649948d68-dnhsv     1/1     Running   0          2m14s
	demo-10-676c554976-52bwt   1/1     Running   0          103s
	demo-2-6947d8d46d-mtxn6    1/1     Running   0          2m11s
	demo-3-79db5fd4d8-qxgbl    1/1     Running   0          2m7s
	demo-4-65678d5484-qlngk    1/1     Running   0          2m4s
	demo-5-f75648d76-wzcmw     1/1     Running   0          2m
	demo-6-856dc89dd8-s9bp4    1/1     Running   0          117s
	demo-7-64f5944f8d-shq4d    1/1     Running   0          114s
	demo-8-7b677bb644-4bhnn    1/1     Running   0          111s
	demo-9-9b88b4c56-m7swp     1/1     Running   0          107s
	$ oc get pvc
	NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
	pvc-0    Bound    pvc-92aaeab8-6c60-4f32-9ab6-a703761ad3cd   1Gi        RWO            topology-aware-0    2m19s
	pvc-1    Bound    pvc-336915af-8154-4e67-b37d-2fa93f2db645   1Gi        RWO            topology-aware-1    2m16s
	pvc-10   Bound    pvc-a7f58825-e465-4e2b-b9ec-36362589d1d2   1Gi        RWO            topology-aware-10   105s
	pvc-2    Bound    pvc-4ded3dab-170d-4573-ba29-8bb10b3e857b   1Gi        RWO            topology-aware-2    2m13s
	pvc-3    Bound    pvc-b3d7ec16-5b7e-4964-9a0a-29480dee31a1   1Gi        RWO            topology-aware-3    2m9s
	pvc-4    Bound    pvc-a6ae7415-00b5-4849-8065-7fe7acace165   1Gi        RWO            topology-aware-4    2m6s
	pvc-5    Bound    pvc-246421b3-2716-41c8-88e9-d6253c49acdd   1Gi        RWO            topology-aware-5    2m2s
	pvc-6    Bound    pvc-61ec111a-d5ae-47d2-83be-1a4a344afea7   1Gi        RWO            topology-aware-6    119s
	pvc-7    Bound    pvc-b65cd362-977a-4613-84a3-bb75b1adf971   1Gi        RWO            topology-aware-7    116s
	pvc-8    Bound    pvc-a79d7bb1-9697-4c1b-8ad3-696a0c2edafd   1Gi        RWO            topology-aware-8    113s
	pvc-9    Bound    pvc-48e37bc7-ff50-4517-8cac-a57fd6f0f47b   1Gi        RWO            topology-aware-9    109s
	$ openstack volume list -c 'Attached to' | grep ostest | awk -F' ' '{print $4}' | sort | uniq -c
		 11 ostest-zh8g7-worker-0-7gl4r
	$ openstack volume list
	+--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+
	| ID                                   | Name                                     | Status | Size | Attached to                                          |
	+--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+
	| ce816f7d-6873-457a-acfa-327cf9357c4a | pvc-a7f58825-e465-4e2b-b9ec-36362589d1d2 | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdl  |
	| dd774cc3-488c-4fa7-bf5d-f84c3d67f99b | pvc-48e37bc7-ff50-4517-8cac-a57fd6f0f47b | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdk  |
	| 1fc32fa8-feeb-4f06-a712-5b9da301e187 | pvc-a79d7bb1-9697-4c1b-8ad3-696a0c2edafd | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdj  |
	| 0ced70af-0a85-4cd2-bfab-f7299b4bd128 | pvc-b65cd362-977a-4613-84a3-bb75b1adf971 | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdi  |
	| dc413629-382b-492b-9e41-ae9163ff542f | pvc-61ec111a-d5ae-47d2-83be-1a4a344afea7 | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdh  |
	| 7aab9069-dc48-4413-b4b9-f7c31bdaeaa6 | pvc-246421b3-2716-41c8-88e9-d6253c49acdd | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdg  |
	| c1656f81-3d55-4f15-aab8-d84b37ceae9b | pvc-a6ae7415-00b5-4849-8065-7fe7acace165 | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdf  |
	| d8608274-5a75-4daf-adb5-575160b296e3 | pvc-b3d7ec16-5b7e-4964-9a0a-29480dee31a1 | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vde  |
	| c59bc638-f410-4d3e-a3b9-d31288a0c4a0 | pvc-4ded3dab-170d-4573-ba29-8bb10b3e857b | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdd  |
	| 7f779063-61f1-462b-bb9d-f77429dc9485 | pvc-336915af-8154-4e67-b37d-2fa93f2db645 | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdc  |
	| 2c17fa64-1b3a-44e1-8ad0-8fb5c4e3ffc1 | pvc-92aaeab8-6c60-4f32-9ab6-a703761ad3cd | in-use |    1 | Attached to ostest-zh8g7-worker-0-7gl4r on /dev/vdb  |
	+--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+

[1] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#driver-config

Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-06-06-025509 RHOS-16.2-RHEL-8-20220311.n.1

How reproducible: Always

Steps to Reproduce:
1. Change cloud cm to node-volume-attach-limit = 10
2. Wait until cluster re-configuration is completed (around 30 minutes)
3. Create 11 pods using cinder PVCs with NodeSelector set to use always the same worker.

Actual results: The node-volume-attach-limit is ignored.
Expected results: The PVC number 11 should not becreated.
Comment 2 Michał Dulko 2022-07-07 16:30:21 UTC 
Okay, a simple workaround to the problem is to delete CSINode objects and restart kubelet on every of the nodes. Then the node-volume-attach-limit is correctly put there and everything will work find. We don't understand the root cause yet, but clearly it's an issue with the fact you're updating the configuration after the installation.
Comment 5 Shiftzilla 2023-03-09 01:21:01 UTC 
OpenShift has moved to Jira for its defect tracking! This bug can now be found in the OCPBUGS project in Jira.

https://issues.redhat.com/browse/OCPBUGS-9308