Bug 2162252

Alexander PTAL

Yes it is broken, I suspect it is because of some selinux labeling. I created a kubevirtci cluster, with both local and nfs storage, then I added a blank local volume and a blank nfs volume. When I look in the virt-launcher pod I see:

bash-5.1$ ls -alZ
total 5267412
drwxrwxrwx. 2 root root system_u:object_r:container_file_t:s0:c449,c794          64 Apr  5 13:35 .
drwxrwxrwx. 5 root root system_u:object_r:container_file_t:s0:c449,c794          96 Apr  5 13:25 ..
-rw-rw----. 1 qemu qemu system_u:object_r:container_file_t:s0:c449,c794  5073010688 Apr  5 13:33 volume-hotplug-local.img
-rw-rw----. 1 qemu qemu system_u:object_r:nfs_t:s0                      10146021376 Apr  5 13:36 volume-hotplug.img

Note the selinux label of the nfs volume contains nfs_t instead of container_file_t. The local volume was added successfully, but the nfs volume is showing the exact error report.

Can you show me nfs storage class? I tried other nfs servers (trident-nfs) and it worked. so I suspect it is simply a configuration issue in the nfs server.

$ oc get sc nfs -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2023-04-03T14:03:00Z"
  name: nfs
  resourceVersion: "73762"
  uid: bd1d7758-2dc5-4e80-a439-81e4e95595b7
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
 
 
$ oc get storageprofile nfs -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  creationTimestamp: "2023-04-03T14:03:00Z"
  generation: 3
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.13.0
    cdi.kubevirt.io: ""
  name: nfs
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi-kubevirt-hyperconverged
    uid: 3bbcd4d7-1305-4cac-bc6a-f470672a6159
  resourceVersion: "74089"
  uid: 924b15d5-b4fe-415f-90cb-26d7f5e2bb4b
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteMany
    volumeMode: Filesystem
status:
  claimPropertySets:
  - accessModes:
    - ReadWriteMany
    volumeMode: Filesystem
  provisioner: kubernetes.io/no-provisioner
  storageClass: nfs

$ oc get sc nfs -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2023-04-03T14:03:00Z"
  name: nfs
  resourceVersion: "73762"
  uid: bd1d7758-2dc5-4e80-a439-81e4e95595b7
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
 
 
$ oc get storageprofile nfs -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  creationTimestamp: "2023-04-03T14:03:00Z"
  generation: 3
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.13.0
    cdi.kubevirt.io: ""
  name: nfs
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi-kubevirt-hyperconverged
    uid: 3bbcd4d7-1305-4cac-bc6a-f470672a6159
  resourceVersion: "74089"
  uid: 924b15d5-b4fe-415f-90cb-26d7f5e2bb4b
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteMany
    volumeMode: Filesystem
status:
  claimPropertySets:
  - accessModes:
    - ReadWriteMany
    volumeMode: Filesystem
  provisioner: kubernetes.io/no-provisioner
  storageClass: nfs

So after some more debugging the problem is not with selinux. I am able to hotplug nfs volumes if the boot volume is not NFS. For some reason when the virt-handler mounts the volume in the virt-launcher pod, it finds the wrong NFS disk, and this why you are seeing the unable to get lock message, since that image is already locked for the boot disk. Investigating why that is happening.

Test on CNV-v4.13.1.rhel9-123, issue has been fixed.

    volumeStatus:
    - hotplugVolume:
        attachPodName: hp-volume-sdn94
        attachPodUID: e350f374-ed7d-4ea5-8687-2096d96dac5b
      message: Successfully attach hotplugged volume blank-dv to VM
      name: blank-dv
      persistentVolumeClaimInfo:
        accessModes:
        - ReadWriteOnce
        capacity:
          storage: 5Gi
        filesystemOverhead: "0.055"
        requests:
          storage: 1Gi
        volumeMode: Filesystem
      phase: Ready
      reason: VolumeReady
      target: sda
    - name: cloudinitdisk
      size: 1048576
      target: vdb
    - name: dv-disk
      persistentVolumeClaimInfo:
        accessModes:
        - ReadWriteOnce
        capacity:
          storage: 25Gi
        filesystemOverhead: "0.055"
        requests:
          storage: 10Gi
        volumeMode: Filesystem
      target: vda
kind: List
metadata:
  resourceVersion: ""

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.13.1 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:3686