Description of the problem: VMExport: can't download a PVC that was created from DV on NFS (when there's no VM that owns this DV/PVC) Version-Release number of selected component (if applicable): 4.12, 4.13 How reproducible: Always on NFS, when DV/PVC is not owned by a VM It works with NFS when VM owns the DV/PVC It works with HPP and OCS with no regard to a VM Steps to Reproduce: 1. Create an NFS DV 2. Create a VMExport for PVC 3. See the VMExport is Ready $ oc get vmexport NAME SOURCEKIND SOURCENAME PHASE export-pvc-nfs PersistentVolumeClaim dv-source Ready 4. Try to download the image, but get an error $ virtctl vmexport download export-pvc-nfs --output=disk-pvc-nfs.img --keep-vme Processing completed successfully Bad status: 500 Internal Server Error 5. See the log: $ oc logs virt-export-export-pvc-nfs | grep error {"component":"virt-exportserver-virt-export-export-pvc-nfs","level":"error","msg":"error opening /export-volumes/dv-source/disk.img","pos":"exportserver.go:315","reason":"open /export-volumes/dv-source/disk.img: permission denied","timestamp":"2022-12-27T09:22:32.371279Z"} Actual results: Bad status: 500 Internal Server Error Expected results: Image downloaded successfully Additional info: $ cat dv-nfs.yaml apiVersion: cdi.kubevirt.io/v1alpha1 kind: DataVolume metadata: name: dv-source spec: source: http: url: <cirros.img> storage: resources: requests: storage: 1Gi storageClassName: nfs volumeMode: Filesystem $ cat vmexport-dv-nfs.yaml apiVersion: export.kubevirt.io/v1alpha1 kind: VirtualMachineExport metadata: name: export-pvc-nfs spec: source: apiGroup: "" kind: PersistentVolumeClaim name: dv-source
This happens because NFS does not support fsGroup https://kubernetes.io/blog/2020/12/14/kubernetes-release-1.20-fsgroupchangepolicy-fsgrouppolicy/#allow-csi-drivers-to-declare-support-for-fsgroup-based-permissions [virt-exportserver@virt-export-export-pvc-nfs ~]$ id uid=1001(virt-exportserver) gid=1001(virt-exportserver) groups=1001(virt-exportserver),107 [virt-exportserver@virt-export-export-pvc-nfs ~]$ ls -la /export-volumes/dv-source/disk.img -rw-rw----. 1 107 99 1073741824 Dec 27 09:17 /export-volumes/dv-source/disk.img [virt-exportserver@virt-export-export-pvc-nfs ~]$ md5sum /export-volumes/dv-source/disk.img md5sum: /export-volumes/dv-source/disk.img: Permission denied I am not sure if there's something we could do here, seeing as we want to run non root (restricted PSA), and that requires fsGroup. While debugging this, I did notice that we don't set runAsGroup in the importer manifests/Dockerfile, which is something we might want to reconsider as it could somehow mitigate this? $ oc exec importer-windows-ey0bvq-installcdrom -it bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. bash-5.1$ id uid=107(107) gid=0(root) groups=0(root),107 bash-5.1$ ls -la /data/disk.img -rw-r--r--. 1 107 99 5694537728 Dec 27 10:30 /data/disk.img @mhenriks WDYT?
Export pod should run as run as user 107.
So I checked https://github.com/kubevirt/kubevirt/blob/main/cmd/virt-launcher/BUILD.bazel#L111-L118 against https://github.com/kubevirt/kubevirt/blob/main/cmd/virt-exportserver/BUILD.bazel#L32-L39 and it doesn't look like the virt-export server is running as 107, but 1001.
Retargeting to 4.13.z, hopefully that is fine if we still want 4.12.z we can make the clone and pursue that as well
Verified on CNV-v4.13.3.rhel9-175 $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros-dv-nfs Bound nfs-pv-01 5Gi RWO,RWX nfs 23s $ virtctl vmexport create export-pvc-nfs --pvc=cirros-dv-nfs VirtualMachineExport 'default/export-pvc-nfs' created succesfully $ oc get vmexport NAME SOURCEKIND SOURCENAME PHASE export-pvc-nfs PersistentVolumeClaim cirros-dv-nfs Ready $ virtctl vmexport download export-pvc-nfs --output=disk-pvc-nfs.img --keep-vme Downloading file: 12.64 MiB [====>______________] 1.23 MiB p/s Download finished succesfully $ ll | grep disk-pvc-nfs.img -rw-rw-r--. 1 cloud-user cloud-user 13255182 Aug 2 10:04 disk-pvc-nfs.img
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.13.3 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:4664