Description of problem: From the doc: https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/persistent-storage-hostpath.html On the kata node, create /mnt/data. $ cat pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain hostPath: path: "/mnt/data" $ oc create -f pv.yaml $ cat pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pvc-volume spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: manual $ oc create -f pvc.yaml $ cat pod.yaml apiVersion: v1 kind: Pod metadata: name: pod-sleep spec: containers: - name: pod-sleep image: ubi8 securityContext: privileged: true volumeMounts: - mountPath: /data name: hostpath-privileged command: ["sleep"] args: ["10000"] securityContext: {} volumes: - name: hostpath-privileged persistentVolumeClaim: claimName: task-pvc-volume runtimeClassName: kata $ oc create -f pod.yaml We end up with unable to create the pod. Warning Failed 30s (x11 over 10m) kubelet Error: CreateContainer failed: Timeout reached after 10s waiting for device 0:0:0:0/block: unknown Version-Release number of selected component (if applicable): kata 1.11.3-1.el8 How reproducible: always
Might be related to https://github.com/kata-containers/runtime/pull/2736
I can reproduce this on 4.7.0-0.nightly-2020-12-09-112139
It might help to have someone amilar with kata block device hotplug code to take a look.
This works fine for kata without: securityContext: privileged: true Will move this to documentation bug instead.
This use case is broken on OCP 4.8 + kata 2.1-alpha now. Without privileged, we have selinux deny where I am unsure why virtiofsd needs to access the block device, but inside the container, we get, [root@pod-sleep /]# ls /data ls: cannot open directory '/data': Permission denied time->Wed Mar 31 17:16:54 2021 type=PROCTITLE msg=audit(1617211014.327:176): proctitle=2F7573722F6C6962657865632F76697274696F667364002D2D66643D33002D6F00736F757263653D2F72756E2F6B6174612D636F6E7461696E6572732F7368617265642F73616E64626F7865732F3731386636356132373535353934343930316131666232376331643538616264653632653037383934623266633566303563 type=SYSCALL msg=audit(1617211014.327:176): arch=c000003e syscall=257 success=no exit=-13 a0=24 a1=56539a354bdf a2=0 a3=0 items=0 ppid=106296 pid=106305 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="pool" exe="/usr/libexec/virtiofsd" subj=system_u:system_r:container_kvm_t:s0:c510,c999 key=(null) type=AVC msg=audit(1617211014.327:176): avc: denied { read } for pid=106305 comm="pool" name="data" dev="sda4" ino=304115508 scontext=system_u:system_r:container_kvm_t:s0:c510,c999 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0 # audit2allow -a #============= container_kvm_t ============== allow container_kvm_t var_t:dir read; With privileged, i.e., securityContext: privileged: true The pod started to fail, Warning Failed 4s (x2 over 6s) kubelet Error: CreateContainer failed: EINVAL: Invalid argument: unknown msg="createContainer failed" error="rpc error: code = Internal desc = EINVAL: Invalid argument" name=containerd-shim-v2 pid=81304 sandbox=15c64f19f2759ca0d2b6cc49a7d1997d937ace8f35dd9c432042ada4d3df2338 source=virtcontainers subsystem=kata_agent msg="container create failed" error="rpc error: code = Internal desc = EINVAL: Invalid argument" name=containerd-shim-v2 pid=81304 sandbox=15c64f19f2759ca0d2b6cc49a7d1997d937ace8f35dd9c432042ada4d3df2338 source=virtcontainers subsystem=container
# rpm -qa | grep kata kata-containers-2.1-0.alpha0.018454b.el8.x86_64 # rpm -qa | grep qemu qemu-guest-agent-4.2.0-34.module+el8.3.0+9903+ca3e42fb.4.x86_64 ipxe-roms-qemu-20181214-6.git133f4c47.el8.noarch qemu-kvm-common-5.1.0-20.module+el8.3.1+9918+230f5c26.x86_64 qemu-kiwi-5.1.0-20.module+el8.3.1+9918+230f5c26.x86_64 # oc version Server Version: 4.8.0-0.nightly-2021-03-31-034951 Kubernetes Version: v1.20.0+29a606d
I've noticed that we had a regression on CRI-O related to privileged what may very well be the cause of the breakage noticed by Qian on the comment#6. A patch has been submitted upstream, to CRI-O (https://github.com/cri-o/cri-o/pull/4896), and Fupan Li is also checking whether it should (also or totally, still unsure) be worked around on the kata-contaienrs side.
The discussion on this issue followed in https://issues.redhat.com/browse/KATA-469.
This has been closed as WONTFIX.
*** Bug 1973184 has been marked as a duplicate of this bug. ***