Bug 1904609

Summary:	unable to use hostPath persistent storage
Product:	OpenShift Container Platform	Reporter:	Qian Cai <qcai>
Component:	sandboxed-containers	Assignee:	Christophe de Dinechin <dinechin>
Status:	CLOSED WONTFIX	QA Contact:	Cameron Meadors <cmeadors>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.8	CC:	aos-bugs, dinechin, ehadley, fgiudici, gkurz, wduan
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-25 08:40:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Qian Cai 2020-12-04 22:42:04 UTC

Description of problem:
From the doc:
https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/persistent-storage-hostpath.html

On the kata node, create /mnt/data.

$ cat pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume 
  labels:
    type: local
spec:
  storageClassName: manual 
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce 
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: "/mnt/data"

$ oc create -f pv.yaml

$ cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pvc-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: manual

$ oc create -f pvc.yaml

$ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-sleep 
spec:
  containers:
    - name: pod-sleep
      image: ubi8
      securityContext:
        privileged: true 
      volumeMounts:
      - mountPath: /data 
        name: hostpath-privileged
      command: ["sleep"]
      args: ["10000"]
  securityContext: {}
  volumes:
    - name: hostpath-privileged
      persistentVolumeClaim:
        claimName: task-pvc-volume
  runtimeClassName: kata

$ oc create -f pod.yaml

We end up with unable to create the pod.

Warning  Failed     30s (x11 over 10m)   kubelet            Error: CreateContainer failed: Timeout reached after 10s waiting for device 0:0:0:0/block: unknown

Version-Release number of selected component (if applicable):
kata 1.11.3-1.el8

How reproducible:
always

Comment 2 Qian Cai 2020-12-10 15:36:23 UTC

Might be related to https://github.com/kata-containers/runtime/pull/2736

Comment 3 Cameron Meadors 2020-12-10 21:08:02 UTC

I can reproduce this on 4.7.0-0.nightly-2020-12-09-112139

Comment 4 Qian Cai 2021-01-27 16:07:59 UTC

It might help to have someone amilar with kata block device hotplug code to take a look.

Comment 5 Qian Cai 2021-02-03 18:54:15 UTC

This works fine for kata without:

      securityContext:
        privileged: true

Will move this to documentation bug instead.

Comment 6 Qian Cai 2021-03-31 17:28:46 UTC

This use case is broken on OCP 4.8 + kata 2.1-alpha now.

Without privileged, we have selinux deny where I am unsure why virtiofsd needs to access the block device, but inside the container, we get,

[root@pod-sleep /]# ls /data
ls: cannot open directory '/data': Permission denied 

time->Wed Mar 31 17:16:54 2021
type=PROCTITLE msg=audit(1617211014.327:176): proctitle=2F7573722F6C6962657865632F76697274696F667364002D2D66643D33002D6F00736F757263653D2F72756E2F6B6174612D636F6E7461696E6572732F7368617265642F73616E64626F7865732F3731386636356132373535353934343930316131666232376331643538616264653632653037383934623266633566303563
type=SYSCALL msg=audit(1617211014.327:176): arch=c000003e syscall=257 success=no exit=-13 a0=24 a1=56539a354bdf a2=0 a3=0 items=0 ppid=106296 pid=106305 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="pool" exe="/usr/libexec/virtiofsd" subj=system_u:system_r:container_kvm_t:s0:c510,c999 key=(null)
type=AVC msg=audit(1617211014.327:176): avc:  denied  { read } for  pid=106305 comm="pool" name="data" dev="sda4" ino=304115508 scontext=system_u:system_r:container_kvm_t:s0:c510,c999 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0

# audit2allow -a


#============= container_kvm_t ==============
allow container_kvm_t var_t:dir read;

With privileged, i.e.,

      securityContext:
        privileged: true

The pod started to fail,

  Warning  Failed          4s (x2 over 6s)  kubelet            Error: CreateContainer failed: EINVAL: Invalid argument: unknown

msg="createContainer failed" error="rpc error: code = Internal desc = EINVAL: Invalid argument" name=containerd-shim-v2 pid=81304 sandbox=15c64f19f2759ca0d2b6cc49a7d1997d937ace8f35dd9c432042ada4d3df2338 source=virtcontainers subsystem=kata_agent
msg="container create failed" error="rpc error: code = Internal desc = EINVAL: Invalid argument" name=containerd-shim-v2 pid=81304 sandbox=15c64f19f2759ca0d2b6cc49a7d1997d937ace8f35dd9c432042ada4d3df2338 source=virtcontainers subsystem=container

Comment 7 Qian Cai 2021-03-31 17:31:31 UTC

# rpm -qa | grep kata
kata-containers-2.1-0.alpha0.018454b.el8.x86_64

# rpm -qa | grep qemu
qemu-guest-agent-4.2.0-34.module+el8.3.0+9903+ca3e42fb.4.x86_64
ipxe-roms-qemu-20181214-6.git133f4c47.el8.noarch
qemu-kvm-common-5.1.0-20.module+el8.3.1+9918+230f5c26.x86_64
qemu-kiwi-5.1.0-20.module+el8.3.1+9918+230f5c26.x86_64

# oc version
Server Version: 4.8.0-0.nightly-2021-03-31-034951
Kubernetes Version: v1.20.0+29a606d

Comment 9 Fabiano Fidêncio 2021-05-13 12:37:46 UTC

I've noticed that we had a regression on CRI-O related to privileged what may very well be the cause of the breakage noticed by Qian on the comment#6.

A patch has been submitted upstream, to CRI-O (https://github.com/cri-o/cri-o/pull/4896), and Fupan Li is also checking whether it should (also or totally, still unsure) be worked around on the kata-contaienrs side.

Comment 12 Francesco Giudici 2022-03-25 08:40:24 UTC

The discussion on this issue followed in https://issues.redhat.com/browse/KATA-469.

Comment 13 Christophe de Dinechin 2022-06-21 09:30:30 UTC

This has been closed as WONTFIX.

Comment 14 Jens Freimann 2022-09-21 08:35:52 UTC

*** Bug 1973184 has been marked as a duplicate of this bug. ***