Bug 1711607 - All driver csi-hostpath e2e tests fail
Summary: All driver csi-hostpath e2e tests fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.4.0
Assignee: Jan Safranek
QA Contact: Chao Yang
URL:
Whiteboard:
: 1695202 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-18 23:27 UTC by Clayton Coleman
Modified: 2020-05-13 21:51 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-13 21:51:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24281 0 None closed Bug 1711607: Run CSI hostpath containers privileged during tests 2020-07-02 05:55:22 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-13 21:51:46 UTC

Description Clayton Coleman 2019-05-18 23:27:13 UTC
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22858/pull-ci-openshift-origin-master-e2e-aws/8784#openshift-tests-sig-storage-csi-volumes-driver-csi-hostpath-v0-testpattern-dynamic-pv-default-fs-provisioning-should-allow-concurrent-writes-on-the-same-node-suiteopenshiftconformanceparallel-suitek8s

fail [k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:314]: Unexpected error:
    <*errors.errorString | 0xc0009f7400>: {
        s: "PersistentVolumeClaims [pvc-pdxsk] not all in phase Bound within 5m0s",
    }
    PersistentVolumeClaims [pvc-pdxsk] not all in phase Bound within 5m0s
occurred
			
Test is currently disabled in 4.2

Comment 2 Tomas Smetana 2019-05-20 14:39:30 UTC
SELinux issue most likely:

May 18 20:50:07 ip-10-0-128-254 kernel: audit: type=1400 audit(1558212607.086:7): avc:  denied  { write } for  pid=27152 comm="csi-attacher" name="csi.sock" dev="xvda3" ino=234906324 scontext=system_u:system_r:container_t:s0:c14,c29 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file permissive=0
May 18 20:50:07 ip-10-0-128-254 kernel: audit: type=1400 audit(1558212607.132:8): avc:  denied  { write } for  pid=27303 comm="csi-provisioner" name="csi.sock" dev="xvda3" ino=234906324 scontext=system_u:system_r:container_t:s0:c14,c29 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file permissive=0

Comment 3 Hemant Kumar 2019-05-23 21:25:08 UTC
I have opened a initial PR to make hostpath containers privileged as necessary - https://github.com/kubernetes/kubernetes/pull/78265 . The other thing though was - how do we make non-privileged pods "consume" the provisioned hostpath. As matthew commented in - https://github.com/openshift/origin/pull/22446#issuecomment-479687469, this can be done by using `/tmp` as "root" directory where hostpath volumes get provisioned. But this was recently changed in hostpath driver - https://github.com/kubernetes-csi/csi-driver-host-path/pull/20/files . We can obviously patch hostpath YAML files so that, "/tmp"(of host) can be mounted as a"/csi-data-dir" inside container but since we will making same change in upstream e2e YAMLs as well, it might be worth choosing a "stable" root directory path.

I filed https://github.com/kubernetes-csi/csi-driver-host-path/issues/52 for that.

In the meanwhile, this issue perhaps can just be worked around in YAML files, so that we can mount "/tmp" of host as "/csi-data-dir" inside container and update these YAMLs once the root directory becomes configurable.

Comment 4 Hemant Kumar 2019-05-23 22:45:58 UTC
Another problem appears to be the incosistent way these tests apply securityContexts. First pod that writes bits of HTML to hostpath volume is privileged - https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/volume/fixtures.go#L645 but second pod which checks the volume is not and uses different security context. Naturally second pod is unable to read file written by first pod (even if we are using tmp), because of mismatch of SELinux categories...

Comment 5 Fabio Bertinatto 2019-07-15 08:41:08 UTC
*** Bug 1695202 has been marked as a duplicate of this bug. ***

Comment 6 Hemant Kumar 2019-07-16 14:53:57 UTC
The problem is - the writer and reader e2e check is shared by all volume drivers and changing their privileges affects all drivers. There is a discussion in upstream to parameterize that and may be make it possible for a particular driver to opt-in for privileged pods.

Comment 15 Chao Yang 2020-02-03 05:56:39 UTC
Checked the testcases including snapshot are passed. 
Update the test case status.

Comment 17 errata-xmlrpc 2020-05-13 21:51:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.