Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1611816

Summary: SeLinux blocks CNS installation on OCP 311
Product: Red Hat Enterprise Linux 7 Reporter: Hongkai Liu <hongkliu>
Component: container-selinuxAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: dwalsh, ekuric, hongkliu, lsu, lvrabec, mgrepl, mifiedle, mmalik, plautrba, ssekidde, vmojzis
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-23 17:40:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hongkai Liu 2018-08-02 19:27:08 UTC 
Description of problem:


Version-Release number of selected component (if applicable):
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
# uname -a
Linux ip-172-31-48-190.us-west-2.compute.internal 3.10.0-926.el7.x86_64 #1 SMP Thu Jul 19 11:52:14 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux


# yum list installed | grep -i selinux
container-selinux.noarch        2:2.68-1.el7      @rhel-7-extras-next           
libselinux.x86_64               2.5-14.1.el7      @rhel-7-next                  
libselinux-python.x86_64        2.5-14.1.el7      @rhel-7-next                  
libselinux-utils.x86_64         2.5-14.1.el7      @rhel-7-next                  
selinux-policy.noarch           3.13.1-207.el7    @rhel-7-next                  
selinux-policy-targeted.noarch  3.13.1-207.el7    @rhel-7-next    

How reproducible:
I start to see this problem there:
https://bugzilla.redhat.com/show_bug.cgi?id=1600498

And today see this again.


Steps to Reproduce:
1. Install CNS on OCP 311
2.
3.

Actual results:
"Permission denied" in heketi log

Expected results:
heketi pod is running.


Additional info:


Tested with
# git log --oneline -1
91bb588 Automatic commit of package [openshift-ansible] release [3.11.0-0.10.0].
and
# git log --oneline -1
3c66516 Automatic commit of package [openshift-ansible] release [3.11.0-0.11.0].

on
# yum list installed | grep openshift
atomic-openshift.x86_64         3.11.0-0.10.0.git.0.766dbc4.el7

# oc get pod
NAME                      READY     STATUS             RESTARTS   AGE
glusterfs-storage-6xv5x   1/1       Running            0          2m
glusterfs-storage-7mmdt   1/1       Running            0          2m
glusterfs-storage-rjhmx   1/1       Running            0          2m
heketi-storage-1-deploy   1/1       Running            0          34s
heketi-storage-1-xd47t    0/1       CrashLoopBackOff   2          31s
root@ip-172-31-33-197: ~ # oc logs heketi-storage-1-xd47t
Heketi 7.0.0
[heketi] INFO 2018/08/02 17:59:12 Loaded kubernetes executor
[heketi] ERROR 2018/08/02 17:59:12 /src/github.com/heketi/heketi/apps/glusterfs/dbcommon.go:173: Unable to open database: permission denied
[heketi] ERROR 2018/08/02 17:59:12 /src/github.com/heketi/heketi/apps/glusterfs/app.go:131: Unable to open database: permission denied. Retrying using read only mode
[heketi] ERROR 2018/08/02 17:59:12 /src/github.com/heketi/heketi/apps/glusterfs/dbcommon.go:166: Unable to open database in read only mode: permission denied
[heketi] ERROR 2018/08/02 17:59:12 /src/github.com/heketi/heketi/apps/glusterfs/app.go:136: Unable to open database: permission denied
ERROR: Unable to start application

Then I reallize that about selinux
https://bugzilla.redhat.com/show_bug.cgi?id=1600498

Tried with 'setenfor 0' on the node where heketi is running. All pods are fine.
# oc get pod
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-9bs6t   1/1       Running   0          13m
glusterfs-storage-g5vtb                       1/1       Running   0          19m
glusterfs-storage-k8q7n                       1/1       Running   0          19m
glusterfs-storage-w4msh                       1/1       Running   0          19m
heketi-storage-2-qk7nz                        1/1       Running   0          13m

See this in the audit log
type=VIRT_CONTROL msg=audit(1533232733.350:10158): pid=3228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='reason=api op=start vm=sha256:4da3aca2eda6339a92a0c5968511b9c953eede0cb4ee736846aac44011cfe61e vm-pid=0 auid=4294967295 user=? exe=/usr/sbin/heketi-start.sh hostname=574b07578ac9 ctr_id_short=574b07578ac9  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1533232733.560:10159): avc:  denied  { map } for  pid=111154 comm="heketi" path="/var/lib/heketi/heketi.db" dev="fuse" ino=12619380653579303580 scontext=system_u:system_r:container_t:s0:c548,c613 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1533232733.560:10159): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=10000 a2=1 a3=1 items=0 ppid=111107 pid=111154 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="heketi" exe="/usr/bin/heketi" subj=system_u:system_r:container_t:s0:c548,c613 key=(null)
type=PROCTITLE msg=audit(1533232733.560:10159): proctitle=2F7573722F62696E2F68656B657469002D2D636F6E6669673D2F6574632F68656B6574692F68656B6574692E6A736F6E
type=AVC msg=audit(1533232733.563:10160): avc:  denied  { map } for  pid=111154 comm="heketi" path="/var/lib/heketi/heketi.db" dev="fuse" ino=12619380653579303580 scontext=system_u:system_r:container_t:s0:c548,c613 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1533232733.563:10160): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=10000 a2=1 a3=1 items=0 ppid=111107 pid=111154 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="heketi" exe="/usr/bin/heketi" subj=system_u:system_r:container_t:s0:c548,c613 key=(null)
type=PROCTITLE msg=audit(1533232733.563:10160): proctitle=2F7573722F62696E2F68656B657469002D2D636F6E6669673D2F6574632F68656B6574692F68656B6574692E6A736F6E
type=VIRT_CONTROL msg=audit(1533232734.815:10161): pid=3228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='vm-pid=? auid=4294967295 exe=? hostname=? reason=api vm=? ctr_id_short=? op=create user=?  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1533232734.849:10162): pid=3228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='vm-pid=0 user=? auid=4294967295 exe=/usr/sbin/heketi-start.sh hostname=235663bc099f ctr_id_short=235663bc099f op=start vm=sha256:4da3aca2eda6339a92a0c5968511b9c953eede0cb4ee736846aac44011cfe61e reason=api  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1533232735.054:10163): avc:  denied  { map } for  pid=111256 comm="heketi" path="/var/lib/heketi/heketi.db" dev="fuse" ino=12619380653579303580 scontext=system_u:system_r:container_t:s0:c548,c613 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1533232735.054:10163): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=10000 a2=1 a3=1 items=0 ppid=111228 pid=111256 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="heketi" exe="/usr/bin/heketi" subj=system_u:system_r:container_t:s0:c548,c613 key=(null)
type=PROCTITLE msg=audit(1533232735.054:10163): proctitle=2F7573722F62696E2F68656B657469002D2D636F6E6669673D2F6574632F68656B6574692F68656B6574692E6A736F6E
type=AVC msg=audit(1533232735.056:10164): avc:  denied  { map } for  pid=111256 comm="heketi" path="/var/lib/heketi/heketi.db" dev="fuse" ino=12619380653579303580 scontext=system_u:system_r:container_t:s0:c548,c613 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1533232735.056:10164): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=10000 a2=1 a3=1 items=0 ppid=111228 pid=111256 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="heketi" exe="/usr/bin/heketi" subj=system_u:system_r:container_t:s0:c548,c613 key=(null)
type=PROCTITLE msg=audit(1533232735.056:10164): proctitle=2F7573722F62696E2F68656B657469002D2D636F6E6669673D2F6574632F68656B6574692F68656B6574692E6A736F6E
type=VIRT_CONTROL msg=audit(1533232735.536:10165): pid=3228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='vm=sha256:4da3aca2eda6339a92a0c5968511b9c953eede0cb4ee736846aac44011cfe61e vm-pid=0 user=? exe=/usr/sbin/heketi-start.sh hostname=heketi-storage-1-xd47t ctr_id_short=574b07578ac9 reason=api op=remove auid=4294967295  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1533232737.627:10166): pid=3228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='op=exec vm=sha256:11d588a78b45dfdbac0c11515a87a23a45dfe0d137df8b8dec31037a8b49c10f auid=4294967295 exe=/usr/local/bin/update-params.sh reason=api user=? hostname=ip-172-31-48-190.us-west-2.compute.internal ctr_id_short=3e875b836a0f vm-pid=108465  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1533232737.628:10167): pid=3228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='reason=api vm-pid=? user=? hostname=? op=start vm=? auid=4294967295 exe=? ctr_id_short=?  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
Comment 3 Milos Malik 2018-08-02 20:02:18 UTC 
I believe that new policy rules need to be added to container-selinux package (different component than selinux-policy), because all above-mentioned SELinux denials contain "scontext=system_u:system_r:container_t:s0:".
Comment 4 Hongkai Liu 2018-08-02 21:24:45 UTC 
@Milos,

do you mean I need to change the component of this bz to "container-selinux"?
Comment 5 Milos Malik 2018-08-03 06:39:29 UTC 
Yes, please.
Comment 6 Daniel Walsh 2019-01-10 17:01:06 UTC 
This is fixed in the latest container-selinux.
Comment 8 Hongkai Liu 2019-03-27 21:24:59 UTC 
It works with the following set up:

# yum list installed | grep openshift
atomic-openshift.x86_64        3.11.96-1.git.0.6ef542e.el7
# uname -a
Linux ip-172-31-8-196.us-west-2.compute.internal 3.10.0-1022.el7.x86_64 #1 SMP Wed Mar 13 13:05:41 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
# yum list installed | grep -i selinux
container-selinux.noarch       2:2.77-1.el7_6     @oso-rhui-rhel-server-extras  
libselinux.x86_64              2.5-14.1.el7       @oso-rhui-rhel-server-releases
libselinux-python.x86_64       2.5-14.1.el7       @oso-rhui-rhel-server-releases
libselinux-utils.x86_64        2.5-14.1.el7       @oso-rhui-rhel-server-releases
selinux-policy.noarch          3.13.1-241.el7     @rhel-7-next                  
selinux-policy-targeted.noarch 3.13.1-241.el7     @rhel-7-next

### openshift-ansible
# git log --oneline -1
fac2b1085 (HEAD -> release-3.11, origin/release-3.11) Merge pull request #11390 from jcpowermac/bz1689230

# oc get pod -o yaml | grep image: | uniq
      image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-gluster-block-prov-rhel7:3.11.1-3.manual.push
      image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-server-rhel7:3.11.1-12.manual.push
      image: registry.reg-aws.openshift.com:443/rhgs3/rhgs-volmanager-rhel7:3.11.1-10.manual.push

# oc get pod -n glusterfs 
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-jjc2d   1/1       Running   0          15m
glusterfs-storage-cc5qk                       1/1       Running   0          18m
glusterfs-storage-pvfth                       1/1       Running   0          18m
glusterfs-storage-xq57m                       1/1       Running   0          18m
heketi-storage-1-s7qb5                        1/1       Running   0          15m
# oc get pvc -n abc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
a1        Bound     pvc-5e9c9403-50d6-11e9-8c63-027f3a0dff30   3Gi        RWO            glusterfs-storage         1m
a2        Bound     pvc-72b796b4-50d6-11e9-8c63-027f3a0dff30   3Gi        RWO            glusterfs-storage-block   42s
Comment 12 errata-xmlrpc 2019-04-23 17:40:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0861