1414764 – recent policy upgrade breaks gitlab-ci-runner (3rd party software)

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1414764 - recent policy upgrade breaks gitlab-ci-runner (3rd party software)

Summary: recent policy upgrade breaks gitlab-ci-runner (3rd party software)

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	selinux-policy
Sub Component:
Version:	7.3
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Lukas Vrabec
QA Contact:	Milos Malik
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-01-19 11:48 UTC by Nikos Mavrogiannopoulos
Modified:	2017-01-19 16:44 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-01-19 16:44:01 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Nikos Mavrogiannopoulos 2017-01-19 11:48:39 UTC

Description of problem:
Gitlab-ci-runner [0] is 3rd party software which can run on RHEL (among others), in order to provide a continuous integration service (runners) for jobs defined in gitlab (see for example [1]). The most popular method of operation of this service is via docker, meaning that the service will spawn a new docker container for each CI run. After upgrading to the latest selinux-policy this functionality fails to operate.

[0]. https://docs.gitlab.com/runner/install/
[1]. https://gitlab.com/gnutls/gnutls/blob/master/.gitlab-ci.yml

Version-Release number of selected component (if applicable):
# rpm -q selinux-policy
selinux-policy-3.13.1-102.el7_3.13.noarch


A way to reproduce via gitlab.com and a runner under your control:

1. Install gitlab-ci-runner as in https://docs.gitlab.com/runner/install/linux-repository.html
2. Create a test repository on gitlab.com containing a single file (.gitlab-ci.yml):
```
image: centos:7
Build/Centos7:
  script:
  - yum install -y epel-release
  tags:
  - mine
```

3. Then got to the repository's settings -> Runners

4. Click: Disable shared runners

5. Follow the instructions on section "How to setup a specific Runner for a new project" in order to configure the previously installed runner.

6. Do a commit on the project and visit "Pipelines"


Actual results:
Build will fail due to permission errors in the runner.

Example:
```
Running with gitlab-ci-multi-runner 1.9.4 (8ce22bd)
Using Docker executor with image registry.gitlab.com/gnutls/build-images:buildenv-f23-x86 ...
Pulling docker image registry.gitlab.com/gnutls/build-images:buildenv-f23-x86 ...
permission denied
permission denied
ERROR: Build failed (system failure): API error (500): Container command could not be invoked.
```

Expected results:
Build will succeed (it did on 7.3.0 GA)


Additional info:

Turning selinux to permissive mode addressed the problem.

Also introducing the following policy (thanks Lukas) addresses the issue:
```
(allow svirt_lxc_net_t unconfined_service_t (process (sigchld)))
(allow unconfined_service_t svirt_lxc_net_t (process (transition)))
```

Comment 3 Milos Malik 2017-01-19 13:04:13 UTC

Could you collect relevant SELinux denials and attach them here?

Comment 4 Milos Malik 2017-01-19 13:05:15 UTC

For example via:

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today

Comment 5 Nikos Mavrogiannopoulos 2017-01-19 14:30:42 UTC

type=SYSCALL msg=audit(01/19/2017 14:29:05.319:230) : arch=x86_64 syscall=wait4 success=yes exit=2946 a0=0xb82 a1=0xc8212ed294 a2=0x0 a3=0xc8217cc240 items=0 ppid=1 pid=2186 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=docker-current exe=/usr/bin/docker-current subj=system_u:system_r:unconfined_service_t:s0 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:05.319:230) : avc:  denied  { sigchld } for  pid=2186 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c483,c739 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=AVC msg=audit(01/19/2017 14:29:06.795:238) : avc:  denied  { sigchld } for  pid=2178 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c483,c739 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:04.046:221) : arch=x86_64 syscall=wait4 success=yes exit=2874 a0=0xb3a a1=0xc820e51294 a2=0x0 a3=0xc8217355f0 items=0 ppid=1 pid=2178 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=docker-current exe=/usr/bin/docker-current subj=system_u:system_r:unconfined_service_t:s0 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:04.046:221) : avc:  denied  { sigchld } for  pid=2178 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c193,c928 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:02.921:239) : arch=x86_64 syscall=wait4 success=yes exit=2782 a0=0xade a1=0xc820e5d294 a2=0x0 a3=0xc821390480 items=0 ppid=1 pid=2505 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=docker-current exe=/usr/bin/docker-current subj=system_u:system_r:unconfined_service_t:s0 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:02.921:239) : avc:  denied  { sigchld } for  pid=2505 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c724,c879 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:03.274:241) : arch=x86_64 syscall=wait4 success=yes exit=2808 a0=0xaf8 a1=0xc820e17294 a2=0x0 a3=0xc8217cda70 items=0 ppid=1 pid=2654 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=docker-current exe=/usr/bin/docker-current subj=system_u:system_r:unconfined_service_t:s0 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:03.274:241) : avc:  denied  { sigchld } for  pid=2654 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c493,c682 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:06.241:252) : arch=x86_64 syscall=wait4 success=yes exit=2985 a0=0xba9 a1=0xc820e51294 a2=0x0 a3=0xc8213914d0 items=0 ppid=1 pid=2900 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=docker-current exe=/usr/bin/docker-current subj=system_u:system_r:unconfined_service_t:s0 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:06.241:252) : avc:  denied  { sigchld } for  pid=2900 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c193,c928 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=AVC msg=audit(01/19/2017 14:29:19.621:284) : avc:  denied  { sigchld } for  pid=2709 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c382,c613 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:19.341:282) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc82066e730 a1=0xc82064c1a0 a2=0xc82061fa40 a3=0x0 items=0 ppid=2178 pid=3386 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sh exe=/usr/bin/bash subj=system_u:system_r:svirt_lxc_net_t:s0:c382,c613 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:19.341:282) : avc:  denied  { transition } for  pid=3386 comm=exe path=/usr/bin/bash dev="dm-2" ino=37749047 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c382,c613 tclass=process 
----
type=AVC msg=audit(01/19/2017 14:29:21.734:297) : avc:  denied  { sigchld } for  pid=2873 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c248,c907 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:20.911:292) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc8200dc4a0 a1=0xc8205ae0b0 a2=0xc820095680 a3=0x0 items=0 ppid=2178 pid=3425 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=gitlab-runner-b exe=/bin/bash subj=system_u:system_r:svirt_lxc_net_t:s0:c193,c928 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:20.911:292) : avc:  denied  { transition } for  pid=3425 comm=exe path=/usr/bin/gitlab-runner-build dev="dm-4" ino=62914869 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c193,c928 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:21.602:293) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc8207161a0 a1=0xc82070c9e0 a2=0xc8206d43c0 a3=0x0 items=0 ppid=2178 pid=3462 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sh exe=/usr/bin/bash subj=system_u:system_r:svirt_lxc_net_t:s0:c248,c907 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:21.602:293) : avc:  denied  { transition } for  pid=3462 comm=exe path=/usr/bin/bash dev="dm-1" ino=37749026 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c248,c907 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:22.430:298) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc82058fdc0 a1=0xc8206a5e80 a2=0xc8201128c0 a3=0x0 items=0 ppid=2178 pid=3517 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=gitlab-runner-b exe=/bin/bash subj=system_u:system_r:svirt_lxc_net_t:s0:c483,c739 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:22.430:298) : avc:  denied  { transition } for  pid=3531 comm=exe path=/usr/bin/gitlab-runner-build dev="dm-3" ino=62914869 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c483,c739 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:26.563:306) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc820577d70 a1=0xc820660580 a2=0xc820092000 a3=0x0 items=0 ppid=2178 pid=3608 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sh exe=/usr/bin/bash subj=system_u:system_r:svirt_lxc_net_t:s0:c218,c557 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:26.563:306) : avc:  denied  { transition } for  pid=3608 comm=exe path=/usr/bin/bash dev="dm-4" ino=37749026 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c218,c557 tclass=process 
----
type=AVC msg=audit(01/19/2017 14:29:26.614:307) : avc:  denied  { sigchld } for  pid=2186 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c218,c557 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=AVC msg=audit(01/19/2017 14:29:29.970:311) : avc:  denied  { sigchld } for  pid=2896 comm=docker-current scontext=system_u:system_r:svirt_lxc_net_t:s0:c644,c729 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process 
----
type=SYSCALL msg=audit(01/19/2017 14:29:29.031:310) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc8206d9d50 a1=0xc8206fe700 a2=0xc820093cc0 a3=0x0 items=0 ppid=2178 pid=3686 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sh exe=/usr/bin/bash subj=system_u:system_r:svirt_lxc_net_t:s0:c644,c729 key=(null) 
type=AVC msg=audit(01/19/2017 14:29:29.031:310) : avc:  denied  { transition } for  pid=3686 comm=exe path=/usr/bin/bash dev="dm-3" ino=37749026 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c644,c729 tclass=process

Comment 6 Nikos Mavrogiannopoulos 2017-01-19 14:39:30 UTC

This issue is fully reproducible on the two systems that I had been running. However, installing the initial image and directly updating to the RHEL version, does _not_ exhibit this behavior. The only difference is that the running systems were being updated gradually over time via yum.cron (with default setting), or may have been compromised (and the one who did messed the selinux policy?).

So for me A work-around exists and is replacing my existing workers with fresh ones. Unless the issue is of interest to you I'll close it.

Note You need to log in before you can comment on or make changes to this bug.