Bug 1434826

Summary: selinux policy for amphora images
Product: Red Hat OpenStack Reporter: Nir Magnezi <nmagnezi>
Component: openstack-selinuxAssignee: Lon Hohberger <lhh>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: amuller, bcafarel, cgoncalves, jlibosva, jschluet, lhh, mburns, mgrepl, nyechiel, oblaut, srevivo, tfreger, tvignaud
Target Milestone: rcKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-selinux-0.8.14-10.el7ost.noarch Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:29:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1414022, 1433523    
Attachments:
Description Flags
amp_selinux.log
none
/var/log/audit/audit.log
none
Updated audit.log
none
amphora1_alerts.txt
none
amphora2_alerts.txt
none
amphora1.log
none
amphora2.log
none
audit.log with selinux enforcing
none
openstack-selinux-0.8.14-7 audit logs
none
master amphora log with permissive openstack-selinux-0.8.14-8
none
slave amphora log with permissive openstack-selinux-0.8.14-8
none
Octavia worker.log with enforcing selinux amphora
none
amp_master_openstack-selinux-0.8.14-8-audit
none
amp_backup_openstack-selinux-0.8.14-8-audit none

Description Nir Magnezi 2017-03-22 12:42:11 UTC
Created attachment 1265395 [details]
amp_selinux.log

Description of problem:
=======================
Lack of SELinux policies prevents normal operation of a RHEL/CentOS/Fedora based amphora image.

For example: haproxy fails to read configuration.
This issue is caused by SELinux, which is Enforcing by default (as it should).

The error (for the above mentioned example):
amphora-a61d0e97-d68f-4246-9f84-b2aae7ed7560 haproxy: [ALERT] 334/114506 (2394) : Could not open configuration file /var/lib/octavia/2c699b77-3983-4d40-a425-cbad188f2067/haproxy.cfg : Permission denied

More SELinux Issues:

SELinux is preventing /usr/sbin/haproxy from read access on the file haproxy.cfg.
SELinux is preventing /usr/sbin/haproxy from getattr access on the file /var/lib/octavia/d842c875-6fea-49cd-ac49-9aa82d12237c/haproxy.cfg.
SELinux is preventing /usr/sbin/ip from mounton access on the directory /run/netns.
SELinux is preventing /usr/sbin/ip from mounton access on the directory /.
SELinux is preventing /usr/sbin/ip from mounton access on the directory /sys.
SELinux is preventing /usr/sbin/ip from mounton access on the directory /etc/sysconfig.
SELinux is preventing /usr/sbin/sysctl from getattr access on the filesystem /sys.
SELinux is preventing /usr/sbin/sysctl from write access on the file sysrq.
SELinux is preventing /usr/sbin/sysctl from getattr access on the file /proc/sys/fs/protected_hardlinks.
SELinux is preventing /usr/sbin/sysctl from write access on the file protected_hardlinks.
SELinux is preventing /usr/sbin/sysctl from getattr access on the file /proc/sys/fs/file-max.
SELinux is preventing /usr/sbin/sysctl from write access on the file file-max.
SELinux is preventing /usr/sbin/haproxy-systemd-wrapper from execute access on the file haproxy-systemd-wrapper.
SELinux is preventing /usr/sbin/haproxy from using the dac_override capability.
SELinux is preventing /usr/sbin/haproxy from using the fowner capability.
SELinux is preventing /usr/sbin/haproxy from create access on the sock_file d842c875-6fea-49cd-ac49-9aa82d12237c.sock.3477.tmp.
SELinux is preventing /usr/sbin/haproxy from setattr access on the sock_file 2c699b77-3983-4d40-a425-cbad188f2067.sock.2454.tmp.
SELinux is preventing /usr/sbin/haproxy from remove_name access on the directory 2c699b77-3983-4d40-a425-cbad188f2067.sock.2454.tmp.
SELinux is preventing /usr/sbin/haproxy from name_bind access on the tcp_socket port 80.
SELinux is preventing /usr/sbin/haproxy from listen access on the tcp_socket port None.
SELinux is preventing /usr/sbin/haproxy from write access on the directory d842c875-6fea-49cd-ac49-9aa82d12237c.
SELinux is preventing /usr/sbin/haproxy from using the setgid capability.
SELinux is preventing /usr/sbin/haproxy from using the setuid capability.
SELinux is preventing /usr/sbin/haproxy from write access on the sock_file 2c699b77-3983-4d40-a425-cbad188f2067.sock.
SELinux is preventing /usr/sbin/haproxy from link access on the sock_file d842c875-6fea-49cd-ac49-9aa82d12237c.sock.
SELinux is preventing /usr/sbin/haproxy from unlink access on the sock_file d842c875-6fea-49cd-ac49-9aa82d12237c.sock.
SELinux is preventing /usr/sbin/haproxy from add_name access on the directory d842c875-6fea-49cd-ac49-9aa82d12237c.pid.
SELinux is preventing /usr/sbin/haproxy-systemd-wrapper from read access on the file haproxy.cfg.
SELinux is preventing /usr/sbin/haproxy-systemd-wrapper from getattr access on the file /var/lib/octavia/2c699b77-3983-4d40-a425-cbad188f2067/haproxy.cfg.
SELinux is preventing /usr/sbin/haproxy from write access on the directory octavia.
SELinux is preventing /usr/sbin/haproxy from unlink access on the file d842c875-6fea-49cd-ac49-9aa82d12237c.pid.
SELinux is preventing /usr/sbin/haproxy from create access on the file 2c699b77-3983-4d40-a425-cbad188f2067.pid.
SELinux is preventing /usr/sbin/haproxy from using the kill capability.
SELinux is preventing /usr/sbin/haproxy from getattr access on the file /var/lib/octavia/2c699b77-3983-4d40-a425-cbad188f2067/haproxy.cfg.

I'll attached the full log to this bug.

The missing SELinux policies:
[root@amphora-a61d0e97-d68f-4246-9f84-b2aae7ed7560 system]# cat /var/log/audit/audit.log | audit2allow -R
require {
 type ifconfig_t;
 type haproxy_t;
 type haproxy_exec_t;
 type var_lib_t;
 type ifconfig_var_run_t;
 type sysctl_fs_t;
 type proc_security_t;
 type sysctl_kernel_t;
 type etc_t;
 class capability { setuid kill setgid fowner net_bind_service dac_override };
 class tcp_socket listen;
 class dir mounton;
 class file { execute read create execute_no_trans write getattr unlink open };
 class sock_file { rename write link setattr create unlink };
}

#============= haproxy_t ==============
allow haproxy_t var_lib_t:file { read getattr open };

#============= ifconfig_t ==============
allow ifconfig_t etc_t:dir mounton;
allow ifconfig_t haproxy_exec_t:file { read execute open execute_no_trans };
allow ifconfig_t ifconfig_var_run_t:dir mounton;
allow ifconfig_t proc_security_t:file { write getattr open };
allow ifconfig_t self:capability { setuid kill setgid fowner net_bind_service dac_override };

#!!!! This avc can be allowed using the boolean 'nis_enabled'
allow ifconfig_t self:tcp_socket listen;
allow ifconfig_t sysctl_fs_t:file { write getattr open };
allow ifconfig_t sysctl_kernel_t:file write;
allow ifconfig_t var_lib_t:file { write getattr read create unlink open };
allow ifconfig_t var_lib_t:sock_file { rename write link setattr create unlink };
corenet_tcp_bind_http_port(ifconfig_t)
dev_getattr_sysfs_fs(ifconfig_t)
files_filetrans_system_db_named_files(ifconfig_t)
files_mounton_isid(ifconfig_t)
files_mounton_rootfs(ifconfig_t)


Additional info:
================
Full log: https://launchpadlibrarian.net/295546331/amp_selinux.log

Comment 4 Lon Hohberger 2017-03-24 12:58:16 UTC
Can you attach audit.log?

Comment 5 Lon Hohberger 2017-03-24 13:11:06 UTC
The audit2allow output is not correct.  There seems to be a domain transition missing. (why is ifconfig_t context trying to do these things?)

Comment 6 Lon Hohberger 2017-03-24 13:11:36 UTC
Rather, audit2allow output is not the correct solution.

Comment 7 Lon Hohberger 2017-03-24 13:23:36 UTC
What kind of amphora image was used (virt, container, ???) ?

Comment 8 Nir Magnezi 2017-03-29 11:02:13 UTC
(In reply to Lon Hohberger from comment #7)
> What kind of amphora image was used (virt, container, ???) ?

a Nova instance currently, so I assume the answer is virt.

Comment 9 Nir Magnezi 2017-03-29 11:03:03 UTC
(In reply to Lon Hohberger from comment #4)
> Can you attach audit.log?

Sadly I don't have it, I'll try to reproduce and attach it.
Permissive state should do the trick, right?

Comment 10 Lon Hohberger 2017-03-29 14:28:00 UTC
Yeah. Also, where did you get the amphora image?

Comment 11 Nir Magnezi 2017-03-29 21:00:13 UTC
(In reply to Lon Hohberger from comment #10)
> Yeah. Also, where did you get the amphora image?

It is generated using https://github.com/openstack/octavia/blob/master/diskimage-create/diskimage-create.sh

I filed the bug using CentOS based image, but we also build a RHEL based image, which has openstack-selinux in it.

Comment 12 Nir Magnezi 2017-03-30 22:31:59 UTC
Created attachment 1267676 [details]
/var/log/audit/audit.log

(In reply to Nir Magnezi from comment #9)
> (In reply to Lon Hohberger from comment #4)
> > Can you attach audit.log?
> 
> Sadly I don't have it, I'll try to reproduce and attach it.
> Permissive state should do the trick, right?

(In reply to Lon Hohberger from comment #10)
> Yeah. Also, where did you get the amphora image?

Hi Lon,

I've attached audit.log taken from a amphora in permissive mode (CentOS)

Comment 13 Nir Magnezi 2017-03-30 22:33:03 UTC
I removed this NEEDINFO by mistake.

Comment 14 Lon Hohberger 2017-04-04 13:27:13 UTC
This needs an entire new service policy crafted.

Comment 17 Assaf Muller 2017-04-13 18:26:51 UTC
Lon and I spoke on IRC, we came to the conclusion that we cannot ship an SELinux policy for an image we don't. I pushed this bug to 12 and made it dependent on an RFE to ship an Amphora image.

Comment 20 Bernard Cafarelli 2017-12-12 14:31:08 UTC
Created attachment 1366689 [details]
Updated audit.log

Current amphora code + https://review.openstack.org/#/c/527073/ (to also run keepalived in proper context)

Most current errors are for commands run when setting up the namespace, plus non-standard paths for haproxy/keepalived files (conf, pid, ...)

Can these be added to selinux policies or do we need another setup modification here?

Comment 21 Bernard Cafarelli 2017-12-14 14:36:29 UTC
Updating status as upstream patches are in review or merged for the Octavia-specific parts

Comment 28 Nir Magnezi 2018-03-22 22:14:59 UTC
Created attachment 1411883 [details]
amphora1_alerts.txt

Comment 29 Nir Magnezi 2018-03-22 22:15:25 UTC
Created attachment 1411884 [details]
amphora2_alerts.txt

Comment 30 Nir Magnezi 2018-03-22 22:15:56 UTC
Created attachment 1411885 [details]
amphora1.log

Comment 31 Nir Magnezi 2018-03-22 22:17:04 UTC
Created attachment 1411886 [details]
amphora2.log

Comment 35 Lon Hohberger 2018-04-24 18:18:03 UTC
OK, so there's a number of things we need to do:

Keepalived_t needs to be able to transition to ifconfig_t when running /sbin/ip)

. = handled by domain transition (policy exists in RHEL 7.5)
! = needs policy 
# = needs policy (policy exists in Fedora)

 ! allow keepalived_t etc_t:dir mounton;
 ! allow keepalived_t user_tmp_t:dir mounton;
 . allow keepalived_t ifconfig_exec_t:file entrypoint;
 . allow keepalived_t sysfs_t:filesystem { mount unmount };
 # allow keepalived_t root_t:dir mounton;


It looks also like haproxy_t needs a transition to ifconfig_t:

 ! allow haproxy_t bin_t:file entrypoint;
 ! allow haproxy_t etc_t:dir mounton;
 . allow haproxy_t ifconfig_exec_t:file { entrypoint read };
 . allow haproxy_t sysfs_t:filesystem { mount unmount };
 # allow haproxy_t root_t:dir mounton;
 ! allow haproxy_t user_tmp_t:dir mounton;

These all seem to need a file context writable by ifconfig_t (since haproxy and keepalived should transition to ifconfig_t when running /sbin/ip):

   allow haproxy_t var_lib_t:file { getattr open read };
   allow haproxy_t var_lib_t:sock_file { create link rename setattr unlink write };
   allow keepalived_t var_lib_t:dir { add_name write };
   allow keepalived_t var_lib_t:file { create execute execute_no_trans ioctl write };
   allow keepalived_t var_lib_t:sock_file write;

Comment 36 Lon Hohberger 2018-04-24 18:20:00 UTC
oops bin_t should have been '.'

Comment 37 Lon Hohberger 2018-04-24 18:50:41 UTC
So, with that, do we know what's in check_script.sh?

Comment 38 Lon Hohberger 2018-04-24 19:39:17 UTC
Also - where haproxy-vrrp-ch[something] is creating sockets?

Progress is here:


https://github.com/redhat-openstack/openstack-selinux/tree/amphora

Comment 46 Bernard Cafarelli 2018-05-04 09:58:08 UTC
Testing with Enforcing SELinux, the amphora-netns service fails to start:
type=AVC msg=audit(1525382109.218:71): avc:  denied  { sys_admin } for  pid=1661 comm="ip" capability=21  scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:system_r:haproxy_t:s0 tclass=capability

# ls -Z /usr/lib/systemd/system/amphora-netns.service
-rw-r--r--. root root system_u:object_r:systemd_unit_file_t:s0 /usr/lib/systemd/system/amphora-netns.service

# cat /usr/lib/systemd/system/amphora-netns.service
[Unit]
Description=Configure amphora-haproxy network namespace
StopWhenUnneeded=true

[Service]
Type=oneshot
RemainAfterExit=yes

# Re-add the namespace
ExecStart=-/sbin/ip netns add amphora-haproxy
# Load the system sysctl into the new namespace
ExecStart=-/sbin/ip netns exec amphora-haproxy sysctl --system
# We need the plugged_interfaces file sorted to join the host interfaces
ExecStart=-/bin/sh -c '/usr/bin/sort -k 1 /var/lib/octavia/plugged_interfaces > /var/lib/octavia/plugged_interfaces.sorted'
# Assign the interfaces into the namespace with the appropriate name
ExecStart=-/bin/sh -c '/sbin/ip link | awk \'{getline n; print $0,n}\' | awk \'{sub(":","",$2)} { for(i=1;i<=NF;i++) if ($i == "link/ether") {print $(i+1) " " $2} }\' | sort -k 1 | join -j 1 - /var/lib/octavia/plugged_interfaces.sorted | awk \'{system("ip link set "$2" netns amphora-haproxy name "$3"")}\''
# Bring up all of the namespace interfaces
ExecStart=-/bin/awk '{system("/sbin/ip netns exec amphora-haproxy ifup " $2)}' /var/lib/octavia/plugged_interfaces

Restarting the unit from root has the same result (strange that it has haproxy_t? the actual haproxy service file is haproxy-<UUID>.service)

Comment 48 Bernard Cafarelli 2018-05-04 10:00:16 UTC
Created attachment 1431205 [details]
audit.log with selinux enforcing

Comment 51 Lon Hohberger 2018-05-10 20:36:45 UTC
The AVC reported is new:

type=AVC msg=audit(1525382110.500:85): avc:  denied  { sys_admin } for  pid=1855 comm="ip" capability=21  scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:system_r:haproxy_t:s0 tclass=capability

However, rather than fix this AVC and try again, could we get a test run in permissive and attach _that_ audit.log ?

Comment 52 Lon Hohberger 2018-05-10 20:37:40 UTC
It seems something changed; the initial policy Nir proposed didn't have sys_admin in the list of capabilities, so we'll have to add it.

Comment 56 Nir Magnezi 2018-05-15 12:26:37 UTC
I did some initial testing (QE will run more in-depth tests)

I used the amphora image from octavia-amphora-image-13.0-20180510.1.el7ost.noarch to create a loadbalancer with a listener.

audit logs look okay to me: http://paste.openstack.org/show/720992/

For some reason that image is still being created with SELInux permissive:
[root@amphora-8df4802e-e9f2-4dbe-ba47-facc127b3b8c audit]# getenforce 
Permissive

Why is that?

Comment 57 Nir Magnezi 2018-05-15 12:37:42 UTC
actually, that image contains openstack-selinux-0.8.14-5.el7ost , not openstack-selinux-0.8.14-6.el7ost.

I'm going to use an up to date one and report shortly.

Comment 60 Lon Hohberger 2018-05-15 15:41:18 UTC
Thanks - I've grabbed the audit.logs.

There's something odd going on which I am investigating.  The AVC denials should be coming as 'ifconfig_t' instead of 'haproxy_t' / 'keepalived_t'.

Both of these security contexts should be transitioned to ifconfig_t when executing the 'ip' command, so I am trying to figure out why.

However, in the short term, I can just allow the rules here since.

Comment 61 Lon Hohberger 2018-05-15 15:41:53 UTC
... since they are not materially different from what we allowed for ifconfig_t after the transition.

Comment 65 Nir Magnezi 2018-05-16 13:52:24 UTC
Created attachment 1437383 [details]
openstack-selinux-0.8.14-7 audit logs

tested openstack-selinux-0.8.14-7.el7ost.noarch

keepalived cannot start when SELiunux is enforcing.


type=AVC msg=audit(1526475726.336:16): avc:  denied  { open } for  pid=444 comm="dbus-daemon" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475726.378:17): avc:  denied  { getattr } for  pid=444 comm="dbus-daemon" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475757.992:25): avc:  denied  { read } for  pid=530 comm="systemd-hostnam" name="config" dev="vda1" ino=2306 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475757.992:25): avc:  denied  { open } for  pid=530 comm="systemd-hostnam" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475758.007:26): avc:  denied  { getattr } for  pid=530 comm="systemd-hostnam" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475843.731:39): avc:  denied  { read } for  pid=1126 comm="systemd-hostnam" name="config" dev="vda1" ino=2306 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475843.731:39): avc:  denied  { open } for  pid=1126 comm="systemd-hostnam" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526475843.731:40): avc:  denied  { getattr } for  pid=1126 comm="systemd-hostnam" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526476187.762:95): avc:  denied  { read } for  pid=1691 comm="sshd" name="config" dev="vda1" ino=2306 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526476187.762:95): avc:  denied  { open } for  pid=1691 comm="sshd" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1526476187.771:96): avc:  denied  { getattr } for  pid=1691 comm="sshd" path="/etc/selinux/config" dev="vda1" ino=2306 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file


attaching the full audit logs for permissive mode.

Comment 66 Lon Hohberger 2018-05-17 11:47:43 UTC
We discussed offline:

* These files are not correctly labeled. /etc/selinux/config should not be 'unlabeled_t'.

* When building images, ensure you relabel after customizing:
  virt-customize --selinux-relabel -a <whatever.qcow2>

Comment 68 Bernard Cafarelli 2018-05-18 14:03:47 UTC
Created attachment 1438585 [details]
master amphora log with permissive openstack-selinux-0.8.14-8

Comment 69 Bernard Cafarelli 2018-05-18 14:04:23 UTC
Created attachment 1438586 [details]
slave amphora log with permissive openstack-selinux-0.8.14-8

Comment 70 Bernard Cafarelli 2018-05-18 14:07:49 UTC
Created attachment 1438587 [details]
Octavia worker.log with enforcing selinux amphora

Comment 71 Bernard Cafarelli 2018-05-18 14:10:20 UTC
With openstack-selinux-0.8.14-8, amphora in enforcing mode is quickly deleted by the Octavia worker shortly after boot, probably because of:
Unable to plug VIP for loadbalancer id eaebe7bf-d5d9-4ad2-a6d5-19f8382f2c50

The permissive log is "semodule -DB" run, haproxy looks fine but for keepalived:
type=AVC msg=audit(1526651291.643:86): avc:  denied  { rlimitinh } for  pid=1578 comm="modprobe" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:system_r:insmod_t:s0 tclass=process
type=AVC msg=audit(1526651291.643:86): avc:  denied  { siginh } for  pid=1578 comm="modprobe" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:system_r:insmod_t:s0 tclass=process
type=AVC msg=audit(1526651291.643:86): avc:  denied  { noatsecure } for  pid=1578 comm="modprobe" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:system_r:insmod_t:s0 tclass=process

Attached audit.log from both nodes (quite similar) and octavia worker.log around that time

Comment 72 Nir Magnezi 2018-05-21 11:13:55 UTC
I've tested openstack-selinux-0.8.14-8 in permissive mode.

The audit2allow results (full logs to be attached in follow up comments):

MASTER Amphora:
===============


require {
	type NetworkManager_t;
	type dhcpc_t;
	type chkpwd_t;
	type insmod_t;
	type sshd_t;
	type postfix_pickup_t;
	type iscsid_t;
	type systemd_tmpfiles_t;
	type setfiles_t;
	type ifconfig_t;
	type sshd_keygen_t;
	type postfix_qmgr_t;
	type initrc_t;
	type keepalived_t;
	type ssh_keygen_t;
	type sendmail_t;
	type postfix_master_t;
	class process { noatsecure rlimitinh siginh };
	class capability net_admin;
}

#============= NetworkManager_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow NetworkManager_t dhcpc_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow NetworkManager_t initrc_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow NetworkManager_t iscsid_t:process { noatsecure rlimitinh siginh };

#============= ifconfig_t ==============
kernel_read_fs_sysctls(ifconfig_t)

#============= keepalived_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow keepalived_t insmod_t:process { noatsecure rlimitinh siginh };

#============= postfix_master_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow postfix_master_t postfix_pickup_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow postfix_master_t postfix_qmgr_t:process { noatsecure rlimitinh siginh };

#============= sendmail_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow sendmail_t postfix_master_t:process { noatsecure rlimitinh siginh };

#============= sshd_keygen_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow sshd_keygen_t setfiles_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow sshd_keygen_t ssh_keygen_t:process { noatsecure rlimitinh siginh };

#============= sshd_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow sshd_t chkpwd_t:process { noatsecure rlimitinh siginh };

#============= systemd_tmpfiles_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow systemd_tmpfiles_t self:capability net_admin;



BACKUP Amphora:
===============

require {
	type sshd_keygen_t;
	type sshd_t;
	type keepalived_t;
	type systemd_tmpfiles_t;
	type postfix_master_t;
	type iscsid_t;
	type dhcpc_t;
	type setfiles_t;
	type ssh_keygen_t;
	type insmod_t;
	type chkpwd_t;
	type initrc_t;
	type sendmail_t;
	type NetworkManager_t;
	type postfix_pickup_t;
	type ifconfig_t;
	type postfix_qmgr_t;
	class process { noatsecure rlimitinh siginh };
	class capability net_admin;
}

#============= NetworkManager_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow NetworkManager_t dhcpc_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow NetworkManager_t initrc_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow NetworkManager_t iscsid_t:process { noatsecure rlimitinh siginh };

#============= ifconfig_t ==============
kernel_read_fs_sysctls(ifconfig_t)

#============= keepalived_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow keepalived_t insmod_t:process { noatsecure rlimitinh siginh };

#============= postfix_master_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow postfix_master_t postfix_pickup_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow postfix_master_t postfix_qmgr_t:process { noatsecure rlimitinh siginh };

#============= sendmail_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow sendmail_t postfix_master_t:process { noatsecure rlimitinh siginh };

#============= sshd_keygen_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow sshd_keygen_t setfiles_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow sshd_keygen_t ssh_keygen_t:process { noatsecure rlimitinh siginh };

#============= sshd_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow sshd_t chkpwd_t:process { noatsecure rlimitinh siginh };

#============= systemd_tmpfiles_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow systemd_tmpfiles_t self:capability net_admin;



Note that there are slight differences between the two.

Comment 74 Nir Magnezi 2018-05-21 11:34:12 UTC
Created attachment 1439568 [details]
amp_master_openstack-selinux-0.8.14-8-audit

Comment 75 Nir Magnezi 2018-05-21 11:34:37 UTC
Created attachment 1439569 [details]
amp_backup_openstack-selinux-0.8.14-8-audit

Comment 76 Nir Magnezi 2018-05-21 14:12:15 UTC
Permissive mode audit log results for openstack-selinux-0.8.14-9.el7ost.noarch

As for the Enforcing mode image, Octavia killed the VMs due to an error. probably it was not able to start keepalived again.

http://paste.openstack.org/show/721493/

Comment 77 Lon Hohberger 2018-05-21 14:34:02 UTC
There's something wrong, every AVC in that log is allowed in 0.8.14-9

It's like the image didn't take the modules or something

Comment 78 Nir Magnezi 2018-05-22 11:56:21 UTC
(In reply to Lon Hohberger from comment #77)
> There's something wrong, every AVC in that log is allowed in 0.8.14-9
> 
> It's like the image didn't take the modules or something

Turns out you were right.
somehow openstack-selinux was not deployed correctly, so I created the image again.

Permissive mode produced no denied errors at all, which made me test Enforcing mode.

Octavia noticed that something is wrong with those Amphora instances and killed them really fast, so I had to pull some tricks to keep those instances alive for debugging.

What I basically did to manually start octavia-keepalived systemd service and see what happens:

[root@amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 ~]# systemctl start octavia-keepalived
Job for octavia-keepalived.service failed because the control process exited with error code. See "systemctl status octavia-keepalived.service" and "journalctl -xe" for details.
[root@amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 ~]# systemctl status octavia-keepalived.service
● octavia-keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/usr/lib/systemd/system/octavia-keepalived.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2018-05-22 07:37:14 EDT; 10s ago
  Process: 1752 ExecStart=/sbin/ip netns exec amphora-haproxy /usr/sbin/keepalived -D -d -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid (code=exited, status=1/FAILURE)

May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243.novalocal systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243.novalocal ip[1752]: setting the network namespace "amphora-haproxy" failed: Operation not permitted
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243.novalocal systemd[1]: octavia-keepalived.service: control process exited, code=exited status=1
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243.novalocal systemd[1]: Failed to start Keepalive Daemon (LVS and VRRP).
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243.novalocal systemd[1]: Unit octavia-keepalived.service entered failed state.
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243.novalocal systemd[1]: octavia-keepalived.service failed.

Logs:



/var/log/messages

May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 systemd: Starting Keepalive Daemon (LVS and VRRP)...
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 ip: setting the network namespace "amphora-haproxy" failed: Operation not permitted
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 amphora-agent: 2018-05-22 07:37:14.855 1195 ERROR octavia.amphorae.backends.health_daemon.health_daemon [-] Missing keepalived PID file /var/lib/octavia/vrrp/octavia-keepalived.pid, skipping health heartbeat.: IOError: [Errno 2] No such file or directory: '/var/lib/octavia/vrrp/octavia-keepalived.pid'
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 systemd: octavia-keepalived.service: control process exited, code=exited status=1
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 systemd: Failed to start Keepalive Daemon (LVS and VRRP).
May 22 07:37:14 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 systemd: Unit octavia-keepalived.service entered failed state.
May 22 07:37:15 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 systemd: octavia-keepalived.service failed.
May 22 07:37:24 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 amphora-agent: 2018-05-22 07:37:24.881 1195 ERROR octavia.amphorae.backends.health_daemon.health_daemon [-] Missing keepalived PID file /var/lib/octavia/vrrp/octavia-keepalived.pid, skipping health heartbeat.: IOError: [Errno 2] No such file or directory: '/var/lib/octavia/vrrp/octavia-keepalived.pid'
May 22 07:37:34 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 amphora-agent: 2018-05-22 07:37:34.898 1195 ERROR octavia.amphorae.backends.health_daemon.health_daemon [-] Missing keepalived PID file /var/lib/octavia/vrrp/octavia-keepalived.pid, skipping health heartbeat.: IOError: [Errno 2] No such file or directory: '/var/lib/octavia/vrrp/octavia-keepalived.pid'
May 22 07:37:44 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 amphora-agent: 2018-05-22 07:37:44.916 1195 ERROR octavia.amphorae.backends.health_daemon.health_daemon [-] Missing keepalived PID file /var/lib/octavia/vrrp/octavia-keepalived.pid, skipping health heartbeat.: IOError: [Errno 2] No such file or directory: '/var/lib/octavia/vrrp/octavia-keepalived.pid'
May 22 07:37:54 amphora-14b88f21-13f6-4aea-a8b5-839bb8787243 amphora-agent: 2018-05-22 07:37:54.935 1195 ERROR octavia.amphorae.backends.health_daemon.health_daemon [-] Missing keepalived PID file /var/lib/octavia/vrrp/octavia-keepalived.pid, skipping health heartbeat.: IOError: [Errno 2] No such file or directory: '/var/lib/octavia/vrrp/octavia-keepalived.pid'

/var/log/audit/audit.log

type=AVC msg=audit(1526989034.757:148): avc:  denied  { sys_admin } for  pid=1752 comm="ip" capability=21  scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:system_r:keepalived_t:s0 tclass=capability
type=SYSCALL msg=audit(1526989034.757:148): arch=c000003e syscall=308 success=no exit=-1 a0=5 a1=40000000 a2=7ffc7252052e a3=7ffc725200c0 items=0 ppid=1 pid=1752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:keepalived_t:s0 key=(null)
type=PROCTITLE msg=audit(1526989034.757:148): proctitle=2F7362696E2F6970006E65746E73006578656300616D70686F72612D686170726F7879002F7573722F7362696E2F6B656570616C69766564002D44002D64002D66002F7661722F6C69622F6F6374617669612F767272702F6F6374617669612D6B656570616C697665642E636F6E66002D70002F7661722F6C69622F6F637461
type=SERVICE_START msg=audit(1526989034.964:149): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=octavia-keepalived comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'


audit2allow output:

require {
    type keepalived_t;
    class capability sys_admin;
}

#============= keepalived_t ==============
allow keepalived_t self:capability sys_admin;


I will attach the full audit log as well if you need me to.

P.S.

via user root, this works when I execute it manually: /sbin/ip netns exec amphora-haproxy /usr/sbin/keepalived -D -d -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid

Comment 79 Lon Hohberger 2018-05-22 12:13:04 UTC
Yeah, executing as root, things will usually work because they will run unconfined.

Comment 80 Lon Hohberger 2018-05-22 12:13:31 UTC
With just one AVC, that's not so bad.

Comment 81 Nir Magnezi 2018-05-22 12:53:13 UTC
Confirmed with: openstack-selinux-0.8.14-10.el7ost.noarch

1. amphora agent works
2. haproxy works (created a listener trigger it) and namespace looks okay:

[root@amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a ~]# ip netns exec amphora-haproxy ip a 
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:15:39:8c brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.7/26 brd 10.0.0.63 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.0.0.16/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.0.0.16/26 brd 10.0.0.63 scope global secondary eth1:0
       valid_lft forever preferred_lft forever
    inet6 fdc1:9f57:424f:0:f816:3eff:fe15:398c/64 scope global mngtmpaddr dynamic 
       valid_lft 86376sec preferred_lft 14376sec
    inet6 fe80::f816:3eff:fe15:398c/64 scope link 
       valid_lft forever preferred_lft forever


3. keepalived works and no errors on audit.log


[root@amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a ~]# systemctl status octavia-keepalived
● octavia-keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/usr/lib/systemd/system/octavia-keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-05-22 08:45:54 EDT; 4min 19s ago
 Main PID: 1567 (keepalived)
   CGroup: /system.slice/octavia-keepalived.service
           ├─1567 /usr/sbin/keepalived -D -d -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid
           ├─1568 /usr/sbin/keepalived -D -d -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid
           └─1569 /usr/sbin/keepalived -D -d -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid

May 22 08:49:57 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
May 22 08:50:02 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
May 22 08:50:02 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: VRRP_Instance(8d6d8dba5a4e4e938c9408f116ecad92) Sending/queueing gratui....0.16
May 22 08:50:02 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
May 22 08:50:07 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
May 22 08:50:07 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: VRRP_Instance(8d6d8dba5a4e4e938c9408f116ecad92) Sending/queueing gratui....0.16
May 22 08:50:07 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
May 22 08:50:12 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
May 22 08:50:12 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: VRRP_Instance(8d6d8dba5a4e4e938c9408f116ecad92) Sending/queueing gratui....0.16
May 22 08:50:12 amphora-c7d16cbf-1b51-4748-a855-3d9b9de8511a.novalocal Keepalived_vrrp[1569]: Sending gratuitous ARP on eth1 for 10.0.0.16
Hint: Some lines were ellipsized, use -l to show in full.


Finally, Octavia declare this loadbalancer ACTIVE:

May 22 12:45:55 octavia-debug.novalocal octavia-worker[13630]: INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: 8d6d8dba-5a4e-4e93-8c94-08f116ecad92

Comment 89 errata-xmlrpc 2018-06-27 13:29:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086