This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 885529

Summary: swift replication produces SELinux AVC denials
Product: Red Hat OpenStack Reporter: Derek Higgins <derekh>
Component: openstack-selinuxAssignee: Lon Hohberger <lhh>
Status: CLOSED ERRATA QA Contact: Martina Kollarova <mkollaro>
Severity: high Docs Contact:
Priority: high    
Version: 2.0 (Folsom)CC: apevec, breu, david, d.busby, derekh, dwalsh, eparis, jhenner, jonathansteffan, lhh, markmc, mgrepl, ncredi, rbryant, silas, zaitcev
Target Milestone: snapshot3Keywords: Triaged
Target Release: 2.1   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=1000732
Whiteboard:
Fixed In Version: openstack-selinux-0.1.2-5.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 809198 Environment:
Last Closed: 2013-03-05 13:29:56 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On: 902486    
Bug Blocks: 918721, 922787    
Attachments:
Description Flags
swift policy files for RHEL6 none

Description Derek Higgins 2012-12-09 20:12:21 EST
+++ This bug was initially created as a clone of Bug #809198 +++

As part of a swift install, rsync is configured to replicate data
between nodes. I've been testing this out on RHEL 6.2 and it all works
great until I turn selinux on.

   Once I turn on selinux I start getting AVCs in the audit log, they
are being generated when rsyc (configured to be run by xinetd) trys to
access the swift data being stored on a mounted files system in
/srv/node/partitions.

   To get around this I've had to
   1. setsebool rsync_export_all_ro on # set the rsync_export_all_ro
boolean on
   2. semodule -i rsyncswift.pp # insert a policy module, I generated by
audit2allow
   3. change the lock files configured in rsyncd.conf to be created in
/var/run/ (the swift docs put them in /var/lock/)

   How would we normally handle policies required for a package in a
different channel? should they be added to the swift package or the RHEL
policies or are there other options?

   Attached are some example AVCs from audit.log along with various security contexts involved

--- Additional comment from Derek Higgins on 2012-04-02 13:45:55 EDT ---

Created attachment 574578 [details]
AVCs generated by swift replication

--- Additional comment from Derek Higgins on 2012-04-02 13:48:24 EDT ---

Created attachment 574579 [details]
module generated by audit2allow

--- Additional comment from Derek Higgins on 2012-04-02 13:53:45 EDT ---

Created attachment 574580 [details]
example rsyncd config file being used

--- Additional comment from Alan Pevec on 2012-10-18 07:18:56 EDT ---

Let's fixed that in Fedora first.
Comment 3 Pete Zaitcev 2013-01-07 11:45:49 EST
This may be relevant: Dan Walsh on issues with rsync and SElinux:
 http://danwalsh.livejournal.com/61646.html

"The biggest problem SELinux has with rsync is there is no way to
 distinguish between the client and the server from an SELinux point of view."
"When a process running as init_t or initrc_t executes /usr/bin/rsync
 (rsync_exec_t), the rsync process will transition to rsync_t and SELinux
 will treat it as the rsync daemon not as a client."
"Best would be to write policy for the init service that is currently running
 without confinement."

Of course in our case Swift daemons are already confined but rsync is not
added there for network or some other kink like that.
Comment 4 Daniel Walsh 2013-02-04 10:42:48 EST
Lets look at a couple of problems here.  First I see file_t files in your audit.log (avc) report, this means that we have files that do not have labels on them.

The simplest fix would be to remove the rsync_exec_t label from rsync, and then I think your stuff will work with SELinux in enforcing mode.
Comment 5 Pete Zaitcev 2013-02-04 11:24:20 EST
I was factually wrong about Swift being confined, or I do not have the right
package installed. On my RHEL 6.3 RHOS 2.1, we have ps -eZ:

unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 6568 ? 00:00:11 swift-object-se
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 6601 ? 00:00:13 swift-object-re
(for Object Server and Object Replicator)

system_u:system_r:inetd_t:s0-s0:c0.c1023 1544 ? 00:00:00 xinetd
Comment 6 Eric Paris 2013-02-04 14:12:57 EST
Pete, did you start these processes by hand from the command line?  Things users run from the command line are typically left unconfined whereas those same things started by the init system may be confined.

I'm not certain which is the right way to start confined daemons from the command line...  either

service blah start
 OR
run_init service blah start

It seems to change ever day....
Comment 7 Pete Zaitcev 2013-02-04 14:30:41 EST
Yes, I started Swift from CLI with swift-init. After launching with service(8),
we have:

unconfined_u:system_r:initrc_t:s0 10283 ?      00:00:00 swift-object-se

Sorry, it didn't occur to me.

This may be a bit of a problem, because using swift-init is rather well
seeded throughout the docs. We didn't even have proper scripts until
very recenly (bz#885530).
Comment 8 Daniel Walsh 2013-02-04 15:55:45 EST
SO we need a policy for swift-object-se* which will not transition to rsync.


What is this executable.
Comment 9 Daniel Walsh 2013-02-04 17:04:55 EST
Just added openstack-swift policy to Rawhide, need to back port this and make it unconfined for RHEL6.  THen it would not transition to rsync_t.
Comment 10 Pete Zaitcev 2013-02-04 17:36:53 EST
Confining Swift daemons is good, but what about the rsync daemon accessing
the /srv/node/*? It throws this:

type=AVC msg=audit(1359784683.733:23858): avc:  denied  { setattr } for  pid=23659 comm="rsync" name=".1355550388.32345.data.WODj0u" dev=vdb ino=3145762 scontext=unconfined_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file
type=AVC msg=audit(1359784683.733:23859): avc:  denied  { getattr } for  pid=23659 comm="rsync" path="/vdb/objects/19579/365/4c7babc3b566d183d6fe30e75b7dd365/.1355550388.32345.data.WODj0u" dev=vdb ino=3145762 scontext=unconfined_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file

(Audit log shows absolute-looking /vdb path for /srv/node/vdb/*, possibly
because it's a mount point.)
Comment 11 Eric Paris 2013-02-04 18:14:41 EST
file_t indicates that the files in question have no label assigned.  were they created with selinux disabled?  You can use restorecon to restore them to 'default'

restorecon -RvF /srv/node/vdb/



to make your rsync unconfined until a new policy arrives:

chcon -t bin_t `which rsync`



audit does all sorts of horrible things around path names.  it sucks, but from the kernel PoV there may be NO path between some / and the file in question...  Everything audit tells you is true, but piecing it back together from the inode number, block device, etc, can be a pain in the ass for an admin.  I agree, but there is no reliable way to do it...   so don't blame selinux for audit's, ummm lets call it, uniqueness   :)
Comment 12 Pete Zaitcev 2013-02-04 18:31:15 EST
The Swift nodes where I test for this bug were not running with SELinux
disabled, ever, but it is set to permissive while I'm figuring this out:

[root@kvm-san zaitcev]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   permissive
Mode from config file:          permissive
Policy version:                 24
Policy from config file:        targeted

Running restorecon changed something:

[root@kvm-san zaitcev]# ls -Z /srv/node/vdb/objects/19579/365/4c7babc3b566d183d6fe30e75b7dd365/1359785130.43946.data
-rw-------. swift swift system_u:object_r:var_t:s0       /srv/node/vdb/objects/19579/365/4c7babc3b566d183d6fe30e75b7dd365/1359785130.43946.data

(the above uses the object that rsync replicated, thanks to permissive mode)

I'll try to scoop Dan's new policy from Rawhide and see if I can adopt it.
Comment 13 Miroslav Grepl 2013-02-05 02:20:26 EST
Created attachment 693173 [details]
swift policy files for RHEL6

I attached policy files for RHEL6.

Download/uncompress it and run

# make -f /usr/share/selinux/devel/Makefile swift.pp
# semodule -i swift.pp
# restorecon -Rv /usr/bin/swift* /var/run/swift

and start the service. Test it and

# ausearch -m avc -ts recent
Comment 14 Pete Zaitcev 2013-02-06 12:46:15 EST
I can easily add account and container to Miroslav's policy, where he
only described the object services. But before that, I don't understand
how it is supposed to deal with the actual data that Swift manages in
/srv/node. First, once you confine the daemons that way, they cannot
write there anymore (:object_r:var_t:), can they? Second, the whole problem
of rsync needing to write in /srv/node that we're taking about, how is
that solved?
Comment 15 Miroslav Grepl 2013-02-07 02:18:41 EST
If you uncomment 

#optional_policy(`
# unconfined_domain(swift_t)
#')

which is what we will do for RHEL6 then the swift_t domain can access everything. 

As Dan wrote

> SO we need a policy for swift-object-se* which will not transition to rsync.
Comment 16 Lon Hohberger 2013-02-12 12:54:06 EST
Is the swift package going to include Miroslav's work, or should this be added to the openstack-selinux policy RHEL / RHOS policy package?
Comment 21 Pete Zaitcev 2013-02-13 00:33:06 EST
I am clueless about the implications of having swift_exec_t as opposed to
swift_var_run_t. Perhaps it's ok to have both, from SELinux perspective.
I trust Lon knows what he's doing.

Here's the list of daemons that write to /srv/node/* volumes and
invoke rsync (and then rsync writes to /srv/node too, on another box):

swift-account-auditor
swift-account-reaper
swift-account-replicator
swift-account-server
swift-container-auditor
swift-container-replicator
swift-container-server
swift-container-sync <==== Container Sync is not well supported, so whatever
swift-container-updater
swift-object-auditor
swift-object-replicator
swift-object-server
swift-object-updater

Miroslav's example only dealt with the Object server, not Account and
Container for some reason.

The stuff that runs on Proxy node, swift-proxy-server and
swift-object-expirer, does not have direct access to /srv/node.
So as long as they have access to network, syslog, and other typical
daemony stuff, it's all good (including /var/cache/swift and /var/run/swift).

In addition, I think, some people can run swift-drive-audit from cron
or other daemon-like mechanism.

All the rest of the tooling is ran by root from shell. This includes
swift-drive-audit.
Comment 22 Miroslav Grepl 2013-02-13 04:28:42 EST
Then daemon binaries should be labeled as swift_exec_t and probably we also need to add new types for

/var/cache/swift 

and maybe others.

Lon,
could you try to label deamon binaries using

# chcon -t swift_exec_t PATHO_SWIFT_DAEMON_BINARIES
# semodule -d unconfined

on a system with swift policy and re-test it and collect AVC msgs. Then we can update the policy file.
Comment 27 Pete Zaitcev 2013-02-14 00:30:14 EST
I installed the packages:

openstack-swift-1.7.4-7.el6.noarch
openstack-swift-account-1.7.4-7.el6.noarch
openstack-swift-container-1.7.4-7.el6.noarch
openstack-swift-object-1.7.4-7.el6.noarch
openstack-swift-proxy-1.7.4-7.el6.noarch
openstack-selinux-0.1.2-3.el6ost.noarch
selinux-policy-3.7.19-195.el6.noarch
selinux-policy-targeted-3.7.19-195.el6.noarch

I started the services using service(8), because I know that using
swift-init makes them run with wrong SElinux credentials.

Then, I damaged an object and received a flurry of AVC denials.

1. auditor cannot even access its own data in /srv/node/*

type=AVC msg=audit(1360818111.266:459): avc:  denied  { create } for  pid=4907 comm="swift-object-au" name="objects" scontext=unconfined_u:system_r:swift_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=dir
type=AVC msg=audit(1360818111.267:460): avc:  denied  { reparent } for  pid=4907 comm="swift-object-au" name="12be63ab2ac79aeb57d8440b686c1457" dev=vdc ino=2107444 scontext=unconfined_u:system_r:swift_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=dir

Note that I do not see any denials for the replicator. Maybe audit
rate-limits denials? In the policy source auditor and replicator looked
like having identical contexts.

2. execution of rsync is prohibited

type=AVC msg=audit(1360818121.768:462): avc:  denied  { execute_no_trans } for  pid=5287 comm="swift-object-re" path="/usr/bin/rsync" dev=dm-0 ino=7095 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:rsync_exec_t:s0 tclass=file

3. rsync cannot take its locks in /var/lock

type=AVC msg=audit(1360818121.814:465): avc:  denied  { lock } for  pid=5288 comm="rsync" path="/var/lock/object.lock" dev=dm-0 ino=152866 scontext=system_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lock_t:s0 tclass=file

4. rsync cannot access the data in /srv/node/* -- when invoked by Swift

type=AVC msg=audit(1360818122.849:471): avc:  denied  { getattr } for  pid=5289 comm="rsync" path="/vdc/objects/4798/457/12be63ab2ac79aeb57d8440b686c1457/.1360022499.86114.data.UVSSuU" dev=vdc ino=2107448 scontext=system_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_t:s0 tclass=file
type=AVC msg=audit(1360818122.851:472): avc:  denied  { rename } for  pid=5289 comm="rsync" name=".1360022499.86114.data.UVSSuU" dev=vdc ino=2107448 scontext=system_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_t:s0 tclass=file

5. Object server has same issues as auditor:


type=AVC msg=audit(1360818122.896:473): avc:  denied  { read } for  pid=5036 comm="swift-object-se" name="457" dev=vdc ino=1048623 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir
type=AVC msg=audit(1360818122.897:474): avc:  denied  { rmdir } for  pid=5036 comm="swift-object-se" name="457" dev=vdc ino=1048623 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir

6. BTW, rsync cannot talk to the network

type=AVC msg=audit(1360818121.792:463): avc:  denied  { name_connect } for  pid=5287 comm="rsync" dest=873 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:rsync_port_t:s0 tclass=tcp_socket
Comment 28 Miroslav Grepl 2013-02-14 01:48:04 EST
Ok, these AVC we will "mask" using unconfined_domain(swift_t) for now. So everything will work with

# semodule -e unconfined

Basically I was looking for different AVC msgs from other daemon binaries.
Comment 29 Pete Zaitcev 2013-02-14 09:30:41 EST
I have a question: should we somehow label mount points under /srv/node,
for example with swift_var_t? If we don't, do we allow Swift daemons
to write anywhere in filesystems (making it essentially unconfined),
or is there other mechanism?

Obviously the location of /srv/node may be changed by configuration,
but let's assume we document that it should be left alone for SELinux.
Can we add it to selinux-openstack?
Comment 30 Daniel Walsh 2013-02-15 08:50:51 EST
Yes we want to confine all of openstack in the future, so any directory that has to be written by an openstack domain needs to have a label.  And we should setup the default labeling.  Any config file that is used to change a label or port should have a comment about SELinux.

For example:

cat /etc/ssh/sshd_config
...
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# If you want to change the port on an SELinux system, you have to tell 
# SELinux about this change. 
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
#Port 22
Comment 34 Martina Kollarova 2013-02-20 14:27:14 EST
I'm still getting denials when rsync is accessing locks. Is this somehow expected?

type=AVC msg=audit(1361387385.114:8310): avc:  denied  { write } for  pid=30929 comm="rsync" name="lock" dev=dm-1 ino=131367 scontext=unconfined_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lock_t:s0 tclass=dir
Comment 35 Miroslav Grepl 2013-02-21 03:08:41 EST
Does it relate with swift? Or did you just test rsync?

$ cat /tmp/log |audit2allow


#============= rsync_t ==============
#!!!! This avc can be allowed using the boolean 'rsync_full_access'

allow rsync_t var_lock_t:dir write;
Comment 36 Martina Kollarova 2013-02-21 07:41:11 EST
I didn't run rsync manually if you mean that.


This gets repeated in /var/log/messages every minute:
Feb 21 12:27:08 x2 container-replicator no_change:2 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
Feb 21 12:27:29 x2 xinetd[28003]: START: rsync pid=6803 from=10.34.1.148
Feb 21 12:27:29 x2 rsyncd[6803]: connect from x3 (10.34.1.148)
Feb 21 12:27:29 x2 rsyncd[6803]: rsync: failed to open lock file /var/lock/object.lock: Permission denied (13)
Feb 21 12:27:29 x2 xinetd[28003]: EXIT: rsync status=255 pid=6803 duration=0(sec)
Feb 21 12:27:30 x2 object-replicator Starting object replication pass.
Feb 21 12:27:30 x2 object-replicator 2/2 (100.00%) partitions replicated in 0.00s (636.32/sec, 0s remaining)
Feb 21 12:27:30 x2 object-replicator 1 suffixes checked - 0.00% hashed, 0.00% synced


Output from audit2allow is:
allow rsync_t var_lock_t:dir { write add_name };
allow rsync_t var_lock_t:file { write create };
Comment 37 Miroslav Grepl 2013-02-21 09:08:11 EST
Do you have latest openstack-selinux pkg installed?

# semodule -l |grep swift
Comment 38 Martina Kollarova 2013-02-21 09:32:09 EST
# semodule -l |grep swift
swift	1.0.0

# rpm -q openstack-selinux
openstack-selinux-0.1.2-3.el6ost.noarch
Comment 39 Daniel Walsh 2013-02-21 11:05:28 EST
This means rsync was started by a domain labeled initrc_t.

ps -eZ | grep initrc_t
Comment 40 Miroslav Grepl 2013-02-21 11:13:05 EST
Ok, we have more swift services running as initrc_t.


unconfined_u:system_r:initrc_t:s0 28933 ?      00:00:00 swift-account-s
unconfined_u:system_r:initrc_t:s0 28969 ?      00:00:04 swift-account-s
unconfined_u:system_r:initrc_t:s0 29161 ?      00:00:00 swift-container
unconfined_u:system_r:initrc_t:s0 29191 ?      00:00:08 swift-container
unconfined_u:system_r:initrc_t:s0 29210 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29221 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29222 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29223 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29224 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29225 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29226 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29227 ?      00:00:00 swift-proxy-ser
unconfined_u:system_r:initrc_t:s0 29228 ?      00:00:00 swift-proxy-ser
Comment 41 Miroslav Grepl 2013-02-21 11:19:57 EST
Martina,
could you re-test it now. I changed swift-*-server labeling. So you need to restart all swift services to test it.
Comment 42 Miroslav Grepl 2013-02-21 12:34:39 EST
Martina,
good catch

#============= inetd_t ==============
allow inetd_t rsync_exec_test_file_t:file { read execute open execute_no_trans };

So this is a policy bug which we need to fix.
Comment 43 Yaniv Kaul 2013-02-21 15:10:18 EST
(In reply to comment #42)
> Martina,
> good catch
> 
> #============= inetd_t ==============
> allow inetd_t rsync_exec_test_file_t:file { read execute open
> execute_no_trans };
> 
> So this is a policy bug which we need to fix.

So should it move back to development?
Comment 44 Miroslav Grepl 2013-02-22 02:56:39 EST
No, it was just a test type. 

We will need to add

type rsync_lock_t;
files_lock_file(rsync_lock_t)

manage_dirs_pattern(rsync_t, rsync_lock_t, rsync_lock_t)
manage_files_pattern(rsync_t, rsync_lock_t, rsync_lock_t)
files_lock_filetrans(rsync_t, rsync_lock_t, { dir file })
Comment 46 Lon Hohberger 2013-02-22 10:50:12 EST
Martina / Miroslav - is that policy known to work / should I add it for now to openstack-selinux?
Comment 52 Martina Kollarova 2013-02-22 13:08:29 EST
The policy in comment #44 didn't pass, but the new package (comment #50) works.

Tests:
Manually removed files from a device (the whole /srv/node/device1/objects/xxx dir). With selinux disabled, it got re-created in under a minute. With selinux enabled and without the fix, I was getting rsync AVC denials and the object didn't get re-created. Now there is no problem.

I also tried adding another device and rebalancing, then doing some more damage to the files.
Comment 54 errata-xmlrpc 2013-03-05 13:29:56 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0593.html