Bug 885529
Summary: | swift replication produces SELinux AVC denials | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Derek Higgins <derekh> | ||||
Component: | openstack-selinux | Assignee: | Lon Hohberger <lhh> | ||||
Status: | CLOSED ERRATA | QA Contact: | Martina Kollarova <mkollaro> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 2.0 (Folsom) | CC: | apevec, breu, david, d.busby, derekh, dwalsh, eparis, jhenner, jonathansteffan, lhh, markmc, mgrepl, ncredi, silas, zaitcev | ||||
Target Milestone: | snapshot3 | Keywords: | Triaged | ||||
Target Release: | 2.1 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | openstack-selinux-0.1.2-5.el6 | Doc Type: | Bug Fix | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | 809198 | Environment: | |||||
Last Closed: | 2013-03-05 18:29:56 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 902486 | ||||||
Bug Blocks: | 918721, 922787 | ||||||
Attachments: |
|
Description
Derek Higgins
2012-12-10 01:12:21 UTC
This may be relevant: Dan Walsh on issues with rsync and SElinux: http://danwalsh.livejournal.com/61646.html "The biggest problem SELinux has with rsync is there is no way to distinguish between the client and the server from an SELinux point of view." "When a process running as init_t or initrc_t executes /usr/bin/rsync (rsync_exec_t), the rsync process will transition to rsync_t and SELinux will treat it as the rsync daemon not as a client." "Best would be to write policy for the init service that is currently running without confinement." Of course in our case Swift daemons are already confined but rsync is not added there for network or some other kink like that. Lets look at a couple of problems here. First I see file_t files in your audit.log (avc) report, this means that we have files that do not have labels on them. The simplest fix would be to remove the rsync_exec_t label from rsync, and then I think your stuff will work with SELinux in enforcing mode. I was factually wrong about Swift being confined, or I do not have the right package installed. On my RHEL 6.3 RHOS 2.1, we have ps -eZ: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 6568 ? 00:00:11 swift-object-se unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 6601 ? 00:00:13 swift-object-re (for Object Server and Object Replicator) system_u:system_r:inetd_t:s0-s0:c0.c1023 1544 ? 00:00:00 xinetd Pete, did you start these processes by hand from the command line? Things users run from the command line are typically left unconfined whereas those same things started by the init system may be confined. I'm not certain which is the right way to start confined daemons from the command line... either service blah start OR run_init service blah start It seems to change ever day.... Yes, I started Swift from CLI with swift-init. After launching with service(8), we have: unconfined_u:system_r:initrc_t:s0 10283 ? 00:00:00 swift-object-se Sorry, it didn't occur to me. This may be a bit of a problem, because using swift-init is rather well seeded throughout the docs. We didn't even have proper scripts until very recenly (bz#885530). SO we need a policy for swift-object-se* which will not transition to rsync. What is this executable. Just added openstack-swift policy to Rawhide, need to back port this and make it unconfined for RHEL6. THen it would not transition to rsync_t. Confining Swift daemons is good, but what about the rsync daemon accessing the /srv/node/*? It throws this: type=AVC msg=audit(1359784683.733:23858): avc: denied { setattr } for pid=23659 comm="rsync" name=".1355550388.32345.data.WODj0u" dev=vdb ino=3145762 scontext=unconfined_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file type=AVC msg=audit(1359784683.733:23859): avc: denied { getattr } for pid=23659 comm="rsync" path="/vdb/objects/19579/365/4c7babc3b566d183d6fe30e75b7dd365/.1355550388.32345.data.WODj0u" dev=vdb ino=3145762 scontext=unconfined_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file (Audit log shows absolute-looking /vdb path for /srv/node/vdb/*, possibly because it's a mount point.) file_t indicates that the files in question have no label assigned. were they created with selinux disabled? You can use restorecon to restore them to 'default' restorecon -RvF /srv/node/vdb/ to make your rsync unconfined until a new policy arrives: chcon -t bin_t `which rsync` audit does all sorts of horrible things around path names. it sucks, but from the kernel PoV there may be NO path between some / and the file in question... Everything audit tells you is true, but piecing it back together from the inode number, block device, etc, can be a pain in the ass for an admin. I agree, but there is no reliable way to do it... so don't blame selinux for audit's, ummm lets call it, uniqueness :) The Swift nodes where I test for this bug were not running with SELinux disabled, ever, but it is set to permissive while I'm figuring this out: [root@kvm-san zaitcev]# sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: permissive Mode from config file: permissive Policy version: 24 Policy from config file: targeted Running restorecon changed something: [root@kvm-san zaitcev]# ls -Z /srv/node/vdb/objects/19579/365/4c7babc3b566d183d6fe30e75b7dd365/1359785130.43946.data -rw-------. swift swift system_u:object_r:var_t:s0 /srv/node/vdb/objects/19579/365/4c7babc3b566d183d6fe30e75b7dd365/1359785130.43946.data (the above uses the object that rsync replicated, thanks to permissive mode) I'll try to scoop Dan's new policy from Rawhide and see if I can adopt it. Created attachment 693173 [details]
swift policy files for RHEL6
I attached policy files for RHEL6.
Download/uncompress it and run
# make -f /usr/share/selinux/devel/Makefile swift.pp
# semodule -i swift.pp
# restorecon -Rv /usr/bin/swift* /var/run/swift
and start the service. Test it and
# ausearch -m avc -ts recent
I can easily add account and container to Miroslav's policy, where he only described the object services. But before that, I don't understand how it is supposed to deal with the actual data that Swift manages in /srv/node. First, once you confine the daemons that way, they cannot write there anymore (:object_r:var_t:), can they? Second, the whole problem of rsync needing to write in /srv/node that we're taking about, how is that solved? If you uncomment
#optional_policy(`
# unconfined_domain(swift_t)
#')
which is what we will do for RHEL6 then the swift_t domain can access everything.
As Dan wrote
> SO we need a policy for swift-object-se* which will not transition to rsync.
Is the swift package going to include Miroslav's work, or should this be added to the openstack-selinux policy RHEL / RHOS policy package? I am clueless about the implications of having swift_exec_t as opposed to swift_var_run_t. Perhaps it's ok to have both, from SELinux perspective. I trust Lon knows what he's doing. Here's the list of daemons that write to /srv/node/* volumes and invoke rsync (and then rsync writes to /srv/node too, on another box): swift-account-auditor swift-account-reaper swift-account-replicator swift-account-server swift-container-auditor swift-container-replicator swift-container-server swift-container-sync <==== Container Sync is not well supported, so whatever swift-container-updater swift-object-auditor swift-object-replicator swift-object-server swift-object-updater Miroslav's example only dealt with the Object server, not Account and Container for some reason. The stuff that runs on Proxy node, swift-proxy-server and swift-object-expirer, does not have direct access to /srv/node. So as long as they have access to network, syslog, and other typical daemony stuff, it's all good (including /var/cache/swift and /var/run/swift). In addition, I think, some people can run swift-drive-audit from cron or other daemon-like mechanism. All the rest of the tooling is ran by root from shell. This includes swift-drive-audit. Then daemon binaries should be labeled as swift_exec_t and probably we also need to add new types for /var/cache/swift and maybe others. Lon, could you try to label deamon binaries using # chcon -t swift_exec_t PATHO_SWIFT_DAEMON_BINARIES # semodule -d unconfined on a system with swift policy and re-test it and collect AVC msgs. Then we can update the policy file. I installed the packages: openstack-swift-1.7.4-7.el6.noarch openstack-swift-account-1.7.4-7.el6.noarch openstack-swift-container-1.7.4-7.el6.noarch openstack-swift-object-1.7.4-7.el6.noarch openstack-swift-proxy-1.7.4-7.el6.noarch openstack-selinux-0.1.2-3.el6ost.noarch selinux-policy-3.7.19-195.el6.noarch selinux-policy-targeted-3.7.19-195.el6.noarch I started the services using service(8), because I know that using swift-init makes them run with wrong SElinux credentials. Then, I damaged an object and received a flurry of AVC denials. 1. auditor cannot even access its own data in /srv/node/* type=AVC msg=audit(1360818111.266:459): avc: denied { create } for pid=4907 comm="swift-object-au" name="objects" scontext=unconfined_u:system_r:swift_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=dir type=AVC msg=audit(1360818111.267:460): avc: denied { reparent } for pid=4907 comm="swift-object-au" name="12be63ab2ac79aeb57d8440b686c1457" dev=vdc ino=2107444 scontext=unconfined_u:system_r:swift_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=dir Note that I do not see any denials for the replicator. Maybe audit rate-limits denials? In the policy source auditor and replicator looked like having identical contexts. 2. execution of rsync is prohibited type=AVC msg=audit(1360818121.768:462): avc: denied { execute_no_trans } for pid=5287 comm="swift-object-re" path="/usr/bin/rsync" dev=dm-0 ino=7095 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:rsync_exec_t:s0 tclass=file 3. rsync cannot take its locks in /var/lock type=AVC msg=audit(1360818121.814:465): avc: denied { lock } for pid=5288 comm="rsync" path="/var/lock/object.lock" dev=dm-0 ino=152866 scontext=system_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lock_t:s0 tclass=file 4. rsync cannot access the data in /srv/node/* -- when invoked by Swift type=AVC msg=audit(1360818122.849:471): avc: denied { getattr } for pid=5289 comm="rsync" path="/vdc/objects/4798/457/12be63ab2ac79aeb57d8440b686c1457/.1360022499.86114.data.UVSSuU" dev=vdc ino=2107448 scontext=system_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_t:s0 tclass=file type=AVC msg=audit(1360818122.851:472): avc: denied { rename } for pid=5289 comm="rsync" name=".1360022499.86114.data.UVSSuU" dev=vdc ino=2107448 scontext=system_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_t:s0 tclass=file 5. Object server has same issues as auditor: type=AVC msg=audit(1360818122.896:473): avc: denied { read } for pid=5036 comm="swift-object-se" name="457" dev=vdc ino=1048623 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir type=AVC msg=audit(1360818122.897:474): avc: denied { rmdir } for pid=5036 comm="swift-object-se" name="457" dev=vdc ino=1048623 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir 6. BTW, rsync cannot talk to the network type=AVC msg=audit(1360818121.792:463): avc: denied { name_connect } for pid=5287 comm="rsync" dest=873 scontext=unconfined_u:system_r:swift_t:s0 tcontext=system_u:object_r:rsync_port_t:s0 tclass=tcp_socket Ok, these AVC we will "mask" using unconfined_domain(swift_t) for now. So everything will work with # semodule -e unconfined Basically I was looking for different AVC msgs from other daemon binaries. I have a question: should we somehow label mount points under /srv/node, for example with swift_var_t? If we don't, do we allow Swift daemons to write anywhere in filesystems (making it essentially unconfined), or is there other mechanism? Obviously the location of /srv/node may be changed by configuration, but let's assume we document that it should be left alone for SELinux. Can we add it to selinux-openstack? Yes we want to confine all of openstack in the future, so any directory that has to be written by an openstack domain needs to have a label. And we should setup the default labeling. Any config file that is used to change a label or port should have a comment about SELinux. For example: cat /etc/ssh/sshd_config ... # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # If you want to change the port on an SELinux system, you have to tell # SELinux about this change. # semanage port -a -t ssh_port_t -p tcp #PORTNUMBER # #Port 22 I'm still getting denials when rsync is accessing locks. Is this somehow expected? type=AVC msg=audit(1361387385.114:8310): avc: denied { write } for pid=30929 comm="rsync" name="lock" dev=dm-1 ino=131367 scontext=unconfined_u:system_r:rsync_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lock_t:s0 tclass=dir Does it relate with swift? Or did you just test rsync? $ cat /tmp/log |audit2allow #============= rsync_t ============== #!!!! This avc can be allowed using the boolean 'rsync_full_access' allow rsync_t var_lock_t:dir write; I didn't run rsync manually if you mean that. This gets repeated in /var/log/messages every minute: Feb 21 12:27:08 x2 container-replicator no_change:2 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0 Feb 21 12:27:29 x2 xinetd[28003]: START: rsync pid=6803 from=10.34.1.148 Feb 21 12:27:29 x2 rsyncd[6803]: connect from x3 (10.34.1.148) Feb 21 12:27:29 x2 rsyncd[6803]: rsync: failed to open lock file /var/lock/object.lock: Permission denied (13) Feb 21 12:27:29 x2 xinetd[28003]: EXIT: rsync status=255 pid=6803 duration=0(sec) Feb 21 12:27:30 x2 object-replicator Starting object replication pass. Feb 21 12:27:30 x2 object-replicator 2/2 (100.00%) partitions replicated in 0.00s (636.32/sec, 0s remaining) Feb 21 12:27:30 x2 object-replicator 1 suffixes checked - 0.00% hashed, 0.00% synced Output from audit2allow is: allow rsync_t var_lock_t:dir { write add_name }; allow rsync_t var_lock_t:file { write create }; Do you have latest openstack-selinux pkg installed? # semodule -l |grep swift # semodule -l |grep swift swift 1.0.0 # rpm -q openstack-selinux openstack-selinux-0.1.2-3.el6ost.noarch This means rsync was started by a domain labeled initrc_t. ps -eZ | grep initrc_t Ok, we have more swift services running as initrc_t. unconfined_u:system_r:initrc_t:s0 28933 ? 00:00:00 swift-account-s unconfined_u:system_r:initrc_t:s0 28969 ? 00:00:04 swift-account-s unconfined_u:system_r:initrc_t:s0 29161 ? 00:00:00 swift-container unconfined_u:system_r:initrc_t:s0 29191 ? 00:00:08 swift-container unconfined_u:system_r:initrc_t:s0 29210 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29221 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29222 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29223 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29224 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29225 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29226 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29227 ? 00:00:00 swift-proxy-ser unconfined_u:system_r:initrc_t:s0 29228 ? 00:00:00 swift-proxy-ser Martina, could you re-test it now. I changed swift-*-server labeling. So you need to restart all swift services to test it. Martina, good catch #============= inetd_t ============== allow inetd_t rsync_exec_test_file_t:file { read execute open execute_no_trans }; So this is a policy bug which we need to fix. (In reply to comment #42) > Martina, > good catch > > #============= inetd_t ============== > allow inetd_t rsync_exec_test_file_t:file { read execute open > execute_no_trans }; > > So this is a policy bug which we need to fix. So should it move back to development? No, it was just a test type. We will need to add type rsync_lock_t; files_lock_file(rsync_lock_t) manage_dirs_pattern(rsync_t, rsync_lock_t, rsync_lock_t) manage_files_pattern(rsync_t, rsync_lock_t, rsync_lock_t) files_lock_filetrans(rsync_t, rsync_lock_t, { dir file }) Martina / Miroslav - is that policy known to work / should I add it for now to openstack-selinux? The policy in comment #44 didn't pass, but the new package (comment #50) works. Tests: Manually removed files from a device (the whole /srv/node/device1/objects/xxx dir). With selinux disabled, it got re-created in under a minute. With selinux enabled and without the fix, I was getting rsync AVC denials and the object didn't get re-created. Now there is no problem. I also tried adding another device and rebalancing, then doing some more damage to the files. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0593.html |