Description of problem: When I (re)start cobblerd, I get traceback and SELinux AVC messages. Version-Release number of selected component (if applicable): cobbler-2.2.3-2.el6.noarch selinux-policy-targeted-3.7.19-154.el6.noarch How reproducible: Most of mine attempts on RHEL6. On F16 I see AVCs only, no traceback. Steps to Reproduce: 1. # getenforce # ensure you are in "Enforcing" 2. # tailf /var/log/audit/audit.log & 3. # service cobblerd restart Actual results: # service cobblerd restart Stopping cobbler daemon: [FAILED] Starting cobbler daemon: Traceback (most recent call last): File "/usr/bin/cobblerd", line 76, in main api = cobbler_api.BootAPI(is_cobblerd=True) File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 127, in __init__ type=AVC msg=audit(1341918718.623:372): avc: denied { execute } for pid=16536 comm="cobblerd" path=2F746D702F666669364468346850202864656C6574656429 dev=dm-0 ino=2892957 scontext=unconfined_u:system_r:cobblerd_t:s0 tcontext=unconfined_u:object_r:cobbler_tmp_t:s0 tclass=file type=SYSCALL msg=audit(1341918718.623:372): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=5 a3=1 items=0 ppid=16535 pid=16536 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=23 comm="cobblerd" exe="/usr/bin/python" subj=unconfined_u:system_r:cobblerd_t:s0 key=(null) module_loader.load_modules() File "/usr/lib/python2.6/site-packages/cobbler/module_loader.py", line 62, in load_modules blip = __import__("modules.%s" % ( modname), globals(), locals(), [modname]) File "/usr/lib/python2.6/site-packages/cobbler/modules/authn_pam.py", line 53, in <module> from ctypes import CDLL, POINTER, Structure, CFUNCTYPE, cast, pointer, sizeof File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module> type=AVC msg=audit(1341918718.623:373): avc: denied { execute } for pid=16536 comm="cobblerd" path=2F7661722F746D702F66666954695834714B202864656C6574656429 dev=dm-0 ino=1183721 scontext=unconfined_u:system_r:cobblerd_t:s0 tcontext=unconfined_u:object_r:cobbler_tmp_t:s0 tclass=file CFUNCTYPE(c_int)(lambda: None) MemoryError type=SYSCALL msg=audit(1341918718.623:373): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=5 a3=1 items=0 ppid=16535 pid=16536 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=23 comm="cobblerd" exe="/usr/bin/python" subj=unconfined_u:system_r:cobblerd_t:s0 key=(null) type=AVC msg=audit(1341918718.623:374): avc: denied { search } for pid=16536 comm="cobblerd" name="/" dev=tmpfs ino=5451 scontext=unconfined_u:system_r:cobblerd_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir type=SYSCALL msg=audit(1341918718.623:374): arch=c000003e syscall=2 success=no exit=-13 a0=7fffd5b06600 a1=c2 a2=180 a3=1 items=0 ppid=16535 pid=16536 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=23 comm="cobblerd" exe="/usr/bin/python" subj=unconfined_u:system_r:cobblerd_t:s0 key=(null) [ OK ] Expected results: Cobbler should be restarted without tracebacks and AVC messages. Additional info: I have found some references to that issue: * http://www.mail-archive.com/cobbler@lists.fedorahosted.org/msg07650.html * https://bugs.gentoo.org/show_bug.cgi?id=332419 * https://bugzilla.redhat.com/show_bug.cgi?id=645193 * https://bugzilla.redhat.com/show_bug.cgi?id=674369 Not sure if this is Cobbler or SELinux or some body else problem. I have not seen this in cobbler-2.2.2-1.el6.
Any news on this issue? As it stands, cobbler is unusable.
Bump. I'm also getting this problem, official support tell me to turn off selinux which will only confirm the problem. No official workarounds or fixes suggested.
(In reply to comment #2) > > I'm also getting this problem, official support tell me to turn off selinux Out of curiosity -- what official support did tell you that?
Jan: https://github.com/cobbler/cobbler/wiki/Selinux On F16, and RHEL 6.x "turn off SELinux". AFAIK, it tries to access /dev/shm Oct 9 16:32:25 Merak setroubleshoot: SELinux is preventing /usr/bin/python from execute access on the file /tmp/ffiN8UuK3 (deleted). For complete SELinux messages. run sealert -l 69b08e98-dccb-4d1b-b36a-abd2cc57e73a Oct 9 16:33:40 Merak setroubleshoot: SELinux is preventing /usr/bin/python from search access on the directory /dev/shm. For complete SELinux messages. run sealert -l 11cc91de-8be6-43ff-aa8e-e650d829f7a1 [root@Merak ~]# LANG=C sealert -l 11cc91de-8be6-43ff-aa8e-e650d829f7a1 SELinux is preventing /usr/bin/python from search access on the directory /dev/shm. ***** Plugin catchall (100. confidence) suggests *************************** If you believe that python should be allowed search access on the shm directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep cobblerd /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp
(In reply to comment #4) And my releases: cobbler-2.2.3-2.el6.noarch selinux-policy-targeted-3.7.19-155.el6_3.4.noarch Running on EL 6.3+ updates on x86_64
(In reply to comment #4) > Jan: https://github.com/cobbler/cobbler/wiki/Selinux > > On F16, and RHEL 6.x "turn off SELinux". Well, the upstream does not care about SELinux. While sad, it's understandable. We however do distribution based on upstream projects and have maintainers for the very reason of integrating the upstream software to Fedora or EPEL. > Oct 9 16:32:25 Merak setroubleshoot: SELinux is preventing /usr/bin/python from execute access on the file /tmp/ffiN8UuK3 (deleted). For complete SELinux messages. run sealert -l 69b08e98-dccb-4d1b-b36a-abd2cc57e73 This is however a different message than the original search issue, isn't it? What is the raw AVC denial?
> (In reply to comment #4) > > Jan: https://github.com/cobbler/cobbler/wiki/Selinux > > > > On F16, and RHEL 6.x "turn off SELinux". > > Well, the upstream does not care about SELinux. While sad, it's > understandable. That is not correct. The wiki has been deprecated (we need to make that more clear), and this is the manual page for SELinux on 2.2.3: http://cobbler.github.com/manuals/2.2.3/5/2_-_SELinux.html Note the "Fedora 16 / RHEL6 / CentOS6 - Python MemoryError" section at the bottom, which is the issue above. What we DO recommend is that you disable SELinux unless you are comfortable writing policy. This is due to the fact that when cobbler includes a new feature it will most likely break with SELinux enabled.
(In reply to comment #7) > > > > Well, the upstream does not care about SELinux. While sad, it's > > understandable. > > That is not correct. The wiki has been deprecated (we need to make that more > clear), and this is the manual page for SELinux on 2.2.3: > > http://cobbler.github.com/manuals/2.2.3/5/2_-_SELinux.html > > Note the "Fedora 16 / RHEL6 / CentOS6 - Python MemoryError" section at the > bottom, which is the issue above. > > What we DO recommend is that you disable SELinux unless you are comfortable > writing policy. This is due to the fact that when cobbler includes a new > feature it will most likely break with SELinux enabled. Nothing of which applies because we expect the package maintainer to integrate with other parts of the distribution *and test* before moving to the next upstream version (which in EPEL should only happen very rarely anyway). Noone should be required to write any policy code when using packages from Fedora, RHEL, or EPEL. Which brings us to: are you planning on fixing this bug (three months old, now)?
As far as I can tell, this isn't a bug with cobbler, it has to do with SELinux blocking python ctypes from executing properly, note the last line of the trace: File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module> The workarounds I mention in the manual page above are to get around that until the underlying issue is fixed, which is not cobbler's issue. That being said, cobbler does not ship with SELinux policy files, as it is included in fedora's SELinux policy because of things like spacewalk: # rpm -qf /etc/selinux/targeted/modules/active/modules/cobbler.pp selinux-policy-targeted-3.10.0-153.fc17.noarch Nor do I intend to ship supplementary policy files, because they can cludge up the system. Instead, if I find something that does break the policy I will create tickets to have the base policy fixed. This leads to a lag in time before the fix is officially available, for which I will make notes in the manual page (as I did above). The percentage of cobbler users running with SELinux enabled is relatively small, so I feel this works for now. Note that I do not ship releases with known policy issues, but they do pop up. If you don't care for that policy and it's important to you, how about submitting some patches to either Dan/Fedora or myself to fix the issue instead?
Clearly our views are completely different. I filed a FESCo ticket to help us find some setup that will work both for you as the package maintainer, and for cobbler users as well. https://fedorahosted.org/fesco/ticket/959
Regarding the bug itself, it looks like this bug in python: https://bugzilla.redhat.com/show_bug.cgi?id=814391 (Looks fixed in F17 but not in RHEL6). How to fix bugs in packages in epel is probably something that should be discussed on the epel mailing list. I'm just one contributor but it seems like past policy has been that we don't control RHEL so we should be fixing our packages to work with RHEL as it exists. If I'm understanding the manual page link correctly, that might mean shipping our cobbler package without authn_pam.py. Or alternately, patching authn_pam.py to catch the error and disable that feature (and logging why the feature isn't working properly).
FWIW, I have found RHEL selinux-policy maintainers very willing to work with EPEL maintainers to adjust policy. has anyone filed a selinux-policy bug on this? Or asked them for comment? I think short term adjusting to work with existing RHEL packages is good, and then longer term asking for fixes in python/selinux-policy is the way to go. You can then remove the workarounds once policy/python is fixed?
Note: For SELinux to work around this, I think it would need to make it so /usr/bin/python can execute files in tmpdirs. /usr/bin/python doesn't have any special contexts (It's bin_t) so it'd likely mean adding a sebool to allow basically anything to execute files in tmpdirs
(In reply to comment #13) > Note: For SELinux to work around this, I think it would need to make it so > /usr/bin/python can execute files in tmpdirs. /usr/bin/python doesn't have What kind of files are they that need to be executed in tmpdirs? Cannot cobbler create them in some place special where they could have well defined label?
Jan, yet again I must point out that the issue is not Cobbler, it is python. Cobbler is not creating these files - the ctypes library is, thus the issue needs to be fixed in the policy for ctypes as Toshio pointed out. For example, there is already a boolean for httpd to do this (httpd_tmp_exec).
(In reply to comment #15) > Jan, yet again I must point out that the issue is not Cobbler, it is python. > Cobbler is not creating these files - the ctypes library is, thus the issue > needs to be fixed in the policy for ctypes as Toshio pointed out. For > example, there is already a boolean for httpd to do this (httpd_tmp_exec). Fair enough. The issues is still with cobbler which is shipping authn_pam.py (IIUIC) without having the ctypes issue cleared / resolved with selinux-policy maintainers first. Can you respin cobbler without authn_pam.py?
Coming from an end-user perspective, shouldn't issues that affect cobbler be reported further up by the package maintainers? Unless I'm mistaken, currently users are faced with cobbler not working out-of-the-box. Reporting the issue as cobbler related would be the appropriate begin of whatever chain of events may have to occur. Expecting end-users to track down what underlying problem involves and trying to address it is a bit of a stretch. If the expectation to have cobbler package maintainers report this issue further up the chain, perhaps adding documentation about having to disable SELINUX or whatever work-around is deemed appropriate would be of use. As noted already, end-users of this package are in a bit of a pickle. Cheers!
I believe that cobbler upstream is aware of the issue as they are the ones who wrote the analysis and workarounds on the online page. The python maintainer is also aware of the issue. I just pinged him about this and he took a look. Bug #814391 currently has status ON_QA which means the fix (along with other changes to the python package for the next RHEL6 point release) is currently in testing. If all goes well, that should be in the next RHEL point release. Someone should test whether that is enough to fix this problem, though. He says that this does fix some uses of ctypes but not all. In particular "this also assume that cobbler doesn't need to create callbacks from C code to Python. If it does, then 814391 won't help".
Created attachment 632243 [details] Workaround ctypes and SELinux incompatibility Here's a cobbler patch. I think it could be proposed upstream, and possibly accepted with some cleanups. It checks whether ctypes can be imported and if so, it lets you use authn_pam. If not, it prints a warning to stderr and disables that module. Built here if someone would like to test it (I don't use cobbler myself, I'm just a python guy). http://koji.fedoraproject.org/koji/taskinfo?taskID=4619186
Few updates: * this also affects cobbler on F16. * On F16, the change to the python package in bug #814391 is enough to make cobbler functional * On a RHEL6 test box, the change to the python package was not enough. We get past the traceback listed but then get this: Starting cobbler daemon: Traceback (most recent call last): File "/usr/bin/cobblerd", line 76, in main api = cobbler_api.BootAPI(is_cobblerd=True) File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 127, in __init__ module_loader.load_modules() File "/usr/lib/python2.6/site-packages/cobbler/module_loader.py", line 62, in load_modules blip = __import__("modules.%s" % ( modname), globals(), locals(), [modname]) File "/usr/lib/python2.6/site-packages/cobbler/modules/authn_pam.py", line 121, in <module> PAM_START = LIBPAM.pam_start File "/usr/lib64/python2.6/ctypes/__init__.py", line 366, in __getattr__ func = self.__getitem__(name) File "/usr/lib64/python2.6/ctypes/__init__.py", line 371, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) AttributeError: /usr/bin/python: undefined symbol: pam_start I also get this traceback if I run audit2allow to create a new policy and load that. I even generated the policy after disabling dontaudit (semodule -DB). Only setting SELinux to permissive gets around that traceback. I think this will require a RHEL SELinux person to diagnose. * I think I was wrong about being able to transition from the python interpreter's type to a cobbler type. cobbler is started by the init system and I believe the init system has the capability to make transitions. So policy can be written to target cobbler specifically. * After looking at the cobbler code some more I'm unsure why we don't catch exceptions from attempting to load any module... if there's an exception for any module at loadtime it seems like we would want logging of the error but not to crash the server. I can easily implement a patch that does that if you think that would fly upstream.
After some IRC discussion in #cobbler, I sent this pull request: https://github.com/cobbler/cobbler/pull/342 It will have the same effect as attachment 632243 [details] but is more generic: it will keep exceptions from crashing cobbler when loading any module.
I suspect this is fixed in cobbler-2.4.0-1.el6, please re-open if not. Will be pushing 2.4.3 out soon too.
Indeed, seems to be fixed (checked reproducer from initial comment in this BZ) in: # rpm -q cobbler selinux-policy-targeted cobbler-2.4.0-1.el6.noarch selinux-policy-targeted-3.7.19-231.el6_5.1.noarch