Bug 838898 - With SELInux enabled, `service cobblerd start` fails with AVC message and "CFUNCTYPE(c_int)(lambda: None)" MemoryError traceback
With SELInux enabled, `service cobblerd start` fails with AVC message and "CF...
Status: CLOSED CURRENTRELEASE
Product: Fedora EPEL
Classification: Fedora
Component: cobbler (Show other bugs)
el6
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: James C.
Fedora Extras Quality Assurance
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-10 07:52 EDT by Jan Hutař
Modified: 2014-04-28 02:46 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-04-21 14:42:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Workaround ctypes and SELinux incompatibility (9.87 KB, patch)
2012-10-23 13:30 EDT, Toshio Ernie Kuratomi
no flags Details | Diff

  None (edit)
Description Jan Hutař 2012-07-10 07:52:53 EDT
Description of problem:
When I (re)start cobblerd, I get traceback and SELinux AVC messages. 


Version-Release number of selected component (if applicable):
cobbler-2.2.3-2.el6.noarch
selinux-policy-targeted-3.7.19-154.el6.noarch


How reproducible:
Most of mine attempts on RHEL6.
On F16 I see AVCs only, no traceback.


Steps to Reproduce:
1. # getenforce   # ensure you are in "Enforcing"
2. # tailf /var/log/audit/audit.log &
3. # service cobblerd restart


Actual results:
# service cobblerd restart
Stopping cobbler daemon: [FAILED]
Starting cobbler daemon: Traceback (most recent call last):
  File "/usr/bin/cobblerd", line 76, in main
    api = cobbler_api.BootAPI(is_cobblerd=True)
  File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 127, in __init__
type=AVC msg=audit(1341918718.623:372): avc:  denied  { execute } for  pid=16536 comm="cobblerd" path=2F746D702F666669364468346850202864656C6574656429 dev=dm-0 ino=2892957 scontext=unconfined_u:system_r:cobblerd_t:s0 tcontext=unconfined_u:object_r:cobbler_tmp_t:s0 tclass=file
type=SYSCALL msg=audit(1341918718.623:372): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=5 a3=1 items=0 ppid=16535 pid=16536 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=23 comm="cobblerd" exe="/usr/bin/python" subj=unconfined_u:system_r:cobblerd_t:s0 key=(null)
    module_loader.load_modules()
  File "/usr/lib/python2.6/site-packages/cobbler/module_loader.py", line 62, in load_modules
    blip =  __import__("modules.%s" % ( modname), globals(), locals(), [modname])
  File "/usr/lib/python2.6/site-packages/cobbler/modules/authn_pam.py", line 53, in <module>
    from ctypes import CDLL, POINTER, Structure, CFUNCTYPE, cast, pointer, sizeof
  File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
type=AVC msg=audit(1341918718.623:373): avc:  denied  { execute } for  pid=16536 comm="cobblerd" path=2F7661722F746D702F66666954695834714B202864656C6574656429 dev=dm-0 ino=1183721 scontext=unconfined_u:system_r:cobblerd_t:s0 tcontext=unconfined_u:object_r:cobbler_tmp_t:s0 tclass=file
    CFUNCTYPE(c_int)(lambda: None)
MemoryError
type=SYSCALL msg=audit(1341918718.623:373): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=5 a3=1 items=0 ppid=16535 pid=16536 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=23 comm="cobblerd" exe="/usr/bin/python" subj=unconfined_u:system_r:cobblerd_t:s0 key=(null)
type=AVC msg=audit(1341918718.623:374): avc:  denied  { search } for  pid=16536 comm="cobblerd" name="/" dev=tmpfs ino=5451 scontext=unconfined_u:system_r:cobblerd_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir
type=SYSCALL msg=audit(1341918718.623:374): arch=c000003e syscall=2 success=no exit=-13 a0=7fffd5b06600 a1=c2 a2=180 a3=1 items=0 ppid=16535 pid=16536 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=23 comm="cobblerd" exe="/usr/bin/python" subj=unconfined_u:system_r:cobblerd_t:s0 key=(null)
[  OK  ]


Expected results:
Cobbler should be restarted without tracebacks and AVC messages.


Additional info:
I have found some references to that issue:

 * http://www.mail-archive.com/cobbler@lists.fedorahosted.org/msg07650.html
 * https://bugs.gentoo.org/show_bug.cgi?id=332419
 * https://bugzilla.redhat.com/show_bug.cgi?id=645193
 * https://bugzilla.redhat.com/show_bug.cgi?id=674369

Not sure if this is Cobbler or SELinux or some body else problem.

I have not seen this in cobbler-2.2.2-1.el6.
Comment 1 Dominik L. Borkowski 2012-09-05 08:22:14 EDT
Any news on this issue? As it stands, cobbler is unusable.
Comment 2 George 2012-09-11 02:08:16 EDT
Bump.

I'm also getting this problem, official support tell me to turn off selinux which will only confirm the problem.  No official workarounds or fixes suggested.
Comment 3 Jan Pazdziora 2012-09-11 02:18:09 EDT
(In reply to comment #2)
> 
> I'm also getting this problem, official support tell me to turn off selinux

Out of curiosity -- what official support did tell you that?
Comment 4 Pablo Iranzo Gómez 2012-10-09 10:34:38 EDT
Jan: https://github.com/cobbler/cobbler/wiki/Selinux 

On F16, and RHEL 6.x "turn off SELinux".

AFAIK, it tries to access /dev/shm


Oct  9 16:32:25 Merak setroubleshoot: SELinux is preventing /usr/bin/python from execute access on the file /tmp/ffiN8UuK3 (deleted). For complete SELinux messages. run sealert -l 69b08e98-dccb-4d1b-b36a-abd2cc57e73a

Oct  9 16:33:40 Merak setroubleshoot: SELinux is preventing /usr/bin/python from search access on the directory /dev/shm. For complete SELinux messages. run sealert -l 11cc91de-8be6-43ff-aa8e-e650d829f7a1

[root@Merak ~]# LANG=C sealert  -l 11cc91de-8be6-43ff-aa8e-e650d829f7a1
SELinux is preventing /usr/bin/python from search access on the directory /dev/shm.

*****  Plugin catchall (100. confidence) suggests  ***************************

If you believe that python should be allowed search access on the shm directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep cobblerd /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp
Comment 5 Pablo Iranzo Gómez 2012-10-09 10:35:37 EDT
(In reply to comment #4)
And my releases:

cobbler-2.2.3-2.el6.noarch
selinux-policy-targeted-3.7.19-155.el6_3.4.noarch

Running on EL 6.3+ updates on x86_64
Comment 6 Jan Pazdziora 2012-10-13 08:08:14 EDT
(In reply to comment #4)
> Jan: https://github.com/cobbler/cobbler/wiki/Selinux 
> 
> On F16, and RHEL 6.x "turn off SELinux".

Well, the upstream does not care about SELinux. While sad, it's understandable.

We however do distribution based on upstream projects and have maintainers for the very reason of integrating the upstream software to Fedora or EPEL.

> Oct  9 16:32:25 Merak setroubleshoot: SELinux is preventing /usr/bin/python from execute access on the file /tmp/ffiN8UuK3 (deleted). For complete SELinux messages. run sealert -l 69b08e98-dccb-4d1b-b36a-abd2cc57e73

This is however a different message than the original search issue, isn't it? What is the raw AVC denial?
Comment 7 James C. 2012-10-13 10:04:57 EDT
> (In reply to comment #4)
> > Jan: https://github.com/cobbler/cobbler/wiki/Selinux 
> > 
> > On F16, and RHEL 6.x "turn off SELinux".
> 
> Well, the upstream does not care about SELinux. While sad, it's
> understandable.

That is not correct. The wiki has been deprecated (we need to make that more clear), and this is the manual page for SELinux on 2.2.3:

http://cobbler.github.com/manuals/2.2.3/5/2_-_SELinux.html


Note the "Fedora 16 / RHEL6 / CentOS6 - Python MemoryError" section at the bottom, which is the issue above.

What we DO recommend is that you disable SELinux unless you are comfortable writing policy. This is due to the fact that when cobbler includes a new feature it will most likely break with SELinux enabled.
Comment 8 Jan Pazdziora 2012-10-13 12:37:54 EDT
(In reply to comment #7)
> > 
> > Well, the upstream does not care about SELinux. While sad, it's
> > understandable.
> 
> That is not correct. The wiki has been deprecated (we need to make that more
> clear), and this is the manual page for SELinux on 2.2.3:
> 
> http://cobbler.github.com/manuals/2.2.3/5/2_-_SELinux.html
> 
> Note the "Fedora 16 / RHEL6 / CentOS6 - Python MemoryError" section at the
> bottom, which is the issue above.
> 
> What we DO recommend is that you disable SELinux unless you are comfortable
> writing policy. This is due to the fact that when cobbler includes a new
> feature it will most likely break with SELinux enabled.

Nothing of which applies because we expect the package maintainer to integrate with other parts of the distribution *and test* before moving to the next upstream version (which in EPEL should only happen very rarely anyway). Noone should be required to write any policy code when using packages from Fedora, RHEL, or EPEL.

Which brings us to: are you planning on fixing this bug (three months old, now)?
Comment 9 James C. 2012-10-13 14:15:32 EDT
As far as I can tell, this isn't a bug with cobbler, it has to do with SELinux blocking python ctypes from executing properly, note the last line of the trace:

File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>

The workarounds I mention in the manual page above are to get around that until the underlying issue is fixed, which is not cobbler's issue.

That being said, cobbler does not ship with SELinux policy files, as it is included in fedora's SELinux policy because of things like spacewalk:

# rpm -qf /etc/selinux/targeted/modules/active/modules/cobbler.pp 
selinux-policy-targeted-3.10.0-153.fc17.noarch

Nor do I intend to ship supplementary policy files, because they can cludge up the system. Instead, if I find something that does break the policy I will create tickets to have the base policy fixed. This leads to a lag in time before the fix is officially available, for which I will make notes in the manual page (as I did above). The percentage of cobbler users running with SELinux enabled is relatively small, so I feel this works for now. Note that I do not ship releases with known policy issues, but they do pop up.

If you don't care for that policy and it's important to you, how about submitting some patches to either Dan/Fedora or myself to fix the issue instead?
Comment 10 Jan Pazdziora 2012-10-15 04:25:45 EDT
Clearly our views are completely different. I filed a FESCo ticket to help us find some setup that will work both for you as the package maintainer, and for cobbler users as well.

https://fedorahosted.org/fesco/ticket/959
Comment 11 Toshio Ernie Kuratomi 2012-10-15 13:34:48 EDT
Regarding the bug itself, it looks like this bug in python:
https://bugzilla.redhat.com/show_bug.cgi?id=814391

(Looks fixed in F17 but not in RHEL6).

How to fix bugs in packages in epel is probably something that should be discussed on the epel mailing list.  I'm just one contributor but it seems like past policy has been that we don't control RHEL so we should be fixing our packages to work with RHEL as it exists.  If I'm understanding the manual page link correctly, that might mean shipping our cobbler package without authn_pam.py.  Or alternately, patching authn_pam.py to catch the error and disable that feature (and logging why the feature isn't working properly).
Comment 12 Kevin Fenzi 2012-10-17 13:30:50 EDT
FWIW, I have found RHEL selinux-policy maintainers very willing to work with EPEL maintainers to adjust policy. has anyone filed a selinux-policy bug on this? Or asked them for comment?

I think short term adjusting to work with existing RHEL packages is good, and then longer term asking for fixes in python/selinux-policy is the way to go. You can then remove the workarounds once policy/python is fixed?
Comment 13 Toshio Ernie Kuratomi 2012-10-17 15:59:41 EDT
Note: For SELinux to work around this, I think it would need to make it so /usr/bin/python can execute files in tmpdirs.  /usr/bin/python doesn't have any special contexts (It's bin_t) so it'd likely mean adding a sebool to allow basically anything to execute files in tmpdirs
Comment 14 Jan Pazdziora 2012-10-23 07:34:28 EDT
(In reply to comment #13)
> Note: For SELinux to work around this, I think it would need to make it so
> /usr/bin/python can execute files in tmpdirs.  /usr/bin/python doesn't have

What kind of files are they that need to be executed in tmpdirs? Cannot cobbler create them in some place special where they could have well defined label?
Comment 15 James C. 2012-10-23 07:54:21 EDT
Jan, yet again I must point out that the issue is not Cobbler, it is python. Cobbler is not creating these files - the ctypes library is, thus the issue needs to be fixed in the policy for ctypes as Toshio pointed out. For example, there is already a boolean for httpd to do this (httpd_tmp_exec).
Comment 16 Jan Pazdziora 2012-10-23 08:22:35 EDT
(In reply to comment #15)
> Jan, yet again I must point out that the issue is not Cobbler, it is python.
> Cobbler is not creating these files - the ctypes library is, thus the issue
> needs to be fixed in the policy for ctypes as Toshio pointed out. For
> example, there is already a boolean for httpd to do this (httpd_tmp_exec).

Fair enough. The issues is still with cobbler which is shipping authn_pam.py (IIUIC) without having the ctypes issue cleared / resolved with selinux-policy maintainers first. Can you respin cobbler without authn_pam.py?
Comment 17 Dominik L. Borkowski 2012-10-23 08:31:00 EDT
Coming from an end-user perspective, shouldn't issues that affect cobbler be reported further up by the package maintainers?

Unless I'm mistaken, currently users are faced with cobbler not working out-of-the-box. Reporting the issue as cobbler related would be the appropriate begin of whatever chain of events may have to occur. Expecting end-users to track down what underlying problem involves and trying to address it is a bit of a stretch. 

If the expectation to have cobbler package maintainers report this issue further up the chain, perhaps adding documentation about having to disable SELINUX or whatever work-around is deemed appropriate would be of use. As noted already, end-users of this package are in a bit of a pickle.

Cheers!
Comment 18 Toshio Ernie Kuratomi 2012-10-23 12:34:09 EDT
I believe that cobbler upstream is aware of the issue as they are the ones who wrote the analysis and workarounds on the online page.  The python maintainer is also aware of the issue.

I just pinged him about this and he took a look.  Bug #814391 currently has status ON_QA which means the fix (along with other changes to the python package for the next RHEL6 point release) is currently in testing.  If all goes well, that should be in the next RHEL point release.

Someone should test whether that is enough to fix this problem, though.  He says that this does fix some uses of ctypes but not all.  In particular "this also assume that cobbler doesn't need to create callbacks from C code to Python.  If it does, then 814391 won't help".
Comment 19 Toshio Ernie Kuratomi 2012-10-23 13:30:33 EDT
Created attachment 632243 [details]
Workaround ctypes and SELinux incompatibility

Here's a cobbler patch.  I think it could be proposed upstream, and possibly accepted with some cleanups.  It checks whether ctypes can be imported and if so, it lets you use authn_pam.  If not, it prints a warning to stderr and disables that module.

Built here if someone would like to test it (I don't use cobbler myself, I'm just a python guy).

http://koji.fedoraproject.org/koji/taskinfo?taskID=4619186
Comment 20 Toshio Ernie Kuratomi 2012-10-23 16:54:11 EDT
Few updates:
* this also affects cobbler on F16.
* On F16, the change to the python package in bug #814391 is enough to make cobbler functional
* On a RHEL6 test box, the change to the python package was not enough.  We get past the traceback listed but then get this:

Starting cobbler daemon: Traceback (most recent call last):
  File "/usr/bin/cobblerd", line 76, in main
    api = cobbler_api.BootAPI(is_cobblerd=True)
  File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 127, in __init__
    module_loader.load_modules()
  File "/usr/lib/python2.6/site-packages/cobbler/module_loader.py", line 62, in load_modules
    blip =  __import__("modules.%s" % ( modname), globals(), locals(), [modname])
  File "/usr/lib/python2.6/site-packages/cobbler/modules/authn_pam.py", line 121, in <module>
    PAM_START = LIBPAM.pam_start
  File "/usr/lib64/python2.6/ctypes/__init__.py", line 366, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib64/python2.6/ctypes/__init__.py", line 371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/bin/python: undefined symbol: pam_start

I also get this traceback if I run audit2allow to create a new policy and load that.  I even generated the policy after disabling dontaudit (semodule -DB).  Only setting SELinux to permissive gets around that traceback.  I think this will require a RHEL SELinux person to diagnose.

* I think I was wrong about being able to transition from the python interpreter's type to a cobbler type.  cobbler is started by the init system and I believe the init system has the capability to make transitions.  So policy can be written to target cobbler specifically.

* After looking at the cobbler code some more I'm unsure why we don't catch exceptions from attempting to load any module... if there's an exception for any module at loadtime it seems like we would want logging of the error but not to crash the server.  I can easily implement a patch that does that if you think that would fly upstream.
Comment 21 Toshio Ernie Kuratomi 2012-10-24 02:35:56 EDT
After some IRC discussion in #cobbler, I sent this pull request: https://github.com/cobbler/cobbler/pull/342

It will have the same effect as attachment 632243 [details] but is more generic: it will keep exceptions from crashing cobbler when loading any module.
Comment 22 Orion Poplawski 2014-04-21 14:42:40 EDT
I suspect this is fixed in cobbler-2.4.0-1.el6, please re-open if not.  Will be pushing 2.4.3 out soon too.
Comment 23 Jan Hutař 2014-04-28 02:46:20 EDT
Indeed, seems to be fixed (checked reproducer from initial comment in this BZ) in:

# rpm -q cobbler selinux-policy-targeted
cobbler-2.4.0-1.el6.noarch
selinux-policy-targeted-3.7.19-231.el6_5.1.noarch

Note You need to log in before you can comment on or make changes to this bug.