Description of problem: Calamari currently does not support SELinux and we advise people to turn it off in order to get Calamari to run. Version-Release number of selected component (if applicable): calamari-server-1.2.3-5.el7cp calamari-server-1.2.3-5.el6cp How reproducible: Always Steps to Reproduce: 1. Install calamari with SELinux in Permissive mode 2. Try to connect to the web server 3. Actual results: 500: Internal Server Error Expected results: Everything works fine. Additional info: I have already been able to make calamari support SELinux in rhel 7 and the code is in a private branch private-branto-wip-rhel-7-selinux of calamari-server dist-git repo. The solution is to load a custom SELinux policy module for calamari-server and enable it in post script together with two SELinux booleans -- the patch does not touch any actual calamari-server source code.
I really think that we should make this happen for 1.2.3 release. Telling customers to turn SELinux off to run a service/app is rather hypocritical from Red Hat... btw: Test el7 scratch build with fixed SELinux issues: https://brewweb.devel.redhat.com/taskinfo?taskID=8702541
CC'ing Dan Mick @Dan: Was there a reason why you didn't run calamari-ctl initialize in post script? Based on my testing it worked just fine. The url to private branch with the changes: http://pkgs.devel.redhat.com/cgit/rpms/calamari-server/tree/?h=private-branto-wip-rhel-7-selinux
It requires user interaction, and enables services, both of which are inappropriate automatic package behavior. But what has that got to do with SELinux?
I had to restart httpd and probably also salt-master after I ran calamari-ctl to make this work. Otherwise I got errors from httpd about missing private key. Hmm, right, I almost forgot that it queries for a username, e-mail and a password. Maybe, we could setup a default username and password and then tell the users to change it?
Fedora policy absolutely prohibits starting services as a result of package install, and in general I think it's a good idea to avoid. It's a separate step on purpose.
Hmm, I was looking at the code and it looks like the calamari-ctl should probably restart the services on its own but the problem is probably related to this line in /opt/calamari/salt-local/services.sls: {% if grains['os'] == 'RedHat' and grains['osrelease'] == '7.0' %} I was running this on 7.1 pre-release so it probably fallbacked to upstart and did not actually restart the services. If that is indeed the case then there is no need to run calamari-ctl in post script. I'm not sure what syntax do the sls files support in {% %} but it looks like python. Can we use grains['osrelease'].startswith('7.') there instead? btw: The same goes for postgres.sls.
Good point. I suspect there are a number of places to check that we're not testing for 7.0 exactly.
Absolutely not a blocker for 1.2.3. We don't have SELinux support for Ceph so not having it for Calamari is fine.
Yeah, I'm not saying it is supposed to be a blocker but I think that if we can get this to work in time then we should include the fix into the release. btw: What issues does Ceph have with SELinux? AFAIK, Dan already fixed the execstack issue although I'm not sure if it made it into 0.80.8 release.
(In reply to Boris Ranto from comment #9) > btw: What issues does Ceph have with SELinux? AFAIK, Dan already fixed the > execstack issue although I'm not sure if it made it into 0.80.8 release. I verified that it did get merged in time for 0.80.8. It's this commit on upstream's firefly branch: 01faf1356f648ded9acda02e7cc67c1adb9e9ee3 I don't know if this is the only SELinux issue with Ceph itself. Upstream has never run tests with SELinux enabled as far as I know. It sounds like something we'll want to fix in 1.3.
I've played with this a bit more and my issues were not caused by SELinux. The issue was that calamari-ctl did not try to restart the services but only issued systemctl start <service> which (if the process is already running) does precisely nothing. After I patched the .sls files to call systemctl restart everything went as expected so rhel 7 Calamari SELinux support is ready. btw: The syntax for conditions in sls files is indeed pythonic and .startswith() works just fine there. We can track that issue in a separate bz. btw2: Yeah, I checked the git logs as well and the execstack patch is in fact in 0.80.8 release so Ceph should play nicely with SELinux in Red Velvet. I am not aware of any other problems with SELinux regarding Ceph. It would be nice if this got properly tested though at least for 1.3 release.
I will be absolutely stunned if we've identified all the SELinux pitfalls in Calamari and Ceph so far. There are lots of communications from cluster to server as well (Diamond and Salt) and lots of assumptions about root being permitted to write files....I wouldn't call this done yet.
@Dan: agreed. I'd like to have "SELinux support" be well-tested upstream before we ship documentation for Red Velvet that claims that it works downstream. I've filed http://tracker.ceph.com/issues/10804 upstream so that the upstream QA team and Neil can prioritize this.
Actually, it is not that difficult to make your app work with SELinux. Especially if you are willing to ship your own custom SELinux policy. It would be nice if a QE could run the tests for Ceph with SELinux in Permissive mode + auditing turned on and then attach /var/log/audit/audit.log here. You can generate the custom SELinux policy directly from the log. That is pretty much what I did to make Calamari run with SELinux (well, that + I had to turn two SELinux booleans on and patch the spec files to install and load the policy). SELinux people made it this easy because they were very aware of the attitude amongst developers about it. Anyway, the policy on rhel 7 that I generated with that approach is pretty simple: Allow httpd to rename log files, allow httpd to open generic log files and allow httpd to name_connect to a tcp socket. With just that I was able to monitor a ceph cluster and attach nodes to the monitor without any issue. Oh, and I do not suggest to delete the remark about SELinux in the docs but it could be changed to something like: SELinux support is currently experimental, please try disabling it if you encounter any issues.
Regardless, I want to do this with full testing. There are edge cases, and diagnosing these faults in the field is far more than painful.
This bz is to allow an elementary SELinux support for calamari, not the full support. The patch does _not_ touch the source code in any way, all it does is that it allows httpd daemon to do a bit more so that users/customers will not have to run the server with SELinux disabled all the time.
This should be aimed at 1.3.0, too.
1.3.x feature: https://trello.com/c/ChnCMlTg This is a Z-stream candidate, but happy to see it in Stockwell.
Aligning the target field with the ceph-1.3.0 field. If this doesn't get fully ack'd for 1.3.0, we might need to kick this to 1.3.1. Have all the patches been submitted upstream?
We didn't get qa_ack; moving to 1.3.z.
*** Bug 1217511 has been marked as a duplicate of this bug. ***
btw: The initial support for this landed upstream: https://github.com/ceph/calamari/pull/287 However, the patches have not been tested, yet.
TODO create the policy and ship as part of calamari-server package
Gregory, Will this be fixed in 1.3.2.
This will be in 1.3.2 although it's unlikely to be in the initial spin. I will have it in the first spin post shutdown
Gregory committed to a delivery by 1/25 for this (thanks Gregory!), punting the other outstanding items.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:0313