Description of problem: Let's try and properly migrate tomcat to native systemd units the tomcat.service is just an ugly hack. I propose that we split tomcat startup into 4 units... 1 tomcat.service 2 tomcat-security.service 3 tomcat@.service ( For multiple instances ) 4.tomcat-security@.service ( For multiple instance ) Have all the required environmental parameters in one configuration file only ( /etc/tomcat/tomcat.conf ) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I mean The tomcat service being shipped is just an ugly hack
What do you mean as tomcat-security.service ? Michal Vyskocil from SUSE team have made some commitments on tomcat systemd support... I'am going to merge them and package 7.0.29 release.
(In reply to comment #2) > What do you mean as tomcat-security.service ? elif [ "$1" = "start-security" ]; then ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \ -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ -Djava.security.manager \ -Djava.security.policy=="${CATALINA_BASE}/conf/catalina.policy" \ -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \ -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \ org.apache.catalina.startup.Bootstrap start \ >> ${CATALINA_BASE}/logs/catalina.out 2>&1 & if [ ! -z "$CATALINA_PID" ]; then echo $! > $CATALINA_PID > > Michal Vyskocil from SUSE team have made some commitments on tomcat systemd > support... I'am going to merge them and package 7.0.29 release. Got a link to those commitments so I can review it? There are being made and have been made some changes to enhance multiple instance support via templates which he might not have caught up to yet unless "frozcat" has been assisting overseeing his change set
" First of all, for service type forking, you have to specify PIDFile=, unless systemctl status reports the invalid state, when daemon is running. However, even if you use the PIDFile, you will get state failed on systemctl stop because jvm ends with code 143. Because of that, I've used a different approach to yours - please consider if that's usable for you or not I've removed a lot of bloat, which is not needed under systemd - like you don't need to source /etc/tomcat/tomcat.conf as EnvironmentFile=/etc/tomcat/tomcat.conf do the same for you. Or the ugly part with nohup su tomcat -c, for which you have to call parseArguments and export the environment for /usr/sbin/tomcat is useless, becase of User=/Group= For that reason I've simplified the structure of scripts and removed all crap needed for sysvinit. You can take a look at Java:packages/tomcat [1] - the most important files are tomcat-7.0-tomcat-sysd [2], tomcat-7.0.service [3] and tomcat-7.0-jsvc.service [4]. For jsvc, I've merged the separate script into one, to reuse most of the parts. But this part should get an another care - I would say -jsvc.service can be Type=forking with appropriate PIDFile=, but I did not want to make the differences between those two. As an alternative, tomcat.service started with CapabilityBoundingSet=CAP_NET_BIND_SERVICE can bind to port 80 and run as tomcat user, but this will need to be checked first. But in case it's true, this can make the -jsvc service and package obsoleted. [1] https://build.opensuse.org/package/files?package=tomcat&project=Java%3Apackages [2] https://build.opensuse.org/package/view_file?file=tomcat-7.0-tomcat-sysd&package=tomcat&project=Java%3Apackages&rev=84c6520bca123c6160be1ec74aa42af9 [3] https://build.opensuse.org/package/view_file?file=tomcat-7.0.service&package=tomcat&project=Java%3Apackages&rev=84c6520bca123c6160be1ec74aa42af9 [4] https://build.opensuse.org/package/view_file?file=tomcat-7.0-jsvc.service&package=tomcat&project=Java%3Apackages&rev=84c6520bca123c6160be1ec74aa42af9 "
as suspected this is no better than we currently have... I'm going to see if I cant come up with something better then this and we want to use separated tomcat config files along with systemd template files for multiple instances.
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle. Changing version to '19'. (As we did not run this process for some time, it could affect also pre-Fedora 19 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Ok, it's fairly simple to do this: [Service] Type=simple Environment="NAME=%I" ExecStart=/usr/sbin/tomcat start-new ExecStop=/usr/sbin/tomcat stop SuccessExitStatus=143 User=tomcat Group=tomcat and then if [ "$1" = "start-new" ]; then exec ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \ -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \ -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \ org.apache.catalina.startup.Bootstrap start also fixing stop elif [ "$1" = "stop" ]; then exec ${JAVACMD} $JAVA_OPTS \ -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ org.apache.catalina.startup.Bootstrap stop the real problem is that everything is in the server.xml, and in the rest of the /etc/tomcat as well I'm not very familiar with tomcat and to me it looks like duplicating the entire /etc/tomcat and then /usr/share/tomcat as well (in which frankly there's only one non-symlink) alternatively, systemd fs namespaces instead of symlinks, map different /etc/tomcatX to /usr/share/tomcat/conf
That's not properly migrating this... " ExecStart=/usr/sbin/tomcat start-new <--- ExecStop=/usr/sbin/tomcat stop <--- " The scripts that start and stop the tomcat instance ( like catalina.sh ) should be migrated to systemd... Calling the old initscripts or otherwise using "scripts" is not properly migrating this to native systemd unit start where the unit(s) would obsolete/replace the function the script
Well, the thing is there's a lot of shell magic that sets up environment variables and common pattern to solve it is to drop a small wrapper for it. "start-new" is just a hack, I've replaced it with start now. In fact, this is my current setup: [Service] Type=simple EnvironmentFile=/etc/tomcat/tomcat.conf EnvironmentFile=-/etc/sysconfig/tomcat@%I Environment="NAME=%I" ExecStart=/usr/sbin/tomcat start ExecStop=/usr/sbin/tomcat stop SuccessExitStatus=143 User=tomcat Group=tomcat tomcat.conf is severely reduced, if you don't drop sysconfig/tomcat@NAME it means that catalina.base is /var/lib/tomcats/NAME (while .home is still /usr/share/tomcat, so binaries and libraries are from there) execstop is needed because this thing shuts down through the message on shutdown port, but then we still have systemd kill whatever is accidentally left. In the end I think the best option is still to have a wrapper script but it's much better than what it was before I started. I'll post the scripts and stuff shortly.
I have posted a git tree with changes to Ivan. There's still jsvc question, which I'll try solving in the next pull.
The changes that are suggested here (and which presumably have been checked into f20) break Dogtag installation and as a result IPA. When a dogtag instance is created, we create a clone of the tomcat service as described in the description below (from /usr/sbin/tomcat-sysd) # This script provides systemd activation of the tomcat service # To create clones of this service: # 1) SERVICE_NAME must be defined before calling this script # 2) Create /etc/sysconfig/${SERVICE_NAME} from /etc/sysconfig/tomcat # to override tomcat defaults # SERVICE_NAME is a required value only if the service name is # different from 'tomcat' # In the file /etc/sysconfig/${SERVICE_NAME}, we override a bunch of defaults including CATALINA_BASE, CATALINA_TMPDIR, TOMCAT_LOG, JAVA_OPTS etc. I would strongly prefer not to change this for Fedora 19/18. The changes you are envisioning will break all existing IPA and dogtag installs out there, unless you leave in place the existing scripts for backward compatibility. I am open to changing how we configure things on F20 - but I need a clear explanation of exactly how things need to be changed.
afaik my proposed changes have not been implemented...
OK great -- so I am trying out tomcat-7.0.42-3 on fedora 20. Not sure what has been implemented and what has not. As I understand it, I need to do the following: If my instance name is tomcat28: 1. Create a file /etc/sysconfig/tomcat@tomcat28 with my overrides. Not that in this script, I do override CATALINA_BASE. 2. From my init scripts, call /usr/sbin/tomcat start tomcat28 When I do (2), I see that the init script attempts to run systemctl start tomcat The config file in step 1 appears to be sourced and I can see some of my overrides. One of the overrides though, is TOMCAT_USER, which appears not to be working as I get errors like: Sep 10 04:10:24 localhost server: java.io.FileNotFoundException: /var/lib/pki/tomcat35/conf/server.xml (Permission denied) Secondly, if I select a service name like pk-tomcat28 - then it appears that the name is decomposed into pki/tomcat28. This will definitely not work for us. The default instance name we use is pki-tomcat.
User/Group is set by systemd and should be overriden in .service as described https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F It is recomended way according to https://fedoraproject.org/wiki/Packaging:Systemd#EnvironmentFiles_and_support_for_.2Fetc.2Fsysconfig_files
Do you have a specific need to put tomcat in /var/lib/pki/tomcat35? Because how it's used by default is - if you don't create any file in sysconfig and just start an instance tomcat@name is it'll set catalina_base to /var/lib/tomcats/name. However if you override you can just CATALINA_BASE=/var/lib/pki/tomcat35. Now, configs. As we run separate instances and they need separate configs, you need conf directory and in it server.xml and some other configs, look at /usr/libexec/tomcat/{preamble,server} to see how it loads them from /etc/tomcat vs. CATALINA_BASE/conf. Ah, you also need to create conf,logs,webapps,work in your instance CATALINA_BASE I understand this is poorly documented, but I guess we can fix it together; I was trying to solve an immediate problem of running this thing and didn't care that much about documentation. I guess we can add some tools like tomcat-create-instance that will do all the directory creation/file copying and also make it work if you just start a named service - to create missing directories. Johann: I've implemented your proposed changes. So tomcat is not a forking service - it's a simple service now. We still need to have ExecStop because we're stopping it properly, by telling tomcat to shutdown instead of just killing it.
systemd instance names seems to be escaped. '-' is used to be able to use '/' in .service filename which is restricted. '-' is converted to '/' '\x2d' is converted to '-' http://www.freedesktop.org/software/systemd/man/systemd.unit.html "Basically, given a path, "/" is replaced by "-", and all unprintable characters and the "-" are replaced by C-style "\x20" escapes."
OK, after a lot of experimentation, I've figured out the best and simplest option for me, which is to simply modify my own unit files (the unit files for pki-tomcat@.service) to be similar to what you currently have. This way, I will not have to create any new unit or config files. For reference, my /lib/systemd/system/pki-tomcatd@.service file will look like: [Unit] Description=PKI Tomcat Server %i After=pki-tomcatd.target syslog.target network.target PartOf=pki-tomcatd.target [Service] Type=simple EnvironmentFile=/etc/tomcat/tomcat.conf Environment="NAME=%i" EnvironmentFile=-/etc/sysconfig/%i ExecStart=/usr/libexec/tomcat/server start ExecStop=/usr/libexec/tomcat/server stop SuccessExitStatus=143 User=pkiuser Group=pkiuser [Install] WantedBy=multi-user.target A couple changes you might want to consider: 1. Adding an identifier %i to the description, so you know which unit file has been executed. 2. Using %i instead of %I for the NAME variable. This will allow names with a dash in them to work correctly. Names as a path with relevant substitutions make no sense for instance names. One more thing. When I use this unit file (or even the tomcat@.service file you propose), all my startup output ends up going to /var/log/messages instead of catalina.out. How do I fix that?
Any details on how to fix where the server output is going? Right now, its going into /var/log/messages instead of catalina.out.
I didn't break it by accident - I thought that having just > catalina.out isn't really manageable, as you can't really rotate it, so I directed all output to journal so you can do systemctl status tomcat@foo and see it. Or journalctl -u tomcat@foo. If you want logging to a specific file here, it might be overall better way to set it up through a syslog. If you really want it to be old-school though, we could change /usr/libexec/tomcat/server to allow it to be configured. (but I like the journal approach better)
We can leave it as journal for now, although I reserve the right to ask for it to be changed later. I did notice a typo in the /usr/libexec/tomcat/server -Djava.security.policy==\"${CATALINA_BASE}/conf/catalina.policy\"" Notice the ==. Should be =. With ==, the policy is not actually loaded. You can see this by appending -Djava.security.debug=all to the JAVA_OPTS. With =, the policies actually load. Notwithstanding, though, when you try to use the security manager, you get the following exception: access: access denied ("java.util.PropertyPermission" "java.util.logging.config.class" "read") Not sure why that is - considering that the rule appears to be in the policy. This could be a separate bug, but it did not occur for previous versions of tomcat.
Any comments on the security manager issues above? We'd like to get these resolved before f20 release if possible.
Sorry for the delay, I'm trying to deal with too many things at once; it'll be nice though if Ivan will help with at least the type you've found :)
Sorry for delay I thought you was loading your custom security manager. "-Djava.security.policy==" could be found in tomcat6 too. http://pkgs.fedoraproject.org/cgit/tomcat6.git/tree/tomcat6-6.0.wrapper?h=f17 Merging Paul's changes I saw "==", but when I find it in old scripts I decided that it is correct. I'll check upstrem scripts, how it is loaded there.
I've checked upstrem catalina.sh and there is also -Djava.security.policy==\"$CATALINA_BASE/conf/catalina.policy\" \ You can find docs about loading security policies here http://docs.oracle.com/javase/7/docs/technotes/guides/security/PolicyFiles.html " (note the double equals) then just the specified policy file will be used; all the ones indicated in the security properties file will be ignored. "
Using the double equals is fine, but the line is still incorrect because the entry is supposed to be a URL. From the link you pointed to above: java -Djava.security.manager -Djava.security.policy==someURL SomeApp As the code currently stands, a MalFormedURLException (no protocol) is thrown when the security manager attempts to load the policy. If the following is used in /usr/libexec/tomcat/server instead, then the policy is actually loaded correctly: if [ "${SECURITY_MANAGER}" = "true" ] ; then OPTIONS="${OPTIONS} \ -Djava.security.manager \ -Djava.security.policy==file:${CATALINA_BASE}/conf/catalina.policy" fi Note the removal of the quotes and the addition of file:
Any update on this?
Upstream catalina.sh has -Djava.security.policy=="$CATALINA_BASE"/conf/catalina.policy \ I've searched the Internet and didn't find any bugreports for this issue in catalina.sh
Ivan, Have you tried to run a basic tomcat app with security manager enabled? My guess is that you will get policy related exceptions, and if you turn on -Djava.security.debug=all (or maybe even if you do not), you will see the MalFormedURLException I mention above. I'm not sure whats going on upstream, but in this environment, security manager just does not work with this init script.
Ivan, It turns out that the file: part is not necessary. The following also works: -Djava.security.policy==${CATALINA_BASE}/conf/catalina.policy" compared to what is currently there: -Djava.security.policy==\"${CATALINA_BASE}/conf/catalina.policy\"" You can easily see this bug. You dont even need a webapp. Simply enable security manager on the default tomcat instance and restart the instance. I did this by adding the following to /etc/sysconfig/tomcat SECURITY_MANAGER="true" JAVA_OPTS="-Djava.security.debug=all" and then restarting tomcat : systemctl start tomcat.service The tomcat service FAILS TO COME UP and there is an error in the log indicating a Malformed URL for the policy. So, upstream is right -- what is wrong is the invocation in this script. This is a clear and simply reproducible bug. Its also going to cause all kinds of problems for IPA and Dogtag users when they upgrade to Fedora 20. We need a fix for this ASAP.
Oh Thanks a lot for finding real root cause. I'll fix it within 7.0.47 update. Current ETA is this weekend.
I've found the issue. In upstream's catalina.sh tomcat is running via eval which seems to remove quotation marks.
tomcat-7.0.47-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/tomcat-7.0.47-1.fc20
Package tomcat-7.0.47-1.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing tomcat-7.0.47-1.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-20684/tomcat-7.0.47-1.fc20 then log in and leave karma (feedback).
tomcat-7.0.47-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.