Bug 842346 - Properly migrate tomcat to systemd
Summary: Properly migrate tomcat to systemd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: tomcat
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ivan Afonichev
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-07-23 14:31 UTC by Jóhann B. Guðmundsson
Modified: 2015-03-09 18:29 UTC (History)
9 users (show)

Fixed In Version: tomcat-7.0.47-1.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-11-10 07:20:34 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jóhann B. Guðmundsson 2012-07-23 14:31:28 UTC
Description of problem:

Let's try and properly migrate tomcat to native systemd units the tomcat.service is just an ugly hack. 

I propose that we split tomcat startup into 4 units... 

1 tomcat.service 
2 tomcat-security.service
3 tomcat@.service ( For multiple instances )
4.tomcat-security@.service ( For multiple instance )

Have all the required environmental parameters in one configuration file only 
( /etc/tomcat/tomcat.conf )


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Jóhann B. Guðmundsson 2012-07-23 14:32:05 UTC
I mean The tomcat service being shipped is just an ugly hack

Comment 2 Ivan Afonichev 2012-07-23 14:41:06 UTC
What do you mean as tomcat-security.service ?

Michal Vyskocil from SUSE team have made some commitments on tomcat systemd support... I'am going to merge them and package 7.0.29 release.

Comment 3 Jóhann B. Guðmundsson 2012-07-23 15:15:44 UTC
(In reply to comment #2)
> What do you mean as tomcat-security.service ?

elif [ "$1" = "start-security" ]; then
  ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    -Djava.security.manager \
    -Djava.security.policy=="${CATALINA_BASE}/conf/catalina.policy" \
    -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
    -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
    org.apache.catalina.startup.Bootstrap start \
    >> ${CATALINA_BASE}/logs/catalina.out 2>&1 &
    if [ ! -z "$CATALINA_PID" ]; then
      echo $! > $CATALINA_PID


> 
> Michal Vyskocil from SUSE team have made some commitments on tomcat systemd
> support... I'am going to merge them and package 7.0.29 release.

Got a link to those commitments so I can review it? 

There are being made and have been made some changes to enhance multiple instance support via templates which he might not have caught up to yet unless "frozcat" has been assisting overseeing his change set

Comment 4 Ivan Afonichev 2012-07-23 16:05:37 UTC
"
First of all, for service type forking, you have to specify PIDFile=,
unless systemctl status reports the invalid state, when daemon is
running. However, even if you use the PIDFile, you will get state failed
on systemctl stop because jvm ends with code 143. Because of that, I've
used a different approach to yours - please consider if that's usable
for you or not

I've removed a lot of bloat, which is not needed under systemd - like
you don't need to source /etc/tomcat/tomcat.conf as
EnvironmentFile=/etc/tomcat/tomcat.conf do the same for you. Or the ugly
part with nohup su tomcat -c, for which you have to call parseArguments
and export the environment for /usr/sbin/tomcat is useless, becase of
User=/Group=

For that reason I've simplified the structure of scripts and removed all
crap needed for sysvinit. You can take a look at Java:packages/tomcat
[1] - the most important files are tomcat-7.0-tomcat-sysd [2],
tomcat-7.0.service [3] and tomcat-7.0-jsvc.service [4].

For jsvc, I've merged the separate script into one, to reuse most of the
parts. But this part should get an another care - I would say
-jsvc.service can be Type=forking with appropriate PIDFile=, but I did
not want to make the differences between those two.

As an alternative, tomcat.service started with
CapabilityBoundingSet=CAP_NET_BIND_SERVICE can bind to port 80 and run
as tomcat user, but this will need to be checked first. But in case it's
true, this can make the -jsvc service and package obsoleted.

[1] https://build.opensuse.org/package/files?package=tomcat&project=Java%3Apackages
[2] https://build.opensuse.org/package/view_file?file=tomcat-7.0-tomcat-sysd&package=tomcat&project=Java%3Apackages&rev=84c6520bca123c6160be1ec74aa42af9
[3] https://build.opensuse.org/package/view_file?file=tomcat-7.0.service&package=tomcat&project=Java%3Apackages&rev=84c6520bca123c6160be1ec74aa42af9
[4] https://build.opensuse.org/package/view_file?file=tomcat-7.0-jsvc.service&package=tomcat&project=Java%3Apackages&rev=84c6520bca123c6160be1ec74aa42af9
"

Comment 5 Jóhann B. Guðmundsson 2012-07-23 16:30:11 UTC
as suspected this is no better than we currently have... 

I'm going to see if I cant come up with something better then this and we want to use separated tomcat config files along with systemd template files for multiple instances.

Comment 6 Fedora End Of Life 2013-04-03 15:04:51 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 7 Paul P Komkoff Jr 2013-06-10 15:26:36 UTC
Ok, it's fairly simple to do this:
[Service]
Type=simple
Environment="NAME=%I"
ExecStart=/usr/sbin/tomcat start-new
ExecStop=/usr/sbin/tomcat stop
SuccessExitStatus=143
User=tomcat
Group=tomcat

and then
if [ "$1" = "start-new" ]; then
  exec ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
    -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
    org.apache.catalina.startup.Bootstrap start

also fixing stop
elif [ "$1" = "stop" ]; then
  exec ${JAVACMD} $JAVA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    org.apache.catalina.startup.Bootstrap stop

the real problem is that everything is in the server.xml, and in the rest of the /etc/tomcat as well
I'm not very familiar with tomcat and to me it looks like duplicating the entire  /etc/tomcat and then /usr/share/tomcat as well (in which frankly there's only one non-symlink)

alternatively, systemd fs namespaces instead of symlinks, map different /etc/tomcatX to /usr/share/tomcat/conf

Comment 8 Jóhann B. Guðmundsson 2013-06-10 15:44:05 UTC
That's not properly migrating this...  

"
ExecStart=/usr/sbin/tomcat start-new <--- 
ExecStop=/usr/sbin/tomcat stop <--- 
"

The scripts that start and stop the tomcat instance ( like catalina.sh ) should be migrated to systemd...

Calling the old initscripts or otherwise using "scripts" is not properly migrating this to native systemd unit start where the unit(s) would obsolete/replace the function the script

Comment 9 Paul P Komkoff Jr 2013-06-10 21:43:59 UTC
Well, the thing is there's a lot of shell magic that sets up environment variables and common pattern to solve it is to drop a small wrapper for it.
"start-new" is just a hack, I've replaced it with start now.
In fact, this is my current setup:
[Service]
Type=simple
EnvironmentFile=/etc/tomcat/tomcat.conf
EnvironmentFile=-/etc/sysconfig/tomcat@%I
Environment="NAME=%I"
ExecStart=/usr/sbin/tomcat start
ExecStop=/usr/sbin/tomcat stop
SuccessExitStatus=143
User=tomcat
Group=tomcat

tomcat.conf is severely reduced, if you don't drop sysconfig/tomcat@NAME it means that catalina.base is /var/lib/tomcats/NAME (while .home is still /usr/share/tomcat, so binaries and libraries are from there)

execstop is needed because this thing shuts down through the message on shutdown port, but then we still have systemd kill whatever is accidentally left.

In the end I think the best option is still to have a wrapper script but it's much better than what it was before I started.

I'll post the scripts and stuff shortly.

Comment 10 Paul P Komkoff Jr 2013-06-12 22:44:17 UTC
I have posted a git tree with changes to Ivan. There's still jsvc question, which I'll try solving in the next pull.

Comment 11 Ade Lee 2013-09-10 16:10:47 UTC
The changes that are suggested here (and which presumably have been checked into f20) break Dogtag installation and as a result IPA.

When a dogtag instance is created, we create a clone of the tomcat service as described in the description below (from /usr/sbin/tomcat-sysd)

# This script provides systemd activation of the tomcat service
# To create clones of this service:
# 1) SERVICE_NAME must be defined before calling this script
# 2) Create /etc/sysconfig/${SERVICE_NAME} from /etc/sysconfig/tomcat
# to override tomcat defaults

# SERVICE_NAME is a required value only if the service name is
# different from 'tomcat'
#

In the file /etc/sysconfig/${SERVICE_NAME}, we override a bunch of defaults including CATALINA_BASE, CATALINA_TMPDIR, TOMCAT_LOG, JAVA_OPTS etc.

I would strongly prefer not to change this for Fedora 19/18.  The changes you are envisioning will break all existing IPA and dogtag installs out there, unless you leave in place the existing scripts for backward compatibility.

I am open to changing how we configure things on F20 - but I need a clear explanation of exactly how things need to be changed.

Comment 12 Jóhann B. Guðmundsson 2013-09-10 16:15:36 UTC
afaik my proposed changes have not been implemented...

Comment 13 Ade Lee 2013-09-10 20:26:33 UTC
OK great -- so I am trying out tomcat-7.0.42-3 on fedora 20.

Not sure what has been implemented and what has not.  As I understand it, I need to do the following:  

If my instance name is tomcat28:
1. Create a file /etc/sysconfig/tomcat@tomcat28
   with my overrides.  Not that in this script, I do override CATALINA_BASE.

2. From my init scripts, call  /usr/sbin/tomcat start tomcat28

When I do (2), I see that the init script attempts to run 

systemctl start tomcat

The config file in step 1 appears to be sourced and I can see some of my overrides.  One of the overrides though, is TOMCAT_USER, which appears not to be working as I get errors like:

Sep 10 04:10:24 localhost server: java.io.FileNotFoundException: /var/lib/pki/tomcat35/conf/server.xml (Permission denied)


Secondly, if I select a service name like pk-tomcat28 - then it appears that the name is decomposed into pki/tomcat28.  This will definitely not work for us.  The default instance name we use is pki-tomcat.

Comment 14 Ivan Afonichev 2013-09-10 20:50:42 UTC
User/Group is set by systemd and should be overriden in .service as described 
https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F
It is recomended way according to https://fedoraproject.org/wiki/Packaging:Systemd#EnvironmentFiles_and_support_for_.2Fetc.2Fsysconfig_files

Comment 15 Paul P Komkoff Jr 2013-09-10 21:01:11 UTC
Do you have a specific need to put tomcat in /var/lib/pki/tomcat35? Because how it's used by default is - if you don't create any file in sysconfig and just start an instance tomcat@name is it'll set catalina_base to /var/lib/tomcats/name.
However if you override you can just CATALINA_BASE=/var/lib/pki/tomcat35.
Now, configs. As we run separate instances and they need separate configs, you need conf directory and in it server.xml and some other configs, look at /usr/libexec/tomcat/{preamble,server} to see how it loads them from /etc/tomcat vs. CATALINA_BASE/conf.

Ah, you also need to create conf,logs,webapps,work in your instance CATALINA_BASE

I understand this is poorly documented, but I guess we can fix it together; I was trying to solve an immediate problem of running this thing and didn't care that much about documentation.
I guess we can add some tools like tomcat-create-instance that will do all the directory creation/file copying and also make it work if you just start a named service - to create missing directories.

Johann: I've implemented your proposed changes. So tomcat is not a forking service - it's a simple service now. We still need to have ExecStop because we're stopping it properly, by telling tomcat to shutdown instead of just killing it.

Comment 16 Ivan Afonichev 2013-09-10 21:16:40 UTC
systemd instance names seems to be escaped. '-' is used to be able to use '/' in .service filename which is restricted.
'-' is converted to '/'
'\x2d' is converted to '-'


http://www.freedesktop.org/software/systemd/man/systemd.unit.html
"Basically, given a path, "/" is replaced by "-", and all unprintable characters and the "-" are replaced by C-style "\x20" escapes."

Comment 17 Ade Lee 2013-09-12 02:24:42 UTC
OK, after a lot of experimentation, I've figured out the best and simplest option for me, which is to simply modify my own unit files (the unit files for pki-tomcat@.service) to be similar to what you currently have.  This way, I will not have to create any new unit or config files.

For reference, my /lib/systemd/system/pki-tomcatd@.service file will look like:

[Unit]
Description=PKI Tomcat Server %i
After=pki-tomcatd.target syslog.target network.target
PartOf=pki-tomcatd.target

[Service]
Type=simple
EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME=%i"
EnvironmentFile=-/etc/sysconfig/%i
ExecStart=/usr/libexec/tomcat/server start
ExecStop=/usr/libexec/tomcat/server stop
SuccessExitStatus=143
User=pkiuser
Group=pkiuser

[Install]
WantedBy=multi-user.target

A couple changes you might want to consider:
1. Adding an identifier %i to the description, so you know which unit file has been executed.

2. Using %i instead of %I for the NAME variable.  This will allow names with a dash in them to work correctly.  Names as a path with relevant substitutions make no sense for instance names.

One more thing.  When I use this unit file (or even the tomcat@.service file you propose), all my startup output ends up going to /var/log/messages instead of catalina.out.  How do I fix that?

Comment 18 Ade Lee 2013-09-13 13:43:42 UTC
Any details on how to fix where the server output is going?  Right now, its going into /var/log/messages instead of catalina.out.

Comment 19 Paul P Komkoff Jr 2013-09-13 13:49:39 UTC
I didn't break it by accident - I thought that having just > catalina.out isn't really manageable, as you can't really rotate it, so I directed all output to journal so you can do systemctl status tomcat@foo and see it. Or journalctl -u tomcat@foo.

If you want logging to a specific file here, it might be overall better way to set it up through a syslog.

If you really want it to be old-school though, we could change /usr/libexec/tomcat/server to allow it to be configured.

(but I like the journal approach better)

Comment 20 Ade Lee 2013-09-19 01:27:56 UTC
We can leave it as journal for now, although I reserve the right to ask for it to be changed later.

I did notice a typo in the /usr/libexec/tomcat/server

-Djava.security.policy==\"${CATALINA_BASE}/conf/catalina.policy\""

Notice the ==.  Should be =.  With ==, the policy is not actually loaded.  You can see this by appending -Djava.security.debug=all to the JAVA_OPTS.

With =, the policies actually load.

Notwithstanding, though, when you try to use the security manager, you get the following exception:

access: access denied ("java.util.PropertyPermission" "java.util.logging.config.class" "read")

Not sure why that is - considering that the rule appears to be in the policy.  This could be a separate bug, but it did not occur for previous versions of tomcat.

Comment 21 Ade Lee 2013-09-25 03:49:04 UTC
Any comments on the security manager issues above?  We'd like to get these resolved before f20 release if possible.

Comment 22 Paul P Komkoff Jr 2013-09-25 21:34:44 UTC
Sorry for the delay, I'm trying to deal with too many things at once; it'll be nice though if Ivan will help with at least the type you've found :)

Comment 23 Ivan Afonichev 2013-09-25 21:44:00 UTC
Sorry for delay I thought you was loading your custom security manager.
"-Djava.security.policy==" could be found in tomcat6 too.
http://pkgs.fedoraproject.org/cgit/tomcat6.git/tree/tomcat6-6.0.wrapper?h=f17

Merging Paul's changes I saw "==", but when I find it in old scripts I decided that it is correct.

I'll check upstrem scripts, how it is loaded there.

Comment 24 Ivan Afonichev 2013-09-25 21:48:53 UTC
I've checked upstrem catalina.sh and there is also 
 -Djava.security.policy==\"$CATALINA_BASE/conf/catalina.policy\" \


You can find docs about loading security policies here
http://docs.oracle.com/javase/7/docs/technotes/guides/security/PolicyFiles.html
"
(note the double equals) then just the specified policy file will be used; all the ones indicated in the security properties file will be ignored. 
"

Comment 25 Ade Lee 2013-10-08 03:08:54 UTC
Using the double equals is fine, but the line is still incorrect because the entry is supposed to be a URL.  From the link you pointed to above:

java -Djava.security.manager -Djava.security.policy==someURL SomeApp

As the code currently stands, a MalFormedURLException (no protocol) is thrown when the security manager attempts to load the policy.

If the following is used in /usr/libexec/tomcat/server instead, then the policy is actually loaded correctly:

  if [ "${SECURITY_MANAGER}" = "true" ] ; then
    OPTIONS="${OPTIONS} \
    -Djava.security.manager \
    -Djava.security.policy==file:${CATALINA_BASE}/conf/catalina.policy"
  fi

Note the removal of the quotes and the addition of file:

Comment 26 Ade Lee 2013-10-14 17:28:33 UTC
Any update on this?

Comment 27 Ivan Afonichev 2013-10-14 20:27:02 UTC
Upstream catalina.sh has 
-Djava.security.policy=="$CATALINA_BASE"/conf/catalina.policy \

I've searched the Internet and didn't find any bugreports for this issue in catalina.sh

Comment 28 Ade Lee 2013-10-15 14:14:51 UTC
Ivan, 

Have you tried to run a basic tomcat app with security manager enabled?  My guess is that you will get policy related exceptions, and if you turn on -Djava.security.debug=all (or maybe even if you do not), you will see the MalFormedURLException I mention above.

I'm not sure whats going on upstream, but in this environment, security manager just does not work with this init script.

Comment 29 Ade Lee 2013-10-29 15:42:31 UTC
Ivan, 

It turns out that the file: part is not necessary.  The following also works:

-Djava.security.policy==${CATALINA_BASE}/conf/catalina.policy"

compared to what is currently there:

-Djava.security.policy==\"${CATALINA_BASE}/conf/catalina.policy\""

You can easily see this bug.  You dont even need a webapp.  Simply enable security manager on the default tomcat instance and restart the instance.  

I did this by adding the following to /etc/sysconfig/tomcat

SECURITY_MANAGER="true"
JAVA_OPTS="-Djava.security.debug=all"

and then restarting tomcat : systemctl start tomcat.service

The tomcat service FAILS TO COME UP and there is an error in the log indicating a Malformed URL for the policy.

So, upstream is right -- what is wrong is the invocation in this script.

This is a clear and simply reproducible bug.  Its also going to cause all kinds of problems for IPA and Dogtag users when they upgrade to Fedora 20.  We need a fix for this ASAP.

Comment 30 Ivan Afonichev 2013-10-29 19:29:01 UTC
Oh
Thanks a lot for finding real root cause. I'll fix it within 7.0.47 update.
Current ETA is this weekend.

Comment 31 Ivan Afonichev 2013-11-02 21:50:59 UTC
I've found the issue. In upstream's catalina.sh tomcat is running via eval which seems to remove quotation marks.

Comment 32 Fedora Update System 2013-11-04 22:24:38 UTC
tomcat-7.0.47-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/tomcat-7.0.47-1.fc20

Comment 33 Fedora Update System 2013-11-05 19:57:55 UTC
Package tomcat-7.0.47-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing tomcat-7.0.47-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20684/tomcat-7.0.47-1.fc20
then log in and leave karma (feedback).

Comment 34 Fedora Update System 2013-11-10 07:20:34 UTC
tomcat-7.0.47-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.