Bug 845993

Summary:	Jetty unpacks deployed apps to /tmp which get cleaned up by systemd-tmpfiles
Product:	[Fedora] Fedora	Reporter:	Owen Fraser-Green <owen>
Component:	jetty	Assignee:	Mikolaj Izdebski <mizdebsk>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	17	CC:	hherdianto, java-sig-commits, kdaniel, mizdebsk, overholt, sochotni, trav270man
Target Milestone:	---	Keywords:	Regression, Reopened
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	8.1.2-11	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-09-04 22:57:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Owen Fraser-Green 2012-08-06 12:31:29 UTC

Description of problem:

After running a Jetty app for some days without problem, it may suddenly give HTTP 404 messages serving static content, and java.lang.NoClassDefFoundError exceptions for dynamic content. Closer investigation reveals that, while the files were present in the WAR file, the WAR file has been unpacked to /tmp/jetty-0.0.0.0-<something> and some of the contents have disappeared. While I don't have evidence that systemd-tmpfiles is responsible, I see it's configured to delete files in /tmp after 10 days so I assume it's the culprit.

Version-Release number of selected component (if applicable):

jetty-8.1.2-9
systemd-44-17

How reproducible:

If my assumption as to the cause are correct, this will occur in most cases a Jetty app is deployed after 10 days.

Steps to Reproduce:
1. Deploy a Jetty app
2. Wait
  
Actual results:

Jetty serves HTTP 404 messages instead of content.

Expected results:

The actual content is served.

Additional info:

I worked around by simply creating the directory /usr/share/jetty/work/
As per http://wiki.eclipse.org/Jetty/Reference/Temporary_Directories Jetty will the create it's working directory under here. While this work around is fine for me, I believe a better place for the work directory would be under /var, perhaps /var/lib/jetty/work.

Comment 1 Mikolaj Izdebski 2012-08-07 05:07:57 UTC

Reproduced on F17. After removing the directory from /tmp jetty stops working.

$ rpm -q jetty
jetty-8.1.2-9.fc17.noarch

$ curl -I 127.0.0.1:8080
HTTP/1.1 200 OK
Date: Tue, 07 Aug 2012 04:46:59 GMT
Content-Type: text/html
Last-Modified: Fri, 09 Mar 2012 04:53:42 GMT
Content-Length: 2239
Accept-Ranges: bytes
Server: Jetty(8.1.0.v20120127)

# rm -rf /tmp/jetty-0.0.0.0-8080-test.war-_-any-

$ curl -I 127.0.0.1:8080
HTTP/1.1 404 Not Found
Date: Tue, 07 Aug 2012 04:47:57 GMT
Content-Type: text/html;charset=ISO-8859-1
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1378
Server: Jetty(8.1.0.v20120127)

Comment 2 Mikolaj Izdebski 2012-08-07 06:44:03 UTC

Fixed in jetty 8.1.2-10. Now jetty will unpack WARs to /var/run/jetty, which is symlinked from /usr/share/jetty/work.

Owen, you'll have to remove the directory you created as a workaround or RPM won't be able to install the update. See bug #447156.

Comment 3 Fedora Update System 2012-08-07 07:02:57 UTC

jetty-8.1.2-10.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/jetty-8.1.2-10.fc17

Comment 4 Fedora Update System 2012-08-09 22:52:42 UTC

Package jetty-8.1.2-10.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing jetty-8.1.2-10.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-11562/jetty-8.1.2-10.fc17
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2012-08-18 01:23:46 UTC

jetty-8.1.2-10.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 6 Tex 2012-08-23 21:17:19 UTC

On my fully updated system, the /var/run/jetty directory did not exist and the symlink at /usr/share/jetty/work was therefore broken.  I believe that this caused the boot failure message stating that systemd-tmpfiles-setup.service had failed since the last couple of days.  
Messages logs include: 
"systemd-tmpfiles[2328]: [/etc/tmpfiles.d/jetty.conf:1] Unknown user 'jetty'." and 
"systemd-tmpfiles-setup.service: main process exited, code=exited, status=1"  Once I created the /var/run/jetty directory, reboots have not thrown any errors and systemd-tmpfiles-setup.service completes correctly.

Comment 7 Mikolaj Izdebski 2012-08-24 09:51:05 UTC

Thank you Tex for the report. I confirm this is a regression, reopening.

Comment 8 Mikolaj Izdebski 2012-08-27 16:27:54 UTC

Fixed in F17 in jetty 8.1.2-11
F18, F19 in 8.1.5-5

Comment 9 Fedora Update System 2012-08-27 16:33:46 UTC

jetty-8.1.5-5.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/jetty-8.1.5-5.fc18

Comment 10 Fedora Update System 2012-08-27 16:49:45 UTC

jetty-8.1.2-11.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/jetty-8.1.2-11.fc17

Comment 11 Fedora Update System 2012-08-27 22:56:05 UTC

Package jetty-8.1.2-11.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing jetty-8.1.2-11.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-12785/jetty-8.1.2-11.fc17
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2012-09-04 22:57:56 UTC

jetty-8.1.2-11.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2012-09-17 23:12:23 UTC

jetty-8.1.5-5.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 14 Herry Herdianto 2012-10-10 08:57:36 UTC

F17 fully updated, jetty-8.1.2-11 still don't have directory /var/run/jetty like in comment #6 https://bugzilla.redhat.com/show_bug.cgi?id=845993#c6
reinstalling jetty-8.1.2-11 didn't have any effect.