Bug 901210 (JBPAPP6-1657) - [GSS] (6.3.0) Cleanup deploy directories - AS7-6031
Summary: [GSS] (6.3.0) Cleanup deploy directories - AS7-6031
Status: CLOSED CURRENTRELEASE
Alias: JBPAPP6-1657
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Other
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: DR5
: EAP 6.3.0
Assignee: Default User
QA Contact:
URL: http://jira.jboss.org/jira/browse/JBP...
Whiteboard:
Keywords: Reopened
: 978062 1043909 (view as bug list)
Depends On:
Blocks: 1078665
TreeView+ depends on / blocked
 
Reported: 2012-11-26 15:36 UTC by Shaun Appleton
Modified: 2019-04-16 14:01 UTC (History)
23 users (show)

(edit)
This release of JBoss EAP 6.3 includes a fix that ensures files and directories created in the *JBOSS_HOME/tmp* and *JBOSS_HOME/tmp/vfs* folders are removed before they can interfere with newly (re)started EAP instances. 

In previous versions of JBoss EAP, older files may have been left behind after a server is shutdown unexpectedly (as JBoss EAP removes files in *JBOSS_HOME/tmp* and *JBOSS_HOME/tmp/vfs* as part of the shutdown process). 

The fix in this release provides a failsafe to mitigate that scenario. If a JBoss EAP 6.3 server does not shut down gracefully the server will not have an opportunity to clean up these temporary files. Upon restart however, the server now queries the above locations and, if files from a previous instance are present, it initiates a rename/remove process that allows the creation of fresh files for the new instance (the old directories are renamed so as to avoid interfering with newly created files). These processes happen in parallel. 

Upon restarting JBoss EAP (either gracefully or otherwise) old temporary files are now removed (either at shutdown or restart), to ensure they do not take up unnecessary disk space.

*NOTE* Users should avoid using the *-Xrs* JAVA_OPT as this limits signals processing and can result in the size of the tmp/vfs directories continuing to grow.
Clone Of:
: 1078665 (view as bug list)
(edit)
Last Closed: 2014-06-28 15:40:18 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 270653 None None None Never
JBoss Issue Tracker JBPAPP6-1657 Major Closed Cleanup deploy directories - AS7-6031 2018-12-03 18:20 UTC
JBoss Issue Tracker WFLY-1547 Major Closed deploy directories not cleaned up 2018-12-03 18:20 UTC

Description Shaun Appleton 2012-11-26 15:36:20 UTC
Help Desk Ticket Reference: https://c.na7.visual.force.com/apex/Case_View?id=500A000000Beywm
project_key: JBPAPP6

Implement AS-6031 in EAP 6.x.y

Comment 1 Shaun Appleton 2013-01-28 17:21:10 UTC
AS-6031 should be AS7-6031

Comment 2 Brian Stansberry 2013-03-06 22:32:27 UTC
This was resolved in AS 7.2 so should be included in the 6.1 ER1 build.

Comment 4 Petr Kremensky 2013-04-15 07:36:58 UTC
This is still not fixed in EAP 6.1.0 ER4. Can not find changes from AS7-6031 in EAP's sources. tmp/vfs is not cleaned up during boot.

Comment 6 Shaun Appleton 2013-06-18 15:20:56 UTC
AS7-6031 is now 
https://issues.jboss.org/browse/WFLY-1547

and this shows this was not fixed in EAP 6.1.0.GA

can this be addressed for EAP 6.1.1?

Comment 7 Petr Kremensky 2013-06-19 11:49:07 UTC
This is still not fixed in EAP 6.1.1 ER1. Tmp directories are still not cleaned up by EAP.

Comment 11 Shaun Appleton 2013-07-29 16:29:46 UTC
So EAP 6.1.1. will be released with this known issue and it will be in the release notes?

Comment 22 Begebel 2013-09-10 16:18:11 UTC
(In reply to Shaun Appleton from comment #11)
> So EAP 6.1.1. will be released with this known issue and it will be in the
> release notes?

I just looked at the release notes it points back this page. Genius!

Comment 24 Brian Stansberry 2013-09-17 11:07:02 UTC
*** Bug 978062 has been marked as a duplicate of this bug. ***

Comment 26 JBoss JIRA Server 2013-09-17 13:57:47 UTC
jaikiran pai <jpai@redhat.com> made a comment on jira WFLY-1547

Shaun, thanks for testing this.

Comment 27 JBoss JIRA Server 2013-09-19 08:39:19 UTC
Dimitris Andreadis <dimitris@redhat.com> made a comment on jira WFLY-1547

BTW, the reaper thread how does it differentiate old tmp content that can be removed from new content that might be created during the startup. Is it checking the date?

Comment 28 JBoss JIRA Server 2013-09-19 08:56:06 UTC
jaikiran pai <jpai@redhat.com> made a comment on jira WFLY-1547

>> BTW, the reaper thread how does it differentiate old tmp content that can be removed from new content that might be created during the startup. Is it checking the date? 

Each temp subdirectory within the tmp/vfs directory belongs to a specific "provider type". The "provider type" relies on a VFS API to create that temp subdirectory content. Now when the server is restarted, the "provider type" code (from within WildFly, for example) will ask the VFS API to create a temp dir for it. The VFS API then checks if there's already a subdirectory for that "provider type" (left behind possibly from a previous run) and if there's one, it renames it to something else and submits a background task to delete the contents of the renamed dir. It then go ahead and creates a new temp dir for this "provider type" which the "provider type" can use to add new content. That way, the deletion of older content and addition of new content can happen parallely without interfering with one another. 

This code has the details https://github.com/jbossas/jboss-vfs/blob/master/src/main/java/org/jboss/vfs/TempFileProvider.java#L86

Comment 32 Petr Kremensky 2013-10-09 10:59:48 UTC
$JBOSS_HOME/standalone/tmp/vfs/
$JBOSS_HOME/domain/servers/$SERVER_NAME/tmp/vfs/
folders are now correctly cleaned up.

Additional info:
User shouldn't use "-Xrs" opt as this limits OS signals processing -> size of tmp/vfs keeps growing.

Verified on EAP 6.2.0.ER5.

Comment 33 Scott Mumford 2013-11-20 22:51:43 UTC
Added Doc Text and marked for inclusion in EAP 6.2 Release Notes.

Comment 34 Shaun Appleton 2013-11-27 10:47:46 UTC
Testing using EAP6.2.beta1 show a circumstance where the folder can grow:

1) start server -> creates tmp folder and add content
2) kill server -> remaining content in tmp, is accepted as it is a kill
3) start server
4) shutdown server using CLI

What is seen is that:
- a shutdown brings back the tmp folder size to the size before the corresponding start, so it's not growing, but
- the trash from a kill is never removed, so a crash or kill increases the folder size.

The workaround would be to remove the left over content manually or via a script.
This tidy-up should be included in the code too.

Comment 37 dereed 2014-01-27 20:40:34 UTC
I'm re-opening because this is NOT fixed as per the release notes -- after a kill the temp directories are not deleted after a restart.

The fix was only partially implemented.

A new method was added to TempFileProvider#create with an extra "cleanExisting" boolean.  This means *all* callers to TempFileProvider#create have to be modified to add an extra ", true" on the end.
But this was only done in 2 out of 5 calls in the main EAP source, so the other 3 uses still leak memory on a kill.

In particular, server/src/main/java/org/jboss/as/server/deployment/DeploymentMountProvider.java is the main user of the VFS temp directory and does not have the fix.

Comment 42 Petr Kremensky 2014-03-24 12:50:40 UTC
Verified on EAP 6.3.0.DR5 

jboss-eap-6.2.0.GA:
 - start standalone
 - deploy some content
 - kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown
$ du -hs jboss-eap-6.2/standalone/tmp/
6.5M	jboss-eap-6.2/standalone/tmp/
 - start standalone, kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown
$ du -hs jboss-eap-6.2/standalone/tmp/
13M	jboss-eap-6.2/standalone/tmp/
... size of tmp folder keeps growing

jboss-eap-6.3.0.DR5:
 - start standalone
 - deploy some content
 - kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown
$ du -hs jboss-eap-6.3/standalone/tmp/
48K	jboss-eap-6.3/standalone/tmp/
 - start standalone, kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown
$ du -hs jboss-eap-6.3/standalone/tmp/
48K	jboss-eap-6.3/standalone/tmp/
... tmp folder is properly cleaned up

Same behaviour for domain mode.

Comment 45 Brian Stansberry 2014-04-30 18:26:36 UTC
*** Bug 1043909 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.