|Summary:||[GSS] (6.3.0) Cleanup deploy directories - AS7-6031|
|Product:||[JBoss] JBoss Enterprise Application Platform 6||Reporter:||Shaun Appleton <sappleto>|
|Component:||Other||Assignee:||Default User <jbpapp-maint>|
|Status:||CLOSED CURRENTRELEASE||QA Contact:|
|Version:||6.0.0||CC:||bmaxwell, brian.stansberry, cdewolf, dereed, dimitris, jaugustin2006, jawilson, jlivings, jnevill, kkhan, msadhukh, myarboro, pjelinek, pjindal, pkremens, rdickens, sappleto, sgilda, smumford, soenke.juergensen, trogers, yann.albou, yann-email|
|Target Release:||EAP 6.3.0|
|Fixed In Version:||Doc Type:||Bug Fix|
This release of JBoss EAP 6.3 includes a fix that ensures files and directories created in the *JBOSS_HOME/tmp* and *JBOSS_HOME/tmp/vfs* folders are removed before they can interfere with newly (re)started EAP instances. In previous versions of JBoss EAP, older files may have been left behind after a server is shutdown unexpectedly (as JBoss EAP removes files in *JBOSS_HOME/tmp* and *JBOSS_HOME/tmp/vfs* as part of the shutdown process). The fix in this release provides a failsafe to mitigate that scenario. If a JBoss EAP 6.3 server does not shut down gracefully the server will not have an opportunity to clean up these temporary files. Upon restart however, the server now queries the above locations and, if files from a previous instance are present, it initiates a rename/remove process that allows the creation of fresh files for the new instance (the old directories are renamed so as to avoid interfering with newly created files). These processes happen in parallel. Upon restarting JBoss EAP (either gracefully or otherwise) old temporary files are now removed (either at shutdown or restart), to ensure they do not take up unnecessary disk space. *NOTE* Users should avoid using the *-Xrs* JAVA_OPT as this limits signals processing and can result in the size of the tmp/vfs directories continuing to grow.
|:||1078665 (view as bug list)||Environment:|
|Last Closed:||2014-06-28 15:40:18 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
|Bug Depends On:|
Description Shaun Appleton 2012-11-26 15:36:20 UTC
Help Desk Ticket Reference: https://c.na7.visual.force.com/apex/Case_View?id=500A000000Beywm project_key: JBPAPP6 Implement AS-6031 in EAP 6.x.y
Comment 1 Shaun Appleton 2013-01-28 17:21:10 UTC
AS-6031 should be AS7-6031
Comment 2 Brian Stansberry 2013-03-06 22:32:27 UTC
This was resolved in AS 7.2 so should be included in the 6.1 ER1 build.
Comment 4 Petr Kremensky 2013-04-15 07:36:58 UTC
This is still not fixed in EAP 6.1.0 ER4. Can not find changes from AS7-6031 in EAP's sources. tmp/vfs is not cleaned up during boot.
Comment 6 Shaun Appleton 2013-06-18 15:20:56 UTC
AS7-6031 is now https://issues.jboss.org/browse/WFLY-1547 and this shows this was not fixed in EAP 6.1.0.GA can this be addressed for EAP 6.1.1?
Comment 7 Petr Kremensky 2013-06-19 11:49:07 UTC
This is still not fixed in EAP 6.1.1 ER1. Tmp directories are still not cleaned up by EAP.
Comment 11 Shaun Appleton 2013-07-29 16:29:46 UTC
So EAP 6.1.1. will be released with this known issue and it will be in the release notes?
Comment 22 Begebel 2013-09-10 16:18:11 UTC
(In reply to Shaun Appleton from comment #11) > So EAP 6.1.1. will be released with this known issue and it will be in the > release notes? I just looked at the release notes it points back this page. Genius!
Comment 24 Brian Stansberry 2013-09-17 11:07:02 UTC
*** Bug 978062 has been marked as a duplicate of this bug. ***
Comment 26 JBoss JIRA Server 2013-09-17 13:57:47 UTC
jaikiran pai <firstname.lastname@example.org> made a comment on jira WFLY-1547 Shaun, thanks for testing this.
Comment 27 JBoss JIRA Server 2013-09-19 08:39:19 UTC
Dimitris Andreadis <email@example.com> made a comment on jira WFLY-1547 BTW, the reaper thread how does it differentiate old tmp content that can be removed from new content that might be created during the startup. Is it checking the date?
Comment 28 JBoss JIRA Server 2013-09-19 08:56:06 UTC
jaikiran pai <firstname.lastname@example.org> made a comment on jira WFLY-1547 >> BTW, the reaper thread how does it differentiate old tmp content that can be removed from new content that might be created during the startup. Is it checking the date? Each temp subdirectory within the tmp/vfs directory belongs to a specific "provider type". The "provider type" relies on a VFS API to create that temp subdirectory content. Now when the server is restarted, the "provider type" code (from within WildFly, for example) will ask the VFS API to create a temp dir for it. The VFS API then checks if there's already a subdirectory for that "provider type" (left behind possibly from a previous run) and if there's one, it renames it to something else and submits a background task to delete the contents of the renamed dir. It then go ahead and creates a new temp dir for this "provider type" which the "provider type" can use to add new content. That way, the deletion of older content and addition of new content can happen parallely without interfering with one another. This code has the details https://github.com/jbossas/jboss-vfs/blob/master/src/main/java/org/jboss/vfs/TempFileProvider.java#L86
Comment 32 Petr Kremensky 2013-10-09 10:59:48 UTC
$JBOSS_HOME/standalone/tmp/vfs/ $JBOSS_HOME/domain/servers/$SERVER_NAME/tmp/vfs/ folders are now correctly cleaned up. Additional info: User shouldn't use "-Xrs" opt as this limits OS signals processing -> size of tmp/vfs keeps growing. Verified on EAP 6.2.0.ER5.
Comment 33 Scott Mumford 2013-11-20 22:51:43 UTC
Added Doc Text and marked for inclusion in EAP 6.2 Release Notes.
Comment 34 Shaun Appleton 2013-11-27 10:47:46 UTC
Testing using EAP6.2.beta1 show a circumstance where the folder can grow: 1) start server -> creates tmp folder and add content 2) kill server -> remaining content in tmp, is accepted as it is a kill 3) start server 4) shutdown server using CLI What is seen is that: - a shutdown brings back the tmp folder size to the size before the corresponding start, so it's not growing, but - the trash from a kill is never removed, so a crash or kill increases the folder size. The workaround would be to remove the left over content manually or via a script. This tidy-up should be included in the code too.
Comment 37 dereed 2014-01-27 20:40:34 UTC
I'm re-opening because this is NOT fixed as per the release notes -- after a kill the temp directories are not deleted after a restart. The fix was only partially implemented. A new method was added to TempFileProvider#create with an extra "cleanExisting" boolean. This means *all* callers to TempFileProvider#create have to be modified to add an extra ", true" on the end. But this was only done in 2 out of 5 calls in the main EAP source, so the other 3 uses still leak memory on a kill. In particular, server/src/main/java/org/jboss/as/server/deployment/DeploymentMountProvider.java is the main user of the VFS temp directory and does not have the fix.
Comment 42 Petr Kremensky 2014-03-24 12:50:40 UTC
Verified on EAP 6.3.0.DR5 jboss-eap-6.2.0.GA: - start standalone - deploy some content - kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown $ du -hs jboss-eap-6.2/standalone/tmp/ 6.5M jboss-eap-6.2/standalone/tmp/ - start standalone, kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown $ du -hs jboss-eap-6.2/standalone/tmp/ 13M jboss-eap-6.2/standalone/tmp/ ... size of tmp folder keeps growing jboss-eap-6.3.0.DR5: - start standalone - deploy some content - kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown $ du -hs jboss-eap-6.3/standalone/tmp/ 48K jboss-eap-6.3/standalone/tmp/ - start standalone, kill -9 <standalone_PID>, start standalone, jboss-cli.sh -c shutdown $ du -hs jboss-eap-6.3/standalone/tmp/ 48K jboss-eap-6.3/standalone/tmp/ ... tmp folder is properly cleaned up Same behaviour for domain mode.