Bug 900709 (JBPAPP6-1027)

Attachment: Added: jmxquery-1.3-bin.zip

Steps to Reproduce: Added: 1) Get the attached jmx client, it came originally from: http://code.google.com/p/jmxquery/downloads/detail?name=jmxquery-1.3-bin.zip&can=2&q=

2) run a JBoss instance (we used domain mode) with its native management interface accessible to the outside

3) On a remote server run the following command in a loop
java -cp ./jmxquery.jar:/opt//jboss/jboss-eap-6-GA/bin/client/jboss-client.jar jmxquery.JMXQuery -U 'service:jmx:remoting-jmx://jbosshost:9999' -username admin -password admin123 -O java.lang:type=Memory -A HeapMemoryUsage -K used

Change jboss host name, -username and -password as needed

4) In the domain tmp/auth directory watch the *.challenge files increase

jboss-sasl/​ src/​ main/​ java/​ org/​ jboss/​ sasl/​ localuser/​ LocalUserServer.java  

indicates that cleanup of unused challenge files is left to the time the JDK exists.

Link: Added: This issue Cloned to JBPAPP-9528

Link: Added: This issue incorporates AS7-5220

Link: Removed: This issue Cloned to AS7-5220 

The changes are now in for AS 7.1.3, will resolve this issue once included in the ER.

The updates to fix this are now included in AS 7.1 ready for AS 7.1.3 - see AS7-5220 for the related issues but in short this is resolved by upgrading to JBoss SASL 1.0.2.Final and JBoss Remoting 3.2.10.

The JBoss SASL change was to add the clean up logic and the JBoss Remoting change was to ensure the clean up methods were called.

We had to revert the remoting 3.2.10 upgrade in upstream, so this will likely not be fixed for AS 7.1.3.

Labels: Added: as713tracking

Even with the the workaround, it eventually leads to an OutOfMemoryError as a result of the deleteOnExit invocation in [1] 

For each connection, deletOnExit is invoked, and the file is added to the DeleteOnExitHook LinkedHashSet and overtime it takes up a sizeable chunk of the heap.

Though the file deleted from disk, the jvm still holds a reference to it.


[1] https://github.com/jboss-remoting/jboss-sasl/blob/master/src/main/java/org/jboss/sasl/localuser/LocalUserServer.java

Feel free to create a new issue for that.

Darran, if dispose/finalize is the fix you did; then I assume the call to deleteOnExit is simply no longer needed ?

So it's not really a new issue, but part of this one imho

imho the release is already tagged so if we want to tag a new release with an additional code change we will need a second issue ;-)

Link: Added: This issue Cloned to JBPAPP-9702

here you go :)

https://issues.jboss.org/browse/JBPAPP-9702

Link: Added: This issue is related to JBPAPP-9885

Remoting3 version 3.2.8.SP1 will include fix for REM3-156.  This will be resolved in 6.0.1.ER2 with that upgrade.

Link: Added: This issue depends JBPAPP-9899

Remoting 3 version 3.2.8.SP1 will be in 6.0.1.ER2 build.

Verified (jboss-eap-6.0.1.ER2).

Does this need a release note for EAP 6.0.1? 
If so, the issue needs to be re-opened and the release notes flags set (Affects Release Notes, Not Yet Documented).
If not, the issue needs to be re-opened and the release notes flag set to "Release notes not required".

Release Notes Docs Status: Added: Not Yet Documented
Affects: Added: Release Notes

Updated the release notes fields.

Release Notes Docs Status: Removed: Not Yet Documented Added: Needs More Info
Writer: Added: Darrin

Hi Shelly - just after a few details for the EAP6.0.1 release notes.

This one looks pretty straight forward: temp files were getting created but not deleted, so therefore the disk could get filled up.  And now we are deleting them when they are no longer needed.

However we need some specifics about when this was happening.  The description only says "the authentication process that creates temporary files in tmp/auth".  Does this mean that there was only a single "authentication process" that had this issue?

Here are some further details regarding how this occurred: -

As JBoss AS7 was secured by default we needed a way for local users to be able to connect to the Remoting interfaces without being forced to enter a username and password, to facilitate this we implemented a new SASL mechanism that challenges users by writing an authentication token to disk and asking the client to send that token to the server to prove that the client is really local to the AS installation and can read the file.

When a client connects to AS7 all clients are offered all currently enabled authentication mechanisms to choose from so any client can choose to use the JBOSS_LOCAL_USER mechanism this includes clients that can not use the mechanism as they can not read the challenge written to disk.  Up until this point this is the correct behaviour and as the client can not read the file the authentication using that mechanism fails and so they move on to using a different mechanism that they can use.

When a local client successfully used this mechanism we would automatically delete this file after receiving the response on the server, however in the case of a failed authentication where a client moves on to use a different mechanism we were not deleting the file - that is what is addressed in this issue - the file is now deleted for both success and failure cases.

Thanks Darran!

Release Notes Docs Status: Removed: Needs More Info Added: Documented as Resolved Issue
Release Notes Text: Added: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods to allow a client to connect without requiring a username and password. A Simple Authentication and Security Layer (SASL) mechanism was implemented that challenges users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.

Darrin, feel free to edit the release note as you see fit.

Release Notes Text: Removed: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods to allow a client to connect without requiring a username and password. A Simple Authentication and Security Layer (SASL) mechanism was implemented that challenges users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.
 Added: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods that allow a client to connect without requiring a username and password. A Simple Authentication and Security Layer (SASL) mechanism was implemented that challenges users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.

Release Notes Text: Removed: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods that allow a client to connect without requiring a username and password. A Simple Authentication and Security Layer (SASL) mechanism was implemented that challenges users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.
 Added: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods one of which allows a local client to connect without requiring a username and password. The JBOSS_LOCAL_USER Simple Authentication and Security Layer (SASL) mechanism was implemented to challenge users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.

Release Notes Text: Removed: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods one of which allows a local client to connect without requiring a username and password. The JBOSS_LOCAL_USER Simple Authentication and Security Layer (SASL) mechanism was implemented to challenge users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.
 Added: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods one, of which allows a local client to connect without requiring a username and password. The JBOSS_LOCAL_USER Simple Authentication and Security Layer (SASL) mechanism was implemented to challenge users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.

Release Notes Text: Removed: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods one, of which allows a local client to connect without requiring a username and password. The JBOSS_LOCAL_USER Simple Authentication and Security Layer (SASL) mechanism was implemented to challenge users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.
 Added: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods, one of which allows a local client to connect without requiring a username and password. The JBOSS_LOCAL_USER Simple Authentication and Security Layer (SASL) mechanism was implemented to challenge users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.

Hi Darrin, Darran just reviewed my release note text and made some corrections to it. So the content is valid, but you may want to change the wording to suit your style. :-)

Release Notes Text: Removed: JBoss Enterprise Application Platform 6 is secured by default and provides a number of authentication methods, one of which allows a local client to connect without requiring a username and password. The JBOSS_LOCAL_USER Simple Authentication and Security Layer (SASL) mechanism was implemented to challenge users by writing an authentication token to disk. A client can choose from all currently enabled authentication mechanisms and if one mechanism fails, it can choose to use another mechanism. The token was only removed from disk when the authentication method succeeded, leaving the token on the file system if the authentication failed. This token is now deleted from the file system whether the authentication succeeds or fails.
 Added: Previously, each unseccessful attempt to connect to JBoss Enterprise Application Platform 6 using the `JBOSS_LOCAL_USER SASL` mechanism was leaving a generated token file in the `tmp/auth` directory. Only a successful login attempt would remove its token file after use.  A large number of failed login attempts over time with this mechanism could result in the host machine running out of disk space.  The token file is now deleted from the file system regardless of whether the authentication attempt succeeds or fails. A failed login attempt using the `JBOSS_LOCAL_USER SASL` mechanism no longer leaves an unused token file.

[~Darrin] & others, if you are done with perfecting the release notes text, can you please close this JIRA? Thank you.

Release Notes Docs Status: Removed: Documented as Resolved Issue 
Writer: Removed: Darrin 
Release Notes Text: Removed: Previously, each unseccessful attempt to connect to JBoss Enterprise Application Platform 6 using the `JBOSS_LOCAL_USER SASL` mechanism was leaving a generated token file in the `tmp/auth` directory. Only a successful login attempt would remove its token file after use.  A large number of failed login attempts over time with this mechanism could result in the host machine running out of disk space.  The token file is now deleted from the file system regardless of whether the authentication attempt succeeds or fails. A failed login attempt using the `JBOSS_LOCAL_USER SASL` mechanism no longer leaves an unused token file. 
Docs QE Status: Removed: NEW