Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1213818

Summary: CustomCacheStore deployed via JON is registered AFTER cache start during server start-up
Product: [JBoss] JBoss Data Grid 6 Reporter: Tomas Sykora <tsykora>
Component: JON PluginAssignee: Sebastian Łaskawiec <slaskawi>
Status: CLOSED WONTFIX QA Contact: Martin Gencur <mgencur>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5.0CC: jdg-bugs, rmarwaha, ttarrant
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
In the case when Custom Cache Store .jar file deployment into JDG server is happening via JBoss Operation Network UI (JDG server -> Create Child -> Deployment), the Deployment is registered by the JDG server too late. I.e. after the start of caches. Consequently, caches report error messages if they are configured to use Custom Cache Store implementation. As a workaround, either copy Custom Cache Store .jar file into $JDG_HOME/standalone/deployments folder or use deploy command in server CLI.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-05-27 06:55:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
this is how it should look like also in JON
none
here you can see reversed (wrong) order of cache start and deployment registration
none
RHQ deployment bundle with custom cache
none
JON deployment bundle with custom cache none

Description Tomas Sykora 2015-04-21 11:19:54 UTC
Created attachment 1016787 [details]
this is how it should look like also in JON

Scenario: 

1) in JON: creating deployment for JDG server -- deploying custom cache store .jar file, deployment is deployed OK

"06:18:44,285 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015859: Deployed "external-store-1.0-SNAPSHOT.jar" (runtime-name : "external-store-1.0-SNAPSHOT.jar")"

2) reconfiguring cache to use custom cache store (manually standalone.xml file change)

3) restarting server in order to apply new configuration 

4) the ISSUE here: server is trying to start caches -- problem: cache does not see custom cache store: "Unable to start cache loaders".

5) Deployment of custom cache store is happening AFTER an attempt to start the cache:

"07:26:27,459 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015859: Deployed "external-store-1.0-SNAPSHOT.jar" (runtime-name : "external-store-1.0-SNAPSHOT.jar")"


The order should be reversed. First, we need the deployment to happen, then start caches. 

NOTE: the functionality works exactly that way ^^^ if using copy-deployment. In that case, cache store .jar is registered into server early and starting caches can see necessary classes in time.

Comment 2 Tomas Sykora 2015-04-21 11:20:38 UTC
Created attachment 1016788 [details]
here you can see reversed (wrong) order of cache start and deployment registration

Comment 3 Sebastian Łaskawiec 2015-04-23 09:17:09 UTC
Created attachment 1017823 [details]
RHQ deployment bundle with custom cache

Comment 4 Sebastian Łaskawiec 2015-04-23 09:21:11 UTC
It seems that deployments added by JON are scanned in different phase then those which are placed in the filesystem. Unfortunately I can't tell why - it needs further investigation.

Luckily, there is a workaround - use Bundle deployment with the following recipe:

# Recipe for Custom Cache
bundle --name "Custom Cache Store" --version=1.0.0 --description "A Custom Cache"
file --source custom-store.jar --destination "@@rhq.deploy.dir@@/deployments/custom-store.jar"

(An exemplary file has been attached to this issue.)

This simulates placing the file on the filesystem, so everything works just like it was deployed locally.

Comment 5 Sebastian Łaskawiec 2015-04-23 10:43:43 UTC
We agreed with Tristan not to implement it. This kind of features will be provided in our own console.

Comment 6 Tomas Sykora 2015-04-23 11:20:18 UTC
JON plugin is in maintenance mode.

Closing as WONTFIX. 

This use-case will not be supported. Users and Admins will use copy or CLI deployment procedures in order to deploy custom cache store implementations on server.

Comment 7 Tomas Sykora 2015-04-29 16:38:40 UTC
Trying the same process with JON 3.3.0.GA, getting: 

[Distribution File] Failed to upload bundle distribution file : javax.ejb.EJBException:java.lang.IllegalArgumentException: Invalid bundle distribution file. BundleType/Recipe not recognized by any deployed server bundle plugin. -> java.lang.IllegalArgumentException:Invalid bundle distribution file. BundleType/Recipe not recognized by any deployed server bundle plugin.
javax.ejb.EJBException: java.lang.IllegalArgumentException: Invalid bundle distribution file. BundleType/Recipe not recognized by any deployed server bundle plugin.

I also tried replace deploy.txt with deploy.xml, getting the same error. The strange thing is that the plugin responsible for that functionality is installed properly: 

Display Name :	
Wildfly Patch Bundle Processor
	Name :	
WflyPatchBundleServerPlugin
Version :	
4.12.0.JON330GA

I recommend to ask JON team for help. I might overlook some minor or major fact and this will be probably the fastest way to get it work -- or to find out other actions.

Comment 8 Sebastian Łaskawiec 2015-04-30 10:26:36 UTC
Created attachment 1020526 [details]
JON deployment bundle with custom cache

Comment 9 Sebastian Łaskawiec 2015-04-30 10:37:01 UTC
The correct Custom Cache installation procedure via JON:
1. Create a deployment bundle
  a. Use "JON deployment bundle with custom cache" attachment as a template or create your own bundle. This should be a zip file with a custom cache store jar and a deploy.xml in it:
<?xml version="1.0"?>
<project name="cc-bundle" default="main" xmlns:rhq="antlib:org.rhq.bundle">
 
  <rhq:bundle name="Mongo DB Custom Cache Store" version="1.0" description="Custom Cache Store">
  	<rhq:deployment-unit name="JDG" compliance="full">
  	  <rhq:file name="custom-store.jar" replace="true"/>
  	</rhq:deployment-unit>
  </rhq:bundle>

  <target name="main" />
    
</project>
2. Log into JON
3. Navigate to "Bundles" (upper bar)
4. Click "New" button and choose "Upload" radio button
5. Click "Select File" and upload bundle created in #1
6. Finish the wizard using default options
7. Get back to "Bundles" tab
8. Select newly created bundle and click Deploy
9. Enter "Destination Name" and choose proper Resource Group (there should be only JDG servers in that group)
10. Choose "Install Directory" from "Base Location"s radio box group
11. Enter "/standalone/deployments" in "Deployment Directory" below
12. Proceed with the wizard using default options
13. Validate the deployment using "find $JDG_HOME -name "custom-store.jar" 2>/dev/null" on server host
14. Note that the bundle should be installed in $JDG_HOME/standalone/deployments

Comment 10 Tomas Sykora 2015-05-04 11:25:48 UTC
Setting back to assigned. There is ongoing parallel email discussion with regards to this issue.

Comment 11 Tomas Sykora 2015-05-04 15:37:39 UTC
Doc text describing an issue...

Comment 12 Tomas Sykora 2015-05-06 11:59:01 UTC
We have to remove replace="true" from our bundle file. When the property is enabled it is supposed to read the file as a string and replace tokens with bundle input parameters. If you remove that attribute, it does not modify your jar.

If anyone wants to thank someone - Filib Brychta from JON QE is the right person and credit goes to him ;)


The corrected version looks like this: 

<?xml version="1.0"?>
<project name="cc-bundle" default="main" xmlns:rhq="antlib:org.rhq.bundle"> 
  <rhq:bundle name="External Custom Cache Store" version="1.1" description="Custom Cache Store">
  	<rhq:deployment-unit name="JDG" compliance="full">
  	  <rhq:file name="external-store-1.0-SNAPSHOT.jar" />
  	</rhq:deployment-unit>
  </rhq:bundle>
  <target name="main" />    
</project>

I verified this is working now! 

I was able to create a bundle like putting this content into JON dialog window, specify my custom cache store .jar file (from my local disk), upload it to jon, finish bundle creating ... select bundle, open, select a group where to deploy (preprepared to include a JDG server) and I deploy that. Then I reconfigured a cache on JDG server to add a cache store... and issued a restart from JON. 

Looking at server logs, the custom cache store .jar file was PUT into standalone/deployments folder (now unchanged!!!) and was registered by our deployment scanner, deployed and was USED SUCCESSFULLY by a cache. 

That works! 
Thank you everyone involved.

Comment 13 Tomas Sykora 2015-05-27 06:55:03 UTC
Closing this one...
We have a workaround -- we will use JON Bundles and recipe for Custom Cache Store .jar file deployment