Bug 1011499

Summary: HotRod Rolling Upgrades NPE when disabling cache store on target node
Product: [JBoss] JBoss Data Grid 6 Reporter: Tomas Sykora <tsykora>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: VERIFIED --- QA Contact: Martin Gencur <mgencur>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: jdg-bugs, myarboro
Target Milestone: ER4   
Target Release: 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
The rolling upgrade process for Hot Rod fails with MBeanException when the new cluster is being disconnected from the old cluster after migrating all the data. The previous steps (being invoked via JMX operations recordKnownGlobalKeyset and synchronizeData) work correctly, however it is not possible to finish the whole procedure by disconnecting the two clusters from one another. Consequently, it is not possible to successfully finish the process of rolling upgrade for Hot Rod client from version JBoss Data Grid 6.1.0.GA to JBoss Data Grid 6.2.0.Beta.
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1026862    
Bug Blocks: 1017190    

Description Tomas Sykora 2013-09-24 12:41:36 UTC
More information in linked JIRA.

Setting severity as High, because this issue prevents the whole process of rolling upgrades to be finished as specified in specification DOCs.

Comment 2 JBoss JIRA Server 2013-10-10 11:26:14 UTC
Tomas Sykora <tsykora> made a comment on jira ISPN-3552

*New observations:*

I found this while testing HotRod rolling upgrades from 2 nodes -- JDG 6.1 GA to 2 nodes -- JDG 6.2 ER3:

-------------------------------------------------------------------------------
Test set: org.infinispan.test.rollingupdatesdist.HotRodRollUpsDistTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 90.917 sec <<< FAILURE!
testRollingUpgrades(org.infinispan.test.rollingupdatesdist.HotRodRollUpsDistTest)  Time elapsed: 86.902 sec  <<< ERROR!
java.io.IOException: java.lang.ClassNotFoundException: org.infinispan.CacheException
        at org.jboss.remotingjmx.protocol.v1.ClientConnection$BaseResponseHandler.handle(ClientConnection.java:1775)
        at org.jboss.remotingjmx.protocol.v1.ClientConnection$MessageReceiver$1.run(ClientConnection.java:455)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: org.infinispan.CacheException
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)
        at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:947)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
        at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
        at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1648)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
        at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
        at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1648)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:72)
        at org.jboss.remotingjmx.protocol.v1.ClientConnection$BaseResponseHandler.handle(ClientConnection.java:1766)
        ... 4 more

Comment 3 JBoss JIRA Server 2013-10-10 14:51:48 UTC
Pedro Ruivo <pedroruivo2> updated the status of jira ISPN-3552 to Coding In Progress

Comment 4 Tomas Sykora 2013-11-05 14:48:15 UTC
This issue depends on: https://bugzilla.redhat.com/show_bug.cgi?id=1026862

Formerly, I thought 1026862 exception is related to disabling remote cache store. This is not true and this issue blocks even the first of rolling upgrades operations -- recordKnownGlobalKeyset.

Once https://bugzilla.redhat.com/show_bug.cgi?id=1026862 is fixed and verified, this issue can be consequently verified as well.

We should remove this (1011499) bugzilla from beta release notes as it is superseded by 1026862 for now.

Comment 6 Tomas Sykora 2013-11-25 12:33:32 UTC
The whole smooth process is still blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1030485 (HotRod Rolling Upgrades from Delirium 5.2.4 (JDG-6.1) to latest server needs dropped org.infinispan.util.ByteArrayKey) which prevents us from synchronization of data.

But disabling stores looks not to be dependant on it. Therefore, we verified disconnectSource operation without issuing sychnonizeData to see, whether this is working properly.

And YES, this is working now.
Thank you for the fix.

VERIFIED