Bug 1389684

Summary: [GSS] Cross site replication fails if authentication is enabled
Product: [JBoss] JBoss Data Grid 6 Reporter: Osamu Nagano <onagano>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: POST --- QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.6.1CC: chuffman, jdg-bugs, pruivo, tterris, txiao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
test-project-with-config.zip none

Description Osamu Nagano 2016-10-28 08:59:47 UTC
Description of problem:
Cross site replication seems not considering authentication configuration, ConfigurationBuilder.security().authentication().

Version-Release number of selected component (if applicable):
JDG 6.6.1 server and Hot Rod Java client

How reproducible:
Always

Steps to Reproduce:
See the next comment.

Actual results:
sever.log in the primary cluster:
~~~
17:31:02,246 INFO  [org.infinispan.AUDIT] (HotRodServerWorker-2) [ALLOW] SimpleUserPrincipal [name=admin] BULK_WRITE cache[default]
17:31:02,330 INFO  [org.infinispan.AUDIT] (HotRodServerWorker-2) [ALLOW] SimpleUserPrincipal [name=admin] WRITE cache[default]
17:31:02,334 INFO  [org.infinispan.AUDIT] (HotRodServerWorker-2) [ALLOW] SimpleUserPrincipal [name=admin] WRITE cache[default]
17:31:02,339 INFO  [org.infinispan.AUDIT] (Incoming-2,shared=tcp-global) [DENY] null ADMIN cache[default]
17:31:02,339 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,shared=tcp-global) ISPN000071: Caught exception when handling command SingleXSiteRpcCommand{command=ClearCommand{flags=null}}: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission
        at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:76)
        at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:44)
        at org.infinispan.security.impl.SecureCacheImpl.getCacheConfiguration(SecureCacheImpl.java:454)
        at org.infinispan.xsite.BackupReceiverRepositoryImpl.createBackupReceiver(BackupReceiverRepositoryImpl.java:163)
        at org.infinispan.xsite.BackupReceiverRepositoryImpl.getBackupReceiver(BackupReceiverRepositoryImpl.java:95)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromRemoteSite(CommandAwareRpcDispatcher.java:283)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:252)
        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.JChannel.up(JChannel.java:739) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.deliver(RELAY2.java:618) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.route(RELAY2.java:514) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.handleMessage(RELAY2.java:489) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.handleRelayMessage(RELAY2.java:470) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.Relayer$Bridge.receive(Relayer.java:265) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.JChannel.up(JChannel.java:769) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1033) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1087) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:790) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:299) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:286) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.Discovery.up(Discovery.java:291) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2842) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
~~~

server.log in the backup cluster:
~~~
17:31:02,265 INFO  [org.infinispan.factories.TransactionManagerFactory] (HotRodServerWorker-2) ISPN000161: Using a batchMode transaction manager
17:31:02,285 INFO  [org.jboss.as.clustering.infinispan] (HotRodServerWorker-2) JBAS010281: Started __cluster_registry_cache__ cache from clustered container
17:31:02,295 INFO  [org.infinispan.AUDIT] (HotRodServerWorker-2) [ALLOW] SimpleUserPrincipal [name=admin] BULK_WRITE cache[default]
17:31:02,304 INFO  [org.infinispan.AUDIT] (Incoming-2,shared=tcp-global) [DENY] null ADMIN cache[default]
17:31:02,304 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,shared=tcp-global) ISPN000071: Caught exception when handling command SingleXSiteRpcCommand{command=ClearCommand
{flags=null}}: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission
        at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:76)
        at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:44)
        at org.infinispan.security.impl.SecureCacheImpl.getCacheConfiguration(SecureCacheImpl.java:454)
        at org.infinispan.xsite.BackupReceiverRepositoryImpl.createBackupReceiver(BackupReceiverRepositoryImpl.java:163)
        at org.infinispan.xsite.BackupReceiverRepositoryImpl.getBackupReceiver(BackupReceiverRepositoryImpl.java:95)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromRemoteSite(CommandAwareRpcDispatcher.java:283)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:252)
        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.JChannel.up(JChannel.java:739) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.deliver(RELAY2.java:618) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.route(RELAY2.java:514) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.handleMessage(RELAY2.java:489) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.RELAY2.handleRelayMessage(RELAY2.java:470) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.relay.Relayer$Bridge.receive(Relayer.java:265) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.JChannel.up(JChannel.java:769) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1033) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.stack.Protocol.up(Protocol.java:420) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1087) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:790) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:299) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:286) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.Discovery.up(Discovery.java:291) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2842) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) [jgroups-3.6.3.Final-redhat-6.jar:3.6.3.Final-redhat-6]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
~~~

Comment 2 Osamu Nagano 2016-10-28 09:11:26 UTC
Created attachment 1214885 [details]
test-project-with-config.zip

test-project-with-config.zip is a test client and configurations to demonstrate the issue. The each set of configuration starts a single node primary cluster and a single node backup cluster both on localhost.

Good case without authentication:
% clustered.sh -c clustered-site1-noauth.xml # primary cluster
% clustered.sh -c clustered-site2-noauth.xml # backup cluster
% mvn test -Dtest='CacheTest#testNoAuthRemoteCache'
  => the test will succeed.

Bad case with authentication:
User "admin:admin" is expected in ApplicationRealm. Use application-users.properties contained in each cluster.
% clustered.sh -c clustered-site1.xml # primary cluster
% clustered.sh -c clustered-site2.xml # backup cluster
% mvn test -Dtest='CacheTest#testRemoteCache'
  ==> the test will fail.