Bug 745917 (EDG-13)

Summary: Cannot store entries in cache via memcached protocol
Product: [JBoss] JBoss Data Grid 6 Reporter: Martin Gencur <mgencur>
Component: unspecifiedAssignee: Default User <jbpapp-maint>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: unspecifiedCC: galder.zamarreno, jdg-bugs, mgencur, paul.ferraro, prabhat.jha
Target Milestone: ---   
Target Release: 6.0.0   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/EDG-13
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Infinispan 5.0.0.FINAL, JBossAS 7.1-SNAPSHOT
Last Closed: 2011-08-22 11:39:02 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
node1.txt
none
node2.txt
none
trace_node0.txt
none
trace_node1.txt none

Description Martin Gencur 2011-08-11 15:26:01 UTC
project_key: EDG

The test for this can be found at https://svn.devel.redhat.com/repos/jboss-qa/edg/edg-functional-tests/trunk/memcached-verification/src/test/java/org/jboss/test/datagrid/memcached/MemcachedClientsTest.java

The test is failing on lines:

assertEquals(1, server1.getDefaultCacheManager().getDefaultCache().getNumberOfEntries());

and

assertEquals("value1", value);

The steps to run this test are similar to those described in JBPAPP-6928.

I also tried to use ispncon tool to verify this finding with the following results:

For EDG containing Infinispan 5.0.0.Final:
> mgencur@mgencur:~/Java/sf-components/project-edg/trunk$ ispncon -c memcached -h test1 -p 11211 put key2 value2
> STORED
>mgencur@mgencur:~/Java/sf-components/project-edg/trunk$ ispncon -c memcached -h test2 -p 11211 get key2
>NOT_FOUND

For EDG build from 2010-08-02 (with Infinispan 5.0.0-SNAPSHOT):

>mgencur@mgencur:~/Java/sf-components/project-edg/trunk$ ispncon -c memcached -h test1 -p 11211 put key1 value1
>STORED

>mgencur@mgencur:~/Java/sf-components/project-edg/trunk$ ispncon -c memcached -h test2 -p 11211 get key1
>value1


From the lines above it seems that there's a problem with replication, but from 
MemcachedClientsTest.java test we can see that no entry was stored in any cache.

Comment 1 Galder Zamarreño 2011-08-11 15:41:39 UTC
Hmmm, Martin, have you verified that the nodes are joining in a cluster? Any errors in the log?...etc.

Comment 2 Martin Gencur 2011-08-11 15:46:57 UTC
Nodes joined in the cluster, no errors seen anywhere but will check once again.

Comment 3 Martin Gencur 2011-08-11 16:02:26 UTC
Attaching log files with TRACE logging for both servers. I tried to put a key=key1 with value=value1 to node1 and retrieve it from node2. 



Comment 4 Martin Gencur 2011-08-11 16:02:26 UTC
Attachment: Added: node1.txt
Attachment: Added: node2.txt


Comment 5 Martin Gencur 2011-08-11 16:04:38 UTC
The following configuration was used for both nodes:

{code:xml}
<cache-container name="default" default-cache="default" listener-executor="infinispan-listener">
                <transport executor="infinispan-transport" lock-timeout="240000"/>
                <distributed-cache owners="2" virtual-nodes="512" l1-lifespan="0" mode="SYNC" remote-timeout="30000" name="default" start="EAGER" batching="false" indexing="NONE">
                    <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="30000" concurrency-level="1000"/>
                    <transaction mode="NON_XA"/>
                    <rehashing enabled="true" timeout="600000"/>
                </distributed-cache>
                <distributed-cache mode="SYNC" name="namedCache"/>
            </cache-container>
{code} 

Comment 6 Galder Zamarreño 2011-08-11 16:40:18 UTC
As a result of ISPN-1206, an issue has been found in AS7 which is leading to this problem:  AS7-1518

Comment 7 Galder Zamarreño 2011-08-11 16:40:46 UTC
Link: Added: This issue depends AS7-1518


Comment 8 Galder Zamarreño 2011-08-11 16:41:17 UTC
Link: Added: This issue is related to ISPN-1206


Comment 9 Martin Gencur 2011-08-12 08:36:56 UTC
I verified this with the new build (JBossAS 7.1.0.Alpha1-SNAPSHOT from today, EDG6 SNAPSHOT including this JBossAS, Infinispan 5.0.0.FINAL) and it works fine already. Both the functional tests and ispncon test. Galder, we can close this IMHO.

Comment 10 Martin Gencur 2011-08-18 15:30:06 UTC
It seems the issue appeared again. I tried EDG using JBossAS 7.0.1.Final and Infinispan 5.0.0.Final and performed the same ispncon test as before which failed. Also our functional tests for memcached are failing: https://hudson.qa.jboss.com/hudson/view/EDG6/job/edg-60-functional-tests-ispn-arq/95/#showFailuresLink

Comment 11 Martin Gencur 2011-08-18 15:30:06 UTC
Attachment: Added: trace_node0.txt
Attachment: Added: trace_node1.txt


Comment 12 Martin Gencur 2011-08-18 15:35:23 UTC
This was already working on 2011-08-12 with JBossAS snapshot from that day and Infinispan 5.0.0.Final. Today, I also tried JBossAS snapshot from 2011-08-12 with todays Infinispan snapshot (5.1.0-SNAPSHOT) and this worked well as well. So it's not in Infinspan itself but probably in JBossAS integration.

Comment 13 Martin Gencur 2011-08-19 08:48:37 UTC
I ran our testsuite with a few more builds based on JBossAS with various HEADs. I found out that this problem is caused by this commit: https://github.com/jbossas/jboss-as/commit/a2fd1dbe6a1609f8532e53f571b22464d67a2833. The default cache for memcached is now named memcachedCache and this commit somehow overlooks it.

Comment 14 Manik Surtani 2011-08-19 10:52:46 UTC
A bug in AS 7 code?

Comment 15 Martin Gencur 2011-08-19 13:28:37 UTC
Paul, the commit I mentioned is exactly the opposite to the one which fixed AS7-1518 (which led to this JIRA being fixed): https://github.com/pferraro/jboss-as/commit/2122489b00e19e5afef6c95586329c0fcd04465d

Comment 16 Paul Ferraro 2011-08-19 19:11:17 UTC
OK - I think the source of the confusion is a semantic difference between the "default-cache" in AS7 and the <default> cache configuration concept in Infinispan.  In AS7, caches do not inherit their configuration from the cache identified as the default cache.  Instead, their configuration is inherited from an internal configuration definition that varies based on the cache mode.  The point of this AS-specific behavior is to simplify the process of switching default cache configurations for a given container, without needing to worry about inheritance.
Consequently, in AS7, calls to EmbeddedCacheManager.defineConfiguration(String, Configuration) should not expect inherit the configuration of the "default-cache".  Judging from your configuration above, it looks like this is what you're expecting.  

Comment 17 Paul Ferraro 2011-08-19 19:36:41 UTC
To be more specific, if the EDG code should use the 3-arg method:
manager.defineConfiguration(name, templateCache, configuration);
Where templateCache is the name of the cache configuration from which you'd like to inherit.

Comment 18 Martin Gencur 2011-08-22 11:37:49 UTC
Thank you for the explanation, Paul. So this is not a real issue and we can get Infinispan to work by defining a named cache called "memcachedCache" in standalone.xml. The memcached module will use this cache as its default one.

Comment 19 Galder Zamarreño 2011-08-29 11:23:43 UTC
Paul, haven't checked javadoc but can you make sure that differences in defineConfiguration semantics between AS7 and Infinispan are documented AS7's defineConfiguration implementation if they're not already?

Comment 20 Anne-Louise Tangring 2011-09-26 19:41:23 UTC
Docs QE Status: Removed: NEW