Hide Forgot
project_key: EDG Strange thing happening while some nodes failed and rehashing is going on. Regardless, this looks like a result of a bigger problem. {code} org.infinispan.client.hotrod.exceptions.InvalidResponseException: Invalid magic number. Expected a1 and received 6e at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:84) at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:48) at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:27) at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:38) at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:166) at org.infinispan.CacheSupport.put(CacheSupport.java:28) at org.jboss.smartfrog.edg.adapter.HotRodAdapter$HotRodRemoteCacheAdapter.put(HotRodAdapter.java:150) at org.jboss.smartfrog.edg.loaddriver.CacheRequestProcessorFactory$InfinispanRequestProcessor.processRequest(CacheRequestProcessorFactory.java:163) at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51) at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87) at java.lang.Thread.run(Thread.java:619) {code}
Link: Added: This issue is incorporated by JBPAPP-6054
No idea
Rado, it's difficult to say what the cause of this is. The first I'd try doing, if the failure can be replicated easily, would be to generate some TRACE logs for both the client and server. We could start with having: For the client side: TRACE on org.infinispan.client For the server side: TRACE on org.infinispan.server
Yeah, its easy to reproduce. I will turn on the logging but it will be very verbose. I belive you can easily reproduce by starting few nodes, start hotrod client and kill a node.
With TRACE logging, the preceeding event (client side) to this issue is: Mar 22, 2011 5:49:08 AM org.infinispan.client.hotrod.impl.operations.HotRodOperation readHeaderAndValidate SEVERE: Invalid magic number. Expected a1 and received 72
Hmmm, that means little to me. Could you send me the TRACE log for the client and can you make sure you use standard log formats used in AS? i.e. {code:xml}<layout class="org.apache.log4j.PatternLayout"> <!-- The full pattern: Date MS Priority [Category] (Thread:NDC) Message\n --> <param name="ConversionPattern" value="%d %-5r %-5p [%c] (%t:%x) %m%n"/> </layout>{code}
I know its a problem, I have a workaround in mind I will try now. Update: due to infrastructure problems JBQA-4431 we wont be able to provide this logging on *client side* atm.
some more occurences: http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-stress-client-size1-hotrod/31/console-perf09/consoleText http://hudson.qa.jboss.com/hudson/job/edg-51x-stress-client-size8-hotrod/4/console-perf02/consoleText
I started to see a lot of these when tried to switch to changed hotrod client (adapted TcpTransportFactory) see JBQA-4479. http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-stress-client-size4-hotrod/20/console-perf01/consoleText (beware: it is a big output)
here I fixed a problem in smartfrog, which leads to much less garbage in the output, but you can still see the problems: http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-stress-client-size4-hotrod/21/console-perf01/consoleText
miraculously I managed to reproduce this on my localhost using 100 threads against two EDG instances. I was able to reproduce it only with the first run of the testcase after fresh boot of both instances. the testcase: testHotRodDisableSmartRouting() in https://svn.devel.redhat.com/repos/jboss-qa/load-testing/sf-components/project-edg/trunk/src/test/java/org/jboss/smartfrog/edg/adapter/BasicAdapterManualTest.java the trace log: attached hotrod-client.zip
Attachment: Added: hotrod-client.zip
Wicked Michal. I'll have a look to them once I've fixed JBPAPP-6113. It could well be the case that the problem with both this and JBPAPP-6113, originate at the same place but can't tell for sure yet.
Link: Added: This issue is a dependency of JBQA-4479
Michal, do you have the corresponding TRACE logs for the two EDG servers? That was key for JBPAPP-6113 and I think it's key here as well. In fact, JBPAPP-6113 could have solved this issue as well cos I see several "Invalid message id" messages too. So, if after using the new snapshot uploaded last Thursday you see this issue again, please make sure you upload both the client and server TRACE logs.
OK.
The issue disappeared in my runs, when I ran with the new snapshot(20110331.163428) containing fix for JBPAPP-6113.
Hmmm, Ok. Let's be optimistic and mark it as resolved and if it appears again, just reopen with some TRACE logs if possible :)
Link: Added: This issue is related to JBPAPP-6113
Hi Galder, this is also still present as described on https://issues.jboss.org/browse/JBPAPP-6113?focusedCommentId=12599088&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12599088
Rado, let's close this cos this JIRA was for "Invalid Magic Number", and let's focus on JBPAPP-6113
note: rado is on PTO May 2-10 and 18-27 galder, we also saw "Invalid magic number" problem in the logs from JBPAPP-6113. but I'm OK with treating them as one issue, because they seem to be connected...