Bug 795722

Summary:	Message too long exception during data stress test
Product:	[JBoss] JBoss Data Grid 6	Reporter:	Michal Linhard <mlinhard>
Component:	unspecified	Assignee:	Tristan Tarrant <ttarrant>
Status:	VERIFIED ---	QA Contact:	Nobody <nobody>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.0.0	CC:	jdg-bugs, nobody
Target Milestone:	---
Target Release:	6.0.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Michal Linhard 2012-02-21 11:38:32 UTC

data stress test on 2 nodes with rest client
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg-60-perf-data-stress-test-rest/7/artifact/report/size2/serverlogs.zip
using
EDG 6.0.0.DR3
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-QE/job/edg-60-build-edg-dr/18/

shows

23:18:43,194 SEVERE [org.jgroups.protocols.UDP] (DiagnosticsHandler,null) failed sending diag rsp to /10.16.89.220:50161: java.io.IOException: Message too long
	at java.net.PlainDatagramSocketImpl.send(Native Method) [rt.jar:1.6.0_30]
	at java.net.DatagramSocket.send(DatagramSocket.java:625) [rt.jar:1.6.0_30]
	at org.jgroups.stack.DiagnosticsHandler.sendResponse(DiagnosticsHandler.java:146) [jgroups-3.0.5.Final.jar:3.0.5.Final]
	at org.jgroups.stack.DiagnosticsHandler.handleDiagnosticProbe(DiagnosticsHandler.java:135) [jgroups-3.0.5.Final.jar:3.0.5.Final]
	at org.jgroups.stack.DiagnosticsHandler.run(DiagnosticsHandler.java:95) [jgroups-3.0.5.Final.jar:3.0.5.Final]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{code}

seems like a JGroups problem, I've no idea why DiagnosticsHandler is receiving anything.

Comment 1 Michal Linhard 2012-02-21 16:00:14 UTC

Also happened in hotrod soak test (in all nodes at the same time 23:52:39)
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg-60-perf-soak-test/34/artifact/report/hotrod/serverlogs.zip

Comment 2 Michal Linhard 2012-03-12 10:30:04 UTC

Another occurence in JDG 6.0.0.ER2:

http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg-60-perf-soak-test/39/artifact/report/memcached/serverlogs.zip

Comment 3 Tristan Tarrant 2012-03-12 10:59:55 UTC

I will disable the diagnostics for ER4

Comment 4 Michal Linhard 2012-03-19 16:43:31 UTC

This might not be relevant, but this issue reappeared in ER4 patched with the 54432439ed3e9d77aaacd720a35f490b  infinispan-core.jar
df7d47d2d7d10f1197c0fa01338bdf70  infinispan-core.jar.xz
to test the size2 performance for https://bugzilla.redhat.com/show_bug.cgi?id=758759

disabling diagnostics meant only editing the bundled standalone.xml right ?

(we'll need to make changes to our configs, we kept diagnostics enabled in ER4 tests)

Comment 5 Tristan Tarrant 2012-03-19 16:56:22 UTC

Yes, you need to disable diagnostics in the config or change the diagnostic port so you don't receive spurious messages from somewhere else on the network.

Comment 6 Michal Linhard 2012-03-28 13:50:51 UTC

I'll close this if it doesn't appear in any of ER5 tests.

Comment 7 Tristan Tarrant 2012-03-28 15:11:43 UTC

this messaged appeared only because jgroups diagnostics was enabled, and since it is a broken feature anyway, we're better off without it