Bug 1228026 - Invalid message id under high load
Summary: Invalid message id under high load
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: CPP Client
Version: 6.4.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ER1
: 6.5.1
Assignee: Pedro Zapata
QA Contact: Alan Field
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-04 06:02 UTC by Osamu Nagano
Modified: 2023-09-30 08:27 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The C++ HotRod client was occasionally logging the following exception when there were many concurrent Put and Get operations: <programlisting>"Invalid message id. Expected $OLD_MSGID and received $NEW_MSGID"</programlisting> This issue was caused by the message id counter not being thread safe. This issue is resolved as of Red Hat JBoss Data Grid 6.5.1.
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)
hotrodwrapper.zip (8.20 KB, application/zip)
2015-07-29 07:33 UTC, Osamu Nagano
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker HRCPP-195 0 Major Resolved Invalid message id under high load 2020-11-06 03:52:53 UTC

Description Osamu Nagano 2015-06-04 06:02:31 UTC
A C++ Hot Rod client, which is executing about 560 get/sec and 50 put/sec concurrently, occasionally receives an exception with the following error message.

Invalid message id. Expected 5774 and received 4436

It seems the message id counter is not thread-safe.

Comment 1 Osamu Nagano 2015-06-04 06:09:51 UTC
Here are snippets from the current master.
https://github.com/infinispan/cpp-client/blob/master/src/hotrod/impl/protocol/Codec10.cpp

Gloal variable:
     25 // TODO : multithread management
     26 long Codec10::msgId = 0;

In writeHeader:
     38     // TODO : multithread management
     39     transport.writeVLong(params.setMessageId(++msgId).messageId);

In readHeader:
     82     if (receivedOpCode != params.opRespCode) {
     83       if (receivedOpCode == HotRodConstants::ERROR_RESPONSE) {
     84         checkForErrorsInResponseStatus(transport, params, status);
     85         }
     86         std::ostringstream message;
     87         message << "Invalid response operation. Expected " << std::hex <<
     88             (int) params.opRespCode << " and received " << std::hex << (int) receivedOpCode;
     89         throw InvalidResponseException(message.str());
     90     }

Comment 11 Osamu Nagano 2015-07-29 07:33:05 UTC
Created attachment 1057224 [details]
hotrodwrapper.zip

Minimum reproducing set of files for Nginx (OpenResty).  Find README.en in the archive.

Comment 15 Alan Field 2015-09-10 20:26:31 UTC
Verified with JDG 6.5.1 CR1

I ran the siege test 10,000 times without seeing the error, so I am closing this BZ for now until there is a reproducer for this.


Note You need to log in before you can comment on or make changes to this bug.