Bug 458637
| Summary: | JVM OutOfMemory memory on Red Hat MRG with HashMaps | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | IBM Bug Proxy <bugproxy> | ||||
| Component: | realtime-kernel | Assignee: | Red Hat Real Time Maintenance <rt-maint> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | |||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 1.0 | CC: | bhu, rattapat+nobody, williams | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2009-10-06 21:11:12 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Following are the details on setting up the test environment. If anything is not clear, please feel free to ask questions. In a nutshell you need to, 1) Build and run the Qpid broker 2) Build the java client 3) Run a workload using the SonicTestHarness. Please note all the above components are based on open source licenses. Following is a detailed description of how to get the above items going. 1) Build and run the Qpid broker --------------------------------- 1.1 mkdir ~/qpid; cd ~/qpid 1.2 you can grab the source code as follows. svn co https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/ -r683682 (Pls note that for other reasons we stopped using the class in question, hence you need to build from r683682 or before to reproduce the issue). 1.3 cd cpp; ./bootstrap && ./configure; If it complains of missing packages you would need to yum install them. Once that is complete it will generate the make files. 1.4 then do make clean && make install 1.5 start the broker as follows (assuming u are inside the cpp dir) sh src/qpidd --auth no --no-data-dir --default-queue-limit 0 2) Build the java client -------------------------- 2.1 cd ~/qpid/java 2.2 ant clean build (If you are compiling using jdk 1.5.x you will see some errors, just do ant build again. It looks like this happens due to a bug in jdk 1.5.x We don't see this when compiling with 1.6.x 3) Run a workload using the SonicTestHarness. ---------------------------------------------- 3.1 untar the attached SonicTestHarness.tgz and cd TestHarness. 3.2 edit the jndi.properties and add the correct broker address. connectionfactory.connectionFactory = amqp://username:password@clientid/test?brokerlist='tcp://localhost:5672' change localhost to the appropriate ip or host name. 3.1 edit setenv.sh and change the following info. set JAVA_HOME to point to the jdk u want to try. set JAR_PATH=~/qpid/java/build/lib (assuming u checked out svn into ~/qpid ) 3.4 start the subscriber sh run_client.sh -Dignore_setclientID=true -props qpid sub.props 3.5 start the producer sh run_client.sh -Dignore_setclientID=true -props qpid pub.props If you need to specify any jvm arguments u can add it before -props. Ex. sh run_client.sh -Dignore_setclientID=true -Xmx1500M -props qpid pub.props Created attachment 314579 [details]
Sonic Test harness
Rajith Attapattu: I have looked at your instructions to reproduce the issue. We are not permitted to download source code from non-IBM repositories, even if the source code is open-source. This is not permitted from our legal department. Would you be able to provide a build of the Qpid broker that contains the issue? Or indicate a binary qpid broker that an be obtained from apache that contains the issue? Or if not, is there some other manner to reproduce the issue? Sripathi, No, there is no corresponding CMVC defect. We can close this defect since no further action is necessary at this time. closing |
=Comment: #0================================================= Sean Foley <Sean_Foley.com> - 2008-08-07 14:17 EDT Problem description: RedHat is experiencing a memory problem with IBM java running on MRG. This is being looked at by the JTC (Java TC). This tracker is to be used to track from within the MRG /RT-linux community and the JTC. Original email chain: > > > > > > > > > > > > > > > > > I'm not sure I exactly understand your scenario here, so > I'll > > > ask > > > > a > > > > > > couple of clarifying questions first. Is the object that > you're > > > > using > > > > > > to store the value in your HashMap the same object (or a > clone) > > > > that's > > > > > > being used to search for that value? > > > > > They are different objects. > > > > > > > > > > > If they're different objects, then are you using that same > > > > > > hashCode() function to compute the hashCode for the Strings > in > > > > both > > > > > > objects? > > > > > They are using the same hash code function and the exact same > > > > string. > > > > > > > > > > > If you're using the same objects, or you're using the same > > > > hashCode > > > > > > function for two different objects, then I'll agree with you > > > that > > > > the > > > > > > hash code should be the same. > > > > > Yes my understanding is that it should be the same. > > > > > In fact this seems to work for more than 50k messages or so. > > > > > All these messages pass through this exact same code path. > > > > > > > > > > > Could you please clarify whether your scenario matches one > of > > > > those > > > > > > two cases? > > > > > > > > > > > > I'm a little confused at the difference in behaviour you're > > > > seeing, > > > > > > because as far as I know the HashMap implementation in WRT > is > > > > > > identical to Sun's (in fact, it is Sun's implementation > afaik). > > > > > > > > > > I'm also baffled as to why the hashCode is different after a > > > certain > > > > > point. Please feel free to ask anymore questions. > > > > > > > > > > Regards, > > > > > > > > > > Rajith. > > > > > > > > > > > > > > > > > Mark Stoodley > > > > > > Technical Lead, WebSphere Real Time > > > > > > Mark Stoodley/Toronto/IBM@IBMCA > > > > > > mstoodle.com (905) 413-5831 > > > > > > > > > > > > > > > > > > Rajith Attapattu > > > > > > <rattapat> > > > > > > > > > > > > 07/22/2008 10:13 PM > > > > > > Please respond to > > > > > > rattapat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To > > > > > > Mark > > > > > > Stoodley/Toronto/IBM@IBMCA > > > > > > cc > > > > > > Darren Hart > > > > > > <dvhltc.com>, Mike Fulton/Toronto/IBM@IBMCA, Paul > Ivinson > > > > <paul_ivinson.com>, Clark Williams <williams> > > > > > > Subject > > > > > > Re: Fw: Help with > > > > > > JVM memory > > > > > > options for Red > > > > > > Hat MRG > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Further to the problem described below, > > > > > > > > > > > > The implementation of the Map in question here is HashMap. > > > > > > At the time the error occurs I iterate over the map and > printed > > > > the > > > > > > following values for all the keys. > > > > > > 1) toString() - added quotes around to see if there any > leading > > > or > > > > > > trailing spaces. > > > > > > > > > > > > 2)length of the String returned by toString() to rule out > the > > > > > > possibility of unprintable characters or any extra space. > > > > > > > > > > > > 3) hashCode() > > > > > > > > > > > > 4) Equality : key.equals(k) where k is the value passed to > the > > > get > > > > > > method in the map. > > > > > > > > > > > > *I found that the hash code generated was different.* > > > > > > It should be the same. > > > > > > > > > > > > k is the value passed to the get method in the map. > > > > > > =================================================== > > > > > > k.toString() : "application/octet-stream" > > > > > > length of k : 24 > > > > > > k.hashCode() : -549568611 > > > > > > > > > > > > This is the key that should have been matched > > > > > > =============================================== > > > > > > Key : "application/octet-stream" > > > > > > Key.length : 24 > > > > > > Key.hasCode : 1178484637 > > > > > > Type (class) : class org.apache.qpid.framing.AMQShortString > > > > > > Equality with k : false > > > > > > > > > > > > > > > > > > The sun impl for the HashMap does the following logic for > the > > > get > > > > > > method. (I assume a similar path is taken in the IBM > > > > implementation). > > > > > > The first check it does to find a match for a given key in > its > > > key > > > > set > > > > > > is to compare a hash computed based on the hashCode of the > two > > > > objects > > > > > > being compared. > > > > > > > > > > > > The AMQShortString class, the type of the key used in the > > > HashMap > > > > has > > > > > > it's own implementation of the hashCode method (pasted > below). > > > > > > > > > > > > The AMQShortString wraps an ordinary String and the _data > array > > > is > > > > the > > > > > > same for two identical Strings. The toString method will > print > > > the > > > > > > String that's wrapped by this class. > > > > > > > > > > > > Please note the values printed above. The toString method > for > > > the > > > > key > > > > > > passed into the get method, and the key that should have > been > > > > matched, > > > > > > have identical Strings. > > > > > > > > > > > > Based on the hashCode function given below, the generated > hash > > > > code > > > > > > should also be the same. > > > > > > > > > > > > [ .... code snippet removed by Mark Stoodley .... ] > > > > > > > > > > > > Regards, > > > > > > > > > > > > Rajith > > > > > > > > > > > > On Tue, 2008-07-22 at 18:03 -0400, Rajith Attapattu wrote: > > > > > > > Hello Mark, > > > > > > > > > > > > > > Nice to meet you. > > > > > > > > > > > > > > I noticed that the script I was using wasn't passing down > the > > > > JVM > > > > > > > arguments to the consumer process :(. Less sleep is > getting to > > > > me:). > > > > > > > Once I fixed that and increased the jvm heap size with > > > -Xmx1025M > > > > the > > > > > > out > > > > > > > of memory issue went away. > > > > > > > > > > > > > > Next I am seeing an interesting issue. > > > > > > > (I don't get this error on regular java (jdk 1.6) or the > Sun > > > > RTJS.) > > > > > > > > > > > > > > Here is a brief description of whats going on. > > > > > > > I am getting an error on the consume side after sending > about > > > > 50k > > > > > > > messages. The exception complains the mime type of the > message > > > > is > > > > > > > invalid. It checks the mime type of the message against a > > > > predefined > > > > > > set > > > > > > > of mime types/messageFactories stored in a map in the > > > consumer. > > > > > > > So when it does the check it returns null for the > > > messageFactory > > > > > > hence > > > > > > > the exception. > > > > > > > > > > > > > > When I print the contents of the map when the exception > occurs > > > I > > > > > > clearly > > > > > > > see that there is an entry of that particular mime type in > the > > > > map. > > > > > > > > > > > > > > This issue seems to happen after a couple of mins at which > > > time > > > > > > around > > > > > > > 50k messages have been exchanged. > > > > > > > > > > > > > > Any ideas as on whats going on? > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Rajith > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 2008-07-22 at 15:07 -0400, Mark Stoodley wrote: > > > > > > > > > > > > > > > > Hit send before seeing the new addition to the > distro :) . > > > > > > > > > > > > > > > > Mark Stoodley > > > > > > > > Technical Lead, WebSphere Real Time > > > > > > > > Mark Stoodley/Toronto/IBM@IBMCA > > > > > > > > mstoodle.com (905) 413-5831 > > > > > > > > ----- Forwarded by Mark Stoodley/Toronto/IBM on > 07/22/2008 > > > > 03:06 > > > > > > PM > > > > > > > > ----- > > > > > > > > Mark Stoodley/Toronto/IBM > > > > > > > > > > > > > > > > 07/22/2008 03:05 PM > > > > > > > > > > > > > > > > > > > > > > > > To > > > > > > > > Darren Hart > > > > > > > > <dvhltc.com> > > > > > > > > cc > > > > > > > > Mike > > > > > > > > Fulton/Toronto/IBM@IBMCA, Paul Ivinson > > > > <paul_ivinson.com>, > > > > > > Clark Williams <williams> > > > > > > > > Subject > > > > > > > > Re: Help with JVM > > > > > > > > memory options > > > > > > > > for Red Hat > > > > > > > > MRGLink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kinda depends on which memory area they're hitting the > limit > > > > on. > > > > > > When > > > > > > > > they hit out of memory, the javacore file that's > generated > > > > should > > > > > > have > > > > > > > > some info in it about which memory area was exhausted. > Once > > > > you > > > > > > > > determine that: > > > > > > > > > > > > > > > > The most likely problem is immortal memory size. The > > > problem > > > > is > > > > > > that > > > > > > > > RTSJ support means classes must be allocated in immortal > > > > memory. > > > > > > The > > > > > > > > most likely option they'll need to add is > > > > > > -Xgc:immortalMemorySize=100m > > > > > > > > (for 100MB of immortal) or some other big number they > like. > > > > > > > > > > > > > > > > Heap size is the next most likely problem: it can be set > via > > > > the > > > > > > > > -Xmx512m to set the heap to 512 MB, for example. > > > > > > > > > > > > > > > > Finally, for completeness, the remaining possibility is > that > > > > they > > > > > > ran > > > > > > > > out of memory in a scope (I doubt this is their problem, > but > > > > you > > > > > > never > > > > > > > > know): in this case their own program had to ask for how > > > much > > > > > > space > > > > > > > > they wanted in the scope memory that became exhausted so > > > they > > > > > > would > > > > > > > > have to increase the requested size. They can increase > the > > > > > > maximum > > > > > > > > allowed scope size with -Xgc:scopedMemoryMaximumSize=16m > (to > > > > > > increase > > > > > > > > the max to 16MB). > > > > > > > > > > > > > > > > You can find out the defaults for these memory area > sizes by > > > > > > running > > > > > > > > "java -Xrealtime -verbose:sizes -version" > > > > > > > > > > > > > > > > Mark Stoodley > > > > > > > > Technical Lead, WebSphere Real Time > > > > > > > > Mark Stoodley/Toronto/IBM@IBMCA > > > > > > > > mstoodle.com (905) 413-5831 > > > > > > > > > > > > > > > > =Comment: #4================================================= Sean Foley <Sean_Foley.com> - 2008-08-08 12:29 EDT The problem is a condition in which a string wrapper and a string being compared to each other are resulting in a false negative. The behaviour of IBM java may be different from Sun java in the hashmap string comparisons ______________________________________________________________________________ Please request Clark Williams choose an appropriate component, as this is certainly not realtime-kernel, but I wasn't sure which of the others were more appropriate.