Bug 508931 - Repo generation fails on s390x ....java/lang/OutOfMemoryError from taskomatic
Summary: Repo generation fails on s390x ....java/lang/OutOfMemoryError from taskomatic
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server
Version: 530
Hardware: s390x
OS: Linux
low
medium
Target Milestone: ---
Assignee: Pradeep Kilambi
QA Contact: John Matthews
URL:
Whiteboard:
Depends On:
Blocks: 485807
TreeView+ depends on / blocked
 
Reported: 2009-06-30 15:14 UTC by John Matthews
Modified: 2009-09-10 18:41 UTC (History)
4 users (show)

Fixed In Version: sat530
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-09-10 18:41:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description John Matthews 2009-06-30 15:14:05 UTC
Description of problem:

Installed Sat 530 on s390x.   I noticed that the repo files were not being generated for al channels, looked at taskomatic logs and saw OutOfMemoryError


Intially I only synced rhel5 i386, repo files were generated for that.
Second I synced x86_64 & s390x.  Now I see OutOfMemoryError


Version-Release number of selected component (if applicable):
Satellite-5.3.0-RHEL5-re20090625.0-s390x-embedded-oracle.iso

How reproducible:
Only tried on one install, but it consistently gets OutOfMemoryError about 10 minutes after taskomatic restart

Steps to Reproduce:
1. Install s390x
2. Sync RHEL 4/5  {s390x, i386, x86_64} including proxy 5.3 and rhn-tools
3. Register a rhel5 x86_64 machine, do a yum install zsh...see repo failure
4. Register a s390x machine, do a yum install zsh...see repo failure

  
Actual results:
Look at taskomatic logs on server.  It might be ~10 minutes or so, taskomatic fails from OutOfMemory.

Expected results:
After period of time repo files get generated.

Additional info:

I bumped the configured java memory and restarted taskomatic and it looks better.  Below is the change I made.

/etc/rhn/default/rhn_taskomatic_daemon.conf
# Initial Java Heap Size (in MB)
#JWM Change 6/30/09, to fix out of memory error with repogen
#wrapper.java.initmemory=32
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
#JWM Change 6/30/09, to fix out of memory error with repogen
#wrapper.java.maxmemory=192
wrapper.java.maxmemory=1024




This is the original error I was seeing with OutOfMemoryError

INFO   | jvm 3    | 2009/06/30 09:15:30 | INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@324c324c [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolD
ataSource@53ce53ce [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checko
utTimeout -> 0, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 53ce53ce, 
idleConnectionTestPeriod -> 300, initialPoolSize -> 5, maxIdleTime -> 300, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange
.v2.c3p0.DriverManagerDataSource@78637863 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 78637863, jdbcUrl -> jdbc:oracle:thin:@localhost:1521:rhnsat, p
roperties -> {user=******, password=******} ], preferredTestQuery -> null, propertyCycle -> 300, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, usesTraditionalReflectiveProxi
es -> false ], factoryClassLocation -> null, identityToken -> 324c324c, numHelperThreads -> 3 ]
INFO   | jvm 3    | 2009/06/30 09:26:35 | JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
INFO   | jvm 3    | 2009/06/30 09:26:35 | JVMDUMP007I JVM Requesting Snap dump using '/usr/sbin/Snap.20090630.092635.26229.0001.trc'
INFO   | jvm 3    | 2009/06/30 09:26:35 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20090630.092635.26229.0001.trc
INFO   | jvm 3    | 2009/06/30 09:26:35 | JVMDUMP007I JVM Requesting Heap dump using '/usr/sbin/heapdump.20090630.092635.26229.0002.phd'
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20090630.092635.26229.0002.phd
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP007I JVM Requesting Java dump using '/usr/sbin/javacore.20090630.092635.26229.0003.txt'
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20090630.092635.26229.0003.txt
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP007I JVM Requesting Snap dump using '/usr/sbin/Snap.20090630.092650.26229.0004.trc'
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20090630.092650.26229.0004.trc
INFO   | jvm 3    | 2009/06/30 09:26:50 | JVMDUMP007I JVM Requesting Heap dump using '/usr/sbin/heapdump.20090630.092650.26229.0005.phd'
INFO   | jvm 3    | 2009/06/30 09:27:04 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20090630.092650.26229.0005.phd
INFO   | jvm 3    | 2009/06/30 09:27:04 | JVMDUMP007I JVM Requesting Java dump using '/usr/sbin/javacore.20090630.092650.26229.0006.txt'
INFO   | jvm 3    | 2009/06/30 09:27:05 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20090630.092650.26229.0006.txt
INFO   | jvm 3    | 2009/06/30 09:27:05 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO   | jvm 3    | 2009/06/30 09:27:05 | Exception in thread "Thread-10" java.lang.InternalError: java.lang.reflect.InvocationTargetException
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at java.lang.Class.copyMethods(Class.java:2406)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at java.lang.Class.lookupCachedMethods(Class.java:2423)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at java.lang.Class.getMethods(Class.java:909)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.util.MethodUtil.callMethod(MethodUtil.java:114)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.addToObject(CachedStatement.java:748)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.processResultSet(CachedStatement.java:598)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:461)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:431)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:335)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:340)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:280)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:105)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.taskomatic.task.repomd.RepomdWriter.getChannelPackageDtoIterator(RepomdWriter.java:92)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter.writeRepomdFiles(RepositoryWriter.java:133)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:91)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at java.lang.Thread.run(Thread.java:735)
INFO   | jvm 3    | 2009/06/30 09:27:05 | Exception in thread "Thread-11" java.lang.OutOfMemoryError
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at java.lang.Object.clone(Native Method)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at java.lang.reflect.Method.getParameterTypes(Method.java:253)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatement.addToObject(CachedStatement.java:719)
INFO   | jvm 3    | 2009/06/30 09:27:05 |       at com.redhat.rhn.common.db.datasource.CachedStatementJVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wai
t.

Comment 2 Clifford Perry 2009-06-30 17:37:58 UTC
Prad - should we worry about these settings, bump them higher for 530 GA? Wish to avoid customers hitting this same issue.

Comment 9 John Matthews 2009-07-23 19:14:42 UTC
VERIFIED
Satellite-5.3.0-RHEL5-re20090714.0-s390x-embedded-oracle.iso

/etc/rhn/default/rhn_taskomatic_daemon.conf
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=512



I synced:
B rhel-i386-server-5
C	rhn-tools-rhel-i386-server-5
 
B rhel-x86_64-server-5
C	redhat-rhn-proxy-5.3-server-x86_64-5
C	rhel-x86_64-server-vt-5
C	rhn-tools-rhel-x86_64-server-5
  
B rhel-s390x-server-5
C	redhat-rhn-proxy-5.3-server-s390x-5
C	rhn-tools-rhel-s390x-server-5


1. deleted the cache /var/cache/rhn/repodata/
2. registered 3 different clients, one each for i386, x86_64, and s390x
3. did a yum install a2ps on each, each failed saying they couldn't fetch repo data.
4. allowed satellite time to generate the repo files, repo files were generated correctly.
5. No errors in taskomatic log
6. each client is able to install packages.

Comment 10 Milan Zázrivec 2009-08-11 11:21:58 UTC
Verified in stage -> RELEASE_PENDING

Comment 11 Brandon Perkins 2009-09-10 18:41:22 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1434.html


Note You need to log in before you can comment on or make changes to this bug.