Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 538559

Summary: Satellite 5.3 webui 500 error and causes high cpu of Oracle/Java when installing errata
Product: Red Hat Satellite 5 Reporter: Akash Chandrashekar <achandra>
Component: WebUIAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Petr Sklenar <psklenar>
Severity: high Docs Contact:
Priority: high    
Version: 530CC: cperry, cvantuin, psklenar, xdmoon
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-01-08 10:02:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 518256    

Description Akash Chandrashekar 2009-11-18 19:48:29 UTC
Description of problem:

Customer is  currently in a POC and experiencing "500 - Internal Server error" any time a push of errata packages is scheduled to a system. 
Total number of Errata was 350 to machine via Satellite. 
Oracle and Java processes spinning at high CPU of 130% -- which then results in 500 time out. 



Version-Release number of selected component (if applicable):

Satellite 5.3 

How reproducible:

100%

Steps to Reproduce:
1.Schedule a push of 350 erratas to systems via webui for "Schedule Action as soon as possible" 

This was done by cloning a base channel, and pushing just 
2. WebUi times out, with Internal 500 Server Error 
3.Top Shows 130% CPU 
  
Actual results:

500 Server Internal Error Message 

Expected results:

Normal response via webui with no time outs. 

Additional info:

UNDO_TBS appears to fill and webui will not recover until space is released. 

Web Trace Included - 


ubject: WEB TRACEBACK from carhsprd1.hottopic.com

java.lang.RuntimeException: XmlRpcException calling cobbler.
	at
com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMetho
d(CobblerXMLRPCHelper.java:72)
	at
com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.j
ava:76)
	at
com.redhat.rhn.taskomatic.task.SingleThreadedTestableTask.execute(Single
ThreadedTestableTask.java:54)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
	at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java
:520)
Caused by: redstone.xmlrpc.XmlRpcException: The response could not be
parsed.
	at
redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:434)
	at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
	at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
	at
com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMetho
d(CobblerXMLRPCHelper.java:69)
	... 4 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:352)
	at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:214)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:201)
	at java.net.Socket.connect(Socket.java:530)
	at java.net.Socket.connect(Socket.java:480)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:169)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:406)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:541)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:658)
	at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnec
tion.java:995)
	at
redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420)
	... 7 more

Comment 1 Akash Chandrashekar 2009-11-18 20:47:26 UTC
Trace back from Oracle 


Tue Nov 17 12:15:09 2009

ORA-00060: Deadlock detected. More info in file /rhnsat/admin/rhnsat/udump/rhnsat_ora_13517.trc.

Tue Nov 17 13:00:44 2009

Thread 1 advanced to log sequence 227 (LGWR switch)

 
DEADLOCK DETECTED ( ORA-00060 )

[Transaction Deadlock]

The following deadlock is not an ORACLE error. It is a

deadlock due to user error in the design of an application

or from issuing incorrect ad-hoc SQL. The following

information may aid in determining the deadlock:

Deadlock graph:

                       ---------Blocker(s)--------  ---------Waiter(s)---------

Resource Name          process session holds waits  process session holds waits

TX-0005002d-00000cda        53     278     X             21     311           S

TX-000a001d-00000cd5        21     311     X             53     278           X

session 278: DID 0001-0035-00000002     session 311: DID 0001-0015-00000003

session 311: DID 0001-0015-00000003     session 278: DID 0001-0035-00000002

Rows waited on:

Session 311: obj - rowid = 000044A4 - AAAESkAAFAAAJf/AAA

  (dictionary objn - 17572, file - 5, block - 38911, slot - 0)

Session 278: obj - rowid = 000041A8 - AAAEGoAALAAAHDYAAB

  (dictionary objn - 16808, file - 11, block - 28888, slot - 1)

Information on the OTHER waiting sessions:

Session 311:

  pid=21 serial=4 audsid=170881 user: 28/RHNSAT

  O/S info: user: tomcat, term: unknown, ospid: 1234, machine: carhsprd1.hottopic.com

            program: JDBC Thin Client

  application name: JDBC Thin Client, hash value=2546894660

  Current SQL Statement:

 

insert into rhnErrata (advisory, advisory_type, product, description, synopsis, topic, solution, issue_date, update_date, notes, refers_to, advisory_name, advisory_rel, locally_modified, org_id, id) values (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, :15, :16)

End of information on OTHER waiting sessions.

Current SQL statement for this session:

UPDATE RHNCHANNEL SET LAST_MODIFIED = :B2 WHERE ID = :B1

Comment 2 Justin Sherrill 2009-11-18 21:26:56 UTC
    That first traceback doesn't really have anything to do with the problem, but I have been able to reproduce that issue (at least the timeout).  It shouldn't be too bad to fix, but will require a code change.   

I'm not sure why you saw a deadlock, possibly you were trying to do something else while the cloning was still happening.  I'll see if i can get a code fix ready.

Comment 3 Justin Sherrill 2009-11-30 18:57:07 UTC
fixed in master:

9f4dbe3d7e6224218a2cbfe5895fec100840c5e7

Comment 8 errata-xmlrpc 2010-01-08 10:02:45 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0021.html