Bug 611846

Summary:	Test bundle subsystem with large bundles
Product:	[Other] RHQ Project	Reporter:	Charles Crouch <ccrouch>
Component:	Content	Assignee:	RHQ Project Maintainer <rhq-maint>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Filip Drabek <fdrabek>
Severity:	medium	Docs Contact:
Priority:	urgent
Version:	unspecified	CC:	cwelton, fdrabek, hbrock, jshaughn
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	2.4	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-08-12 16:54:47 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	601949

Description Charles Crouch 2010-07-06 16:37:05 UTC

We need to make sure that no part of the bundle system is susceptible to problems due to large bundle files:

Couple of areas we need to test 
1) No OOM errors on agent or server, basically that we don't read the whole bundle file into memory while processing it, this applies to both server and agent
2) Database txn timeouts, make sure that if an upload of a bundle or download of a bundle to an agent takes a long time, due to it being large or the link being slow that we don't end up with a txn timeout, e.g. after 10mins

Test cases
1)
a) Upload a 1gb bundle using all of the various upload mechanisms
b) Install 1gb bundle on an agent

2)
a) Upload a 1gb bundle using all of the various upload mechanisms, where the source of the bundle (url, browser) does not share a fast network with the JON server, e.g. browser upload from laptop to RH based jon server, RH url upload to laptop based jon server
b) Install 1gb bundle on an agent which does not share a fast network with the JON server

Comment 1 Charles Crouch 2010-07-09 16:39:37 UTC

Setting directly to ON_QA

Comment 2 Jay Shaughnessy 2010-07-09 17:44:24 UTC

Note: to really test a huge file going from server to agent then what is important is that you have a large file in the distribution, not just a large distribution file. This is because each bundle file will be dragged over individually.  Meaning, the 1G distribution zip would need to contain probably 1 huge bundle file (maybe a zip of like 3 EAPs or something :) and the recipe file.

Comment 3 Filip Drabek 2010-07-14 20:24:21 UTC

I have 1GB file to upload via web. 
After 3 minutes I'm getting this exception:
HTTP Status 500 -

org.rhq.enterprise.server.auth.SessionNotFoundException
	org.rhq.enterprise.server.auth.SessionManager.getSubject(SessionManager.java:167)
	org.rhq.enterprise.server.auth.SubjectManagerBean.getSubjectBySessionId(SubjectManagerBean.java:568)
	sun.reflect.GeneratedMethodAccessor993.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)

Content is loaded to RAM (increase about 870MB).

Comment 4 Jay Shaughnessy 2010-07-15 20:58:33 UTC

I'm fairly sure this large-bundle load failure is not a bug but probably a manifestation of our shared session id behavior.  Make sure you absolutely don't have any other rhq browser windows open. Restart the server, start a gui portal war session, nav to bundles and try the upload.

If the coregui has a valid sessionid this issue does not appear for me no matter what size bundle I use.  If I invalidate the user's session I can recreate it with any size bundle.  (note - to invalidate the session: in another browser tab log into a new gui session as the same user, then log out of it. Then, go back to your bundle test tab, it's session is now invalid.)  

Assuming the upload now works the test can continue.

Comment 5 Jay Shaughnessy 2010-07-15 21:33:17 UTC

I was able to deploy a bundle with a bundle file ~850MB. It wasn't fast, the streaming into and out of the db is a bit cumbersome, but it succeeded.

Comment 6 Filip Drabek 2010-07-19 12:22:28 UTC

When bundle site is opened and there is no activity for 30 minutes the session is invalidated, but because there is no callback to server and no data to show when you open the bundle site for first time you will not recognize that your session is invalidate. When you want to create a new bundle with already invalidate session the window will be opened because it is javascript code without callback to server and when you will choose file and click on button 'Next', uploading will be processed and when 100% is uploaded you will get message: HTTP Status 500 -javax.servlet.ServletException: Cannot authenticate request, org.rhq.enterprise.server.auth.SessionNotFoundException.

Comment 7 Corey Welton 2010-07-20 20:14:38 UTC

Closing this one out, following testing.  Dev will open a separate bz on the timeout issue.

Comment 8 Corey Welton 2010-08-12 16:54:47 UTC

Mass-closure of verified bugs against JON.