Bug 525339

Summary: Channel cloning performance problem
Product: Red Hat Satellite 5 Reporter: Xixi <xdmoon>
Component: ServerAssignee: Jan Pazdziora <jpazdziora>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Jiri Kastner <jkastner>
Severity: high Docs Contact:
Priority: high    
Version: 530CC: cperry, cvantuin, cwyse, drussell, gbrooksc, jesusr, jkastner, johnh, jpazdziora, mdavis, mmccune, mmraka, mstadtle, mzazrivec, stanislav.polasek, xdmoon
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 472595 Environment:
Last Closed: 2010-07-20 17:20:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 472595    
Bug Blocks:    

Description Xixi 2009-09-23 22:52:10 UTC
Cloning for Satellite 5.3.0.

+++ This bug was initially created as a clone of Bug #472595 +++

Kickstarts from Spacewalk take roughly 2-3 times as long as they should and the load on the Spacewalk server is around 4-5 on rlx-0-04.rhndev.


The slowness starts as the machine is downloading packages from the Spacewalk box.  Watch top and you can see httpd and oracle taking up most of the resources on the box.

--- Additional comment from jsherril on 2009-01-09 16:52:56 EDT ---

I think this may have just been a temporary problem, as the issue has gone away.  Moving to modified just to be tested.

--- Additional comment from mmccune on 2009-01-28 17:54:43 EDT ---

Customers are still reporting this in 0.4.  Re-opening.

--- Additional comment from mmccune on 2009-02-03 18:16:08 EDT ---

Short term workaround for this bug:

* Edit: /etc/httpd/conf.d/zz-spacewalk-server.conf

Add: EnableMMAP off, EnableSendfile off to the Directory stanza:

<Directory "/var/www/html">
        AllowOverride all
        EnableMMAP off
        EnableSendfile off
</Directory>

* Add this index to your database:

# sqlplus spacewalk/spacewalk@xe

SQL> CREATE INDEX rhn_package_path_idx
ON rhnPackage(id, path);

--- Additional comment from jesusr on 2009-02-24 16:06:43 EDT ---



*** This bug has been marked as a duplicate of 470234 ***

--- Additional comment from mmccune on 2009-02-24 17:32:07 EDT ---

*** Bug 470234 has been marked as a duplicate of this bug. ***

--- Additional comment from mmccune on 2009-03-12 01:49:12 EDT ---

i have the java port of the kickstart file downloader 90% complete but it is failing a few of my testcases.  Might not make 0.5 but I may release packages right after 0.5 goes out.

--- Additional comment from mmccune on 2009-03-23 17:19:51 EDT ---

ON_QA

--- Additional comment from mzazrivec on 2009-03-24 06:56:43 EDT ---

Greetings Mike, I think you forgot to create rhn_package_path_idx index
in the schema itself; right now the index is present in the sql upgrade
script (153-rhnPackage-pathidx.sql) only.

--- Additional comment from mmraka on 2009-03-24 09:08:18 EDT ---

FAILS_QA, see comment #8

--- Additional comment from mmccune on 2009-03-25 13:17:20 EDT ---

will fix!

--- Additional comment from mmccune on 2009-03-25 13:28:01 EDT ---

schema fix: 7b974a41d4ec7f7ea0bc02257ee5d90941b71ccb

--- Additional comment from mmccune on 2009-03-25 16:49:11 EDT ---

This actually should still be in FAILS_QA.  Discovered some fatal errors while kickstarting systems so it should not be tested.

--- Additional comment from jesusr on 2009-04-14 10:11:55 EDT ---

Spacewalk 0.5 released.

--- Additional comment from msuchy on 2009-09-17 03:08:39 EDT ---

Spacewalk 0.5 has been released for long time ago.

Comment 1 Xixi 2009-09-23 22:56:28 UTC
Cloning this for Satellite 5.3.0 to make sure the fix s ported to Satellite (5.3.1 ideally) as same issue was reported at a customer's, and the config change above resolved it.

Issue-tracker upcoming.

Comment 3 graeme 2009-10-06 10:47:54 UTC
Hi I am experiencing the same issue as above, only the problem has simply gone away. Are these problems experienced with embedded or standalone or both? I am currently getting them on standalone(external).

Comment 4 Charlie Wyse 2009-10-16 19:32:16 UTC
I was having problems with cloning a channel.  4+ hours to clone 8117 packages on Satellite 5.3.  After modifying the /etc/httpd/conf.d/zz-spacewalk-server.conf and restarting httpd.  We cloned the exact same channel in about 10 seconds.  HUGE difference.

Comment 5 Jan Pazdziora 2009-11-09 14:11:40 UTC
Taking this bugzilla.

Comment 6 Jan Pazdziora 2009-11-10 12:25:08 UTC
So, is this bugzilla about kickstarting performance or about clonning? The original bug 472595 is not about cloning at all. However, the comment 2 and comment 4 talk about cloning, not about kickstarts. And comment 3 is not clear about what the issue actually way.

Comment 7 Akash Chandrashekar 2009-11-10 20:50:15 UTC
Update to this bug. 

The issues experienced on the field have specifically been around the following: 

Slowness on front end and resluting in time out on FRONT-END of Satellite 5.3, when initiating system-group creation and or channel cloning. 

The fix related to making changes in /etc/httpd/conf.d/zz-spacewalk-server.conf and restarting httpd appear to make the issue go away. 

The changes are here : 

<Directory "/var/www/html">
        AllowOverride all
        EnableMMAP off
        EnableSendfile off
</Directory>

Comment 9 Jan Pazdziora 2009-11-11 14:39:56 UTC
I just tried that having

        EnableMMAP off
        EnableSendfile off

does not have any impact on the speed of channel cloning. I've also tried that the speed of channel cloning on 5.3.0 is comparable to 5.2.0 (tested on rlx-1-*).

Comment 10 Jan Pazdziora 2009-11-13 08:33:36 UTC
Per communication on the mailing list, moving from sat531-blockers back to sat531-triage.

Comment 18 Clifford Perry 2010-07-20 17:20:09 UTC
Please re-open if needed.

Comment 19 Matthew Davis 2011-03-23 13:27:52 UTC
I don't believe this should be reopened (for reasons stated at the end), but wanted to share some more information for anybody that hits this in the future.

At a customer site, we had /var/satellite hosted on SAN, and /rhnsat as local storage. We tried to clone RHEL5-64bit, and every clone would cause an oracle process to peg the CPU and we let it sit for a while >30min (the browser would eventually time-out). Bouncing tomcat or all satellite services had no effect. Eventually a channel was cloned (I do not know how long it took, we left for the night).

The next day I added the follow params to zz-spacewalk-www.conf to the /var/www/html directive.

        EnableMMAP off
        EnableSendfile off

And bounced all satellite services just to make sure. And now cloning rhel5 takes under a minute.

I'm suspecting the variable that is used by customers, but not by developers is the remote storage (nfs / SAN / etc). And I realize we specifically state /rhnsat should be on local storage, but the docs are not clear as to where /var/satellite should live. If they are, I couldn't find it.