Bug 840593 - Performance issues using SAM
Performance issues using SAM
Status: CLOSED CURRENTRELEASE
Product: Subscription Asset Manager
Classification: Red Hat
Component: katello (Show other bugs)
1.1.0
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Michael Stead
Og Maciel
: Triaged
Depends On:
Blocks: sam12-tracker
  Show dependency treegraph
 
Reported: 2012-07-16 12:04 EDT by Jacob Hunt
Modified: 2013-06-07 13:25 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-07 13:24:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jacob Hunt 2012-07-16 12:04:24 EDT
Description of problem:

The SAM server is very slow to take more than about 10 registrations simultaneously. I'm running the server on a VM with 6G of RAM and 2 CPU cores. The systems tends to use a lot of memory at times getting into swap. I currently have 750 hosts registered/subscribed, but I have about 1000 more to go and I am concerned about the performance.  The web UI has become slow, taking about 60 seconds to switch from the Systems tab to the Dashboard tab.


Version-Release number of selected component (if applicable):

candlepin-0.6.5-1.el6_2.noarch
candlepin-tomcat6-0.6.5-1.el6_2.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.1.5-1.el6_2.noarch
katello-cli-common-0.3.5-2.el6_2.noarch
katello-cli-headpin-0.2.2-1.el6_2.noarch
katello-common-0.3.3-1.el6_2.noarch
katello-configure-0.3.7-1.el6_2.noarch
katello-glue-candlepin-0.3.3-1.el6_2.noarch
katello-headpin-0.2.13-1.el6_2.noarch
katello-headpin-all-0.2.13-1.el6_2.noarch
katello-selinux-0.2.4-1.el6_2.noarch
Comment 4 Bryan Kearney 2012-11-02 10:05:24 EDT
The new CP version is SAM 1.2 should help this.
Comment 5 Bryan Kearney 2012-11-02 10:08:03 EDT
The new candlepi,, 07.17.1, is in the current pudddle.
Comment 6 Og Maciel 2012-11-21 09:01:25 EST
I verified this yesterday.
Comment 7 Mark Huth 2013-02-25 01:18:13 EST
This (long) update is a bit of an FYI (and warning) about applying the hotfix packages ...

Just letting you know that while attempting to resolve BZ887124, we had the customer use the hotfix packages mentioned in comment #2.  They didn't fix the issue, but we eventually got it fixed (see the other BZ).  But anyway, with the hotfix packages the customer reported the following problem:

<customer>
After clicking System->Systename I get following error.

Candlepin::CandlepinResource: 500 Internal Server Error {"displayMessage":"Runtime Error TypeError: Cannot find function getHostConsumer in object org.candlepin.policy.js.entitlement.PreEntHelper@6a8cd8f0. (rules#409) at org.mozilla.javascript.ScriptRuntime.constructError:3,654"} (GET /candlepin/pools/?consumer=b37050ac-5069-4bcc-b1d1-fec1c2e95a3a&listall=false)

This happens when cklicking any system.
</customer>

-------------------------------------------------------------------------------------------------------

It seems in the hotfix packages the PreEntHelper class is missing the getHostConsumer method.

Looking at the original candlepin packages on SAM1.1 I can see that method:
<original_packages>
[root@sam entitlement]# rpm -qa | grep candlepin
candlepin-0.6.5-1.el6_2.noarch
candlepin-tomcat6-0.6.5-1.el6_2.noarch

[root@sam ~]# locate PreEntHelper.class
/var/lib/tomcat6/webapps/candlepin/WEB-INF/classes/org/candlepin/policy/js/entitlement/PreEntHelper.class

[root@sam ~]# locate PreEntHelper.class | xargs rpm -qf
candlepin-tomcat6-0.6.5-1.el6_2.noarch

[root@sam ~]# locate PreEntHelper.class | xargs grep -a getHostConsumer
addWarning	getResult)()Lorg/candlepin/policy/ValidationResult;
StackMapTablegetHostConsumer
(Ljava/lang/String;)Lorg/candlepin/model/Consumer;guestId/ReadOnlyPool;

... and the getHostConsumer method is there
</original_packages>


But looking at the hotfix packages I can see that it is missing:

<hotfix_packages>
[root@vm89 ~]# rpm -qa | grep candlepin
candlepin-0.6.7.1-1.el6_2.noarch
candlepin-tomcat6-0.6.7.1-1.el6_2.noarch

[root@vm89 ~]# locate PreEntHelper.class
/var/lib/tomcat6/webapps/candlepin/WEB-INF/classes/org/candlepin/policy/js/entitlement/PreEntHelper.class
[root@vm89 ~]# locate PreEntHelper.class | xargs rpm -qf
candlepin-tomcat6-0.6.7.1-1.el6_2.noarch
[root@vm89 ~]# locate PreEntHelper.class | xargs grep -a getHostConsumer
[root@vm89 ~]# 

... getHostConsumer method is missing.
</hotfix_packages>

-------------------------------------------------------------------------------------------------------

So I tried adding the method back into the hotfix packages:

<attempt_to_add_missing_method>
Ok, I was able to get a copy of the hotfix source package here:
https://brewweb.devel.redhat.com/buildinfo?buildID=225464

# rpm -ivh candlepin-0.6.7.1-1.el6_2.src.rpm
...
/root/rpmbuild/SOURCES/candlepin-0.6.7.1/src/main/java/org/candlepin/policy/js/entitlement/PreEntHelper.java

However, its not as easy as just adding the missing getHostConsumer method to the PreEntHelper.java file.

    public Consumer getHostConsumer(String guestId) {
        return consumerCurator.getHost(guestId);
    }

[root@vm89 entitlement]# javac -extdirs /var/lib/tomcat6/webapps/candlepin/WEB-INF/lib/ -classpath /var/lib/tomcat6/webapps/candlepin/WEB-INF/classes PreEntHelper.java
PreEntHelper.java:105: cannot find symbol
symbol  : variable consumerCurator
location: class org.candlepin.policy.js.entitlement.PreEntHelper
        return consumerCurator.getHost(guestId);
               ^
1 error

The way the PreEntHelper class is constructed is different between the SAM 1.1 release containing candlepin-tomcat6-0.6.5-1.el6_2.noarch and the candlepin-tomcat6-0.6.7.1-1.el6_2.noarch in the hotfix package.  The hotfix release no longer has reference to Consumers and ConsumerCurators.
</attempt_to_add_missing_method>

-------------------------------------------------------------------------------------------------------

So I had to give up trying to re-add the method and just get the customer to revert back to the original candlepin packages on SAM1.1:

# yum downgrade candlepin candlepin-selinux candlepin-tomcat6
... which installs the original versions

Currently waiting for the customers feedback that this works.  I was a little concerned about what change this introduced: /usr/share/candlepin/cpdb --update in comment #2, but it seems that it only added indexes to the foreign keys, so I'm assuming it will be ok to just downgrade the packages and the earlier version of candlepin won't have any problems in interacting with the DB, even though the DB was updated.

-- Mark
Comment 8 Devan Goodwin 2013-02-25 13:22:10 EST
Mark this is definitely a separate issue and could be filed as such, but I believe I can provide a workaround. 

We can see this breaking in commit: 85703e05c8fa4ae6cb5d949f5caafcbf2953fce0 which was done in the hotfix branch between 0.6.6 and 0.6.7. The java to javascript API changed in that something was removed, but the customer's deployment likely already has a newer rules file in their database from a manifest import.

I believe you can safely workaround by having customer delete whatever is in the cp_rules table in the candlepin postgresql database. This would force the server to use the rules file included in the RPM. I don't know the credentials in a SAM deployment to hit the Candlepin database but once in, it should just be a simple "delete from cp_rules;" After this they should be able to use the new hotfix package.
Comment 9 Mark Huth 2013-03-05 02:39:17 EST
Thanks for the update Devan.  I have passed this information onto the customer and am waiting on their feedback.  I'm not sure if they have already downgraded the candlepin packages or if they will try the SQL.  Will let you know.

BTW I have filed another BZ for this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=917947

Thanks,
Mark
Comment 10 Mark Huth 2013-03-20 15:10:11 EDT
Hi Devan,

The customer has confirmed that running "delete from cp_rules;" fixed the problem for them.  They didn't need to downgrade the hotfix packages back to the GA packages.

Thanks for the help,
Mark
Comment 11 Bryan Kearney 2013-06-07 13:24:19 EDT
Closing out verified bugs from 1.2. Somehow the errata tool missed them.
Comment 12 Bryan Kearney 2013-06-07 13:25:34 EDT
Closing out verified bugs from 1.2. Somehow the errata tool missed them.

Note You need to log in before you can comment on or make changes to this bug.