Description of problem: Customer upgraded their RHN Satellite from 4.2 to 5.3 and migrated API scripts. One of the scripts calls "errata.listPackages" RHN API method on the server. ( HW Description: 8x Intel(R) Xeon(TM) MP CPU 3.66GHz, Mem 6Gb) Customer made some tests using it, in both RHN 4.2 and 5.3 Satellites, and they see a big difference between them - RHN 4.2 takes 5 seconds to run. RHN 5.3 takes 10 minute to run. Check below (minimal reproducer script): $ time ./pkgsrhn42.py real 0m5.480s $ time ./pkgrhn53.py real 1m0.509s Version-Release number of selected component (if applicable): Red Hat Network (RHN) Satellite 5.3.0 How reproducible: Always. Steps to Reproduce: 1. Import customer's 5.3 Satellite database (internal reproducer environment info attached). 2. use errata.getPackages API call (internal reproducer script attached). 3. time it & wait. Actual results: See above for customer's comparison between 4.2 and 5.3 Satellites. On internal 5.3 reproducer, errata.getPackages call takes almost a minute to return for a single errata. Example: [~]# time ./pkgsrhn.py real 0m38.465s user 0m0.140s sys 0m0.015s Expected results: Call returns in reasonable time, no slower (at least not that much slower) than previous version. Additional info: Already discussed with Engineering.
Blocking sat531-triage due to customer business impact: "Since we are about to reach the deadline date to the replacement and this are a blocker for us, we really would like to have a high priority treatment on this issue." In the meantime I'm working on a workaround script for customer. Also, more information from customer's tests: "To make sure we were not being blocked by some special table inside the database, we tested a different method also. This is the script of the second test: #!/usr/bin/python import xmlrpclib rhnuser = 'USERNAME' passwd = 'PASSWORD' url = 'http://localhost/rpc/api' client = xmlrpclib.Server(url, verbose=0) key = client.auth.login(rhnuser, passwd, 3600) adv = 'RHSA-2010:0499' pkgs = client.errata.listPackages(key,adv) #print pkgs print 'Done' The output: [list_packages]$ time ./list_packages_53.py Done real 0m51.554s user 0m0.194s sys 0m0.015s Now running against RHN 4.2 (by changing url): [list_packages]$ time ./list_packages_42.py Done real 0m2.435s user 0m0.204s sys 0m0.022s Again, the difference is impressive between the two servers. == Hardware == These are the specifications for the RHEL 4.2 server: Processors: 4 Intel(R) Xeon(TM) MP CPU 3.00GHz Memory: 5G of RAM, 2 partition of 2GB each for swap Disk: Filesystem Type Size Used Avail Use% Mounted on /dev/os/root ext3 7.9G 1.1G 6.5G 14% / /dev/sda1 ext3 99M 57M 37M 62% /boot /dev/os/home ext3 4.0G 2.1G 1.8G 55% /home /dev/os/opt ext3 12G 7.6G 4.1G 66% /opt none tmpfs 2.5G 0 2.5G 0% /dev/shm /dev/os/tmp ext3 4.0G 39M 3.7G 1% /tmp /dev/os/usr ext3 7.9G 1.7G 5.8G 23% /usr /dev/os/usr_local ext3 3.0G 537M 2.4G 19% /usr/local /dev/os/var ext3 34G 16G 19G 46% /var /dev/os/var_log ext3 7.9G 2.5G 5.1G 33% /var/log /dev/rhndata/rhnsat ext3 99G 43G 55G 44% /rhnsat /dev/rhndata/var_satellite ext3 690G 439G 251G 64% /var/satellite These are the specifications for the RHEL 5.3 server: Processors: 8 Intel(R) Xeon(TM) MP CPU 3.66GHz Memory: 8G of RAM, Disk: Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-root ext3 20G 6.8G 12G 37% / /dev/mapper/VolGroup00-tmp ext3 7.8G 7.4G 0 100% /tmp /dev/mapper/VolGroup00-home ext3 7.8G 1.1G 6.4G 14% /home /dev/mapper/VolGroup00-var ext3 30G 3.5G 25G 13% /var /dev/mapper/VolGroup00-var_log ext3 7.8G 557M 6.9G 8% /var/log /dev/sda1 ext3 190M 29M 152M 16% /boot tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/mapper/VolGroup00-opt ext3 12G 880M 11G 8% /opt /dev/mapper/VolGroup00-usr_local ext3 2.0G 68M 1.9G 4% /usr/local /dev/mapper/vgsan00-var_backup ext3 60G 218M 59G 1% /var/backup /dev/mapper/vgsan00-var_rhnsat ext3 69G 43G 26G 63% /rhnsat /dev/mapper/vgsan00-var_satellite ext3 690G 571G 112G 84% /var/satellite
Created attachment 438291 [details] getErrataPackagesBySynopsis.TEST.1202273.1.py Sample workaround script to bypass the API call.
Per confirmation w/ Justin, this appears due to slowness in hibernate layer. Perhaps changing to a datasource query such as union of the following will help? select p.* from rhnErrata e, rhnErrataPackage ep, rhnPackage p where e.advisory_name = :advisory and e.id = ep.errata_id and p.id = ep.package_id; select p.* from rhnErrataTmp e, rhnErrataPackageTmp ep, rhnPackage p where e.advisory_name = :advisory and e.id = ep.errata_id and p.id = ep.package_id; Relevant code snippets - spacewalk-java-0.5.44/code/src/com/redhat/rhn/frontend/xmlrpc/errata/ErrataHandler.java ... 614 public Object[] listPackages(String sessionKey, String advisoryName) 615 throws FaultException { 616 // Get the logged in user 617 User loggedInUser = getLoggedInUser(sessionKey); 618 Errata errata = lookupErrata(advisoryName, loggedInUser.getOrg()); 619 620 //The set of packages for this erratum 621 Set packages = errata.getPackages(); 622 623 //Main List containing the maps 624 List returnList = new ArrayList(); 625 626 /* 627 * Loop through the packages and add each one to the returnList array 628 */ 629 for (Iterator itr = packages.iterator(); itr.hasNext();) { 630 Package pkg = (Package) itr.next(); 631 // fill out a new row containing the package info map 632 Map pmap = PackageHelper.packageToMap(pkg, loggedInUser); 633 returnList.add(pmap); 634 } 635 636 return returnList.toArray(); 637 } ... 759 private Errata lookupErrata(String advisoryName, Org org) throws FaultException { 760 Errata errata = ErrataManager.lookupByAdvisory(advisoryName); 761 ... spacewalk-java-0.5.44/code/src/com/redhat/rhn/manager/errata/ErrataManager.java ... public static Errata lookupByAdvisory(String advisoryName) { return ErrataFactory.lookupByAdvisory(advisoryName); } ... spacewalk-java-0.5.44/code/src/com/redhat/rhn/domain/errata/ErrataFactory.java ... public static Errata lookupByAdvisory(String advisory) { Session session = null; Errata retval = null; // try { //look for a published errata first session = HibernateFactory.getSession(); retval = (Errata) session.getNamedQuery("PublishedErrata.findByAdvisoryName") .setString("advisory", advisory) .uniqueResult(); //if nothing was found, check the unpublished errata table if (retval == null) { retval = (Errata) session.getNamedQuery("UnpublishedErrata.findByAdvisoryName") .setString("advisory", advisory) .uniqueResult(); } // } // catch (HibernateException e) { // throw new // HibernateRuntimeException("Error looking up errata by advisory name"); // } return retval; } ... spacewalk-java-0.5.44/code/src/com/redhat/rhn/domain/errata/impl/PublishedErrata.hbm.xml ... <class name="com.redhat.rhn.domain.errata.impl.PublishedErrata" table="rhnErrata" > ... <set name="packages" lazy="true" table="rhnErrataPackage" order-by="PACKAGE_ID asc"> <key column="errata_id"/> <many-to-many class="com.redhat.rhn.domain.rhnpackage.Package" column="package_id"/> </set> ... <query name="PublishedErrata.findByAdvisoryName"> <![CDATA[from com.redhat.rhn.domain.errata.impl.PublishedErrata as e where e.advisoryName = :advisory]]> ... spacewalk-java-0.5.44/code/src/com/redhat/rhn/domain/errata/impl/UnpublishedErrata.hbm.xml ... <class name="com.redhat.rhn.domain.errata.impl.UnpublishedErrata" table="rhnErrataTmp" > ... <set name="packages" lazy="true" table="rhnErrataPackageTmp" > <key column="errata_id"/> <many-to-many class="com.redhat.rhn.domain.rhnpackage.Package" column="package_id"/> </set> ... <query name="UnpublishedErrata.findByAdvisoryName"> <![CDATA[from com.redhat.rhn.domain.errata.impl.UnpublishedErrata as e where e.advisoryName = :advisory]]> </query> ...
Event posted on 08-18-2010 01:07pm EDT by jkachuck Hello, Updated request for information from engineering: I do like to get it in only one list." you mean instead of passing it one package id at a time, you would like to pass it a list of package ids? Or is it they want to do an overall query for all packages for a given channel/name/criteria? Thank You Joe Kachuck Internal Status set to 'Waiting on Customer' Status set to: Waiting on Client This event sent from IssueTracker by jkachuck issue 1202273
Event posted on 08-23-2010 10:14am EDT by rvaralda.com Hello Joe, I fixed it using a custom SQL query, and it works now. And looks faster than using the API methods. I hope RH fix these methods, to we using easily. Thanks for your help. BR -Rogério This event sent from IssueTracker by jkachuck issue 1202273
Converted to use data source instead of hibernate: 15d04d2f146546a0ca4fcd1e9bb2224cc824ff47 c943e71d0160f9a622e70cf90b588d46a28aab23
Good catch Dimitar. Looks like the providing channel lookup was at fault too :/ Fixed in master: cc27591d44f60ec1dca516e92eab215485404a3c Will rebuild packages for 5.3
Thanks Justin. Works just perfect. spacewalk-java-0.5.44-84.el5sat Customer DB: real 0m24.921s My DB : real 1m37.134s spacewalk-java-0.5.44-92.el5sat Customer DB: real 0m0.588s My DB : real 0m0.228s Dimi
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0897.html