Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1341338 - slow SQL query: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1;
slow SQL query: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_co...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Provisioning (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity medium (vote)
: 6.2.5
: Unused
Assigned To: Tomer Brisker
jcallaha
: Triaged
: 1344709 1408299 1455464 (view as bug list)
Depends On:
Blocks: 1394369
  Show dependency treegraph
 
Reported: 2016-05-31 17:37 EDT by Jan Hutař
Modified: 2017-06-01 07:40 EDT (History)
16 users (show)

See Also:
Fixed In Version: foreman-1.11.0.55-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1394369 (view as bug list)
Environment:
Last Closed: 2016-12-12 12:11:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
content host page load (277.72 KB, image/png)
2016-12-10 21:16 EST, jcallaha
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 12193 None None None 2016-06-02 05:15 EDT
Red Hat Product Errata RHBA-2016:2940 normal SHIPPED_LIVE Satellite 6.2.5 Async Bug Release 2016-12-12 17:08:21 EST

  None (edit)
Description Jan Hutař 2016-05-31 17:37:52 EDT
Description of problem:
There is slow SQL query: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1;


Version-Release number of selected component (if applicable):
satellite-6.2.0-13.0.el7sat.noarch


How reproducible:
sometimes


Steps to Reproduce:
1. Observed on Sat with ~1100 hosts registered with katello-agent/gofferd running (each have 3 tzdata erratas applicable&installable)


Actual results:
This query pops up into the logs from time to time


Expected results:
Should be quick


Additional info:
Extreme case from logs (not this bad every time):

==> /var/lib/pgsql/data/pg_log/postgresql-Tue.log <==
2016-05-31 16:36:01 EDT LOG:  duration: 334153.512 ms  execute <unnamed>: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1
Comment 3 Tomer Brisker 2016-06-02 05:15:19 EDT
Connecting redmine issue http://projects.theforeman.org/issues/12193 from this bug
Comment 4 Tomer Brisker 2016-06-02 05:27:58 EDT
This is an issue caused by rails updating the counter_cache inside the same transaction as the host creation/update. When multiple hosts are updated at the same time, this may cause locks when two hosts attempt to update the same column at the same time. There may be some tuning on the database level that can be done to reduce these locks, but I'm not sure as to precisely which.
Another possibility is to remove these cached counters altogether which will cause slowness on pages that use the counts as each page view will cause several count queries, but may be still within reasonable.
Comment 5 Bryan Kearney 2016-06-02 06:18:14 EDT
Upstream bug component is Provisioning
Comment 6 Chris Duryee 2016-06-15 13:40:06 EDT
*** Bug 1344709 has been marked as a duplicate of this bug. ***
Comment 10 Lukas Zapletal 2016-06-20 08:32:43 EDT
WORKAROUND:

cat >>/etc/httpd/conf.d/passenger.conf <<EOPS
# Allow only one Foreman and one Puppet Master process
<IfModule mod_passenger.c>
PassengerMaxPoolSize 2
</IfModule>
EOPS

This limits number of processes one each, this will slow down processing of requests tho.
Comment 15 Bryan Kearney 2016-07-19 12:05:19 EDT
Moving to POST since upstream bug http://projects.theforeman.org/issues/12193 has been closed
Comment 25 jcallaha 2016-12-10 21:15:26 EST
I've currently got a Satellite with xxx content host records, 1100 of which are active with approximately 150 applicable errata each. 

The content host page loads slightly slower than usual, but not to the point where it is an issue. New content host registration takes roughly the same time. I will attach a screenshot of the content host page with network timeline as well as the log activity during new content host registration.
Comment 26 jcallaha 2016-12-10 21:16 EST
Created attachment 1230456 [details]
content host page load

content host registration logs http://pastebin.test.redhat.com/438183
Comment 28 jcallaha 2016-12-11 11:16:38 EST
Verified in Satellite 6.2.5 Snap 3

Based on the system in comment #25. Enabled debugging mode for postgres, then registered a new host to the system. The captured output does not contain the query outlined in comment #27. Also, the exact query did not show up when editing the architecture of a host. Finally, there was no deviation in performance from what I outlined in #25. 

captured logs during host registration
http://pastebin.test.redhat.com/438201
Comment 29 jcallaha 2016-12-11 14:27:48 EST
and now with the right pastebin link! http://pastebin.test.redhat.com/438203
Comment 31 errata-xmlrpc 2016-12-12 12:11:29 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2940
Comment 32 Tomer Brisker 2016-12-28 08:48:07 EST
*** Bug 1408299 has been marked as a duplicate of this bug. ***
Comment 33 Tomer Brisker 2017-05-25 06:48:49 EDT
*** Bug 1455464 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.