Bug 1341338 - slow SQL query: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1;
Summary: slow SQL query: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_co...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Provisioning
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: Unspecified
Assignee: Tomer Brisker
QA Contact: jcallaha
URL:
Whiteboard:
: 1344709 1408299 1455464 (view as bug list)
Depends On:
Blocks: 1394369
TreeView+ depends on / blocked
 
Reported: 2016-05-31 21:37 UTC by Jan Hutař
Modified: 2020-07-16 08:47 UTC (History)
16 users (show)

Fixed In Version: foreman-1.11.0.55-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1394369 (view as bug list)
Environment:
Last Closed: 2016-12-12 17:11:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
content host page load (277.72 KB, image/png)
2016-12-11 02:16 UTC, jcallaha
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 12193 0 Normal Closed Deadlock occuring when creating host 2020-10-14 17:19:13 UTC
Red Hat Product Errata RHBA-2016:2940 0 normal SHIPPED_LIVE Satellite 6.2.5 Async Bug Release 2016-12-12 22:08:21 UTC

Description Jan Hutař 2016-05-31 21:37:52 UTC
Description of problem:
There is slow SQL query: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1;


Version-Release number of selected component (if applicable):
satellite-6.2.0-13.0.el7sat.noarch


How reproducible:
sometimes


Steps to Reproduce:
1. Observed on Sat with ~1100 hosts registered with katello-agent/gofferd running (each have 3 tzdata erratas applicable&installable)


Actual results:
This query pops up into the logs from time to time


Expected results:
Should be quick


Additional info:
Extreme case from logs (not this bad every time):

==> /var/lib/pgsql/data/pg_log/postgresql-Tue.log <==
2016-05-31 16:36:01 EDT LOG:  duration: 334153.512 ms  execute <unnamed>: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1

Comment 3 Tomer Brisker 2016-06-02 09:15:19 UTC
Connecting redmine issue http://projects.theforeman.org/issues/12193 from this bug

Comment 4 Tomer Brisker 2016-06-02 09:27:58 UTC
This is an issue caused by rails updating the counter_cache inside the same transaction as the host creation/update. When multiple hosts are updated at the same time, this may cause locks when two hosts attempt to update the same column at the same time. There may be some tuning on the database level that can be done to reduce these locks, but I'm not sure as to precisely which.
Another possibility is to remove these cached counters altogether which will cause slowness on pages that use the counts as each page view will cause several count queries, but may be still within reasonable.

Comment 5 Bryan Kearney 2016-06-02 10:18:14 UTC
Upstream bug component is Provisioning

Comment 6 Chris Duryee 2016-06-15 17:40:06 UTC
*** Bug 1344709 has been marked as a duplicate of this bug. ***

Comment 10 Lukas Zapletal 2016-06-20 12:32:43 UTC
WORKAROUND:

cat >>/etc/httpd/conf.d/passenger.conf <<EOPS
# Allow only one Foreman and one Puppet Master process
<IfModule mod_passenger.c>
PassengerMaxPoolSize 2
</IfModule>
EOPS

This limits number of processes one each, this will slow down processing of requests tho.

Comment 15 Bryan Kearney 2016-07-19 16:05:19 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/12193 has been closed

Comment 25 jcallaha 2016-12-11 02:15:26 UTC
I've currently got a Satellite with xxx content host records, 1100 of which are active with approximately 150 applicable errata each. 

The content host page loads slightly slower than usual, but not to the point where it is an issue. New content host registration takes roughly the same time. I will attach a screenshot of the content host page with network timeline as well as the log activity during new content host registration.

Comment 26 jcallaha 2016-12-11 02:16:17 UTC
Created attachment 1230456 [details]
content host page load

content host registration logs http://pastebin.test.redhat.com/438183

Comment 28 jcallaha 2016-12-11 16:16:38 UTC
Verified in Satellite 6.2.5 Snap 3

Based on the system in comment #25. Enabled debugging mode for postgres, then registered a new host to the system. The captured output does not contain the query outlined in comment #27. Also, the exact query did not show up when editing the architecture of a host. Finally, there was no deviation in performance from what I outlined in #25. 

captured logs during host registration
http://pastebin.test.redhat.com/438201

Comment 29 jcallaha 2016-12-11 19:27:48 UTC
and now with the right pastebin link! http://pastebin.test.redhat.com/438203

Comment 31 errata-xmlrpc 2016-12-12 17:11:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2940

Comment 32 Tomer Brisker 2016-12-28 13:48:07 UTC
*** Bug 1408299 has been marked as a duplicate of this bug. ***

Comment 33 Tomer Brisker 2017-05-25 10:48:49 UTC
*** Bug 1455464 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.