Bug 1463803 - host update task can take some time, leading to growing task backlog
host update task can take some time, leading to growing task backlog
Status: ASSIGNED
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Hosts - Content (Show other bugs)
6.2.9
Unspecified Unspecified
high Severity high (vote)
: GA
: --
Assigned To: Shimon Shtein
Radovan Drazny
: FieldEngineering, PrioBumpField, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-21 15:36 EDT by pm-sat@redhat.com
Modified: 2018-05-18 01:22 EDT (History)
14 users (show)

See Also:
Fixed In Version: tfm-rubygem-katello-3.4.2
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1458857
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 20024 None None None 2017-06-21 15:36 EDT

  None (edit)
Comment 2 Radovan Drazny 2017-11-08 08:50:54 EST
I have tested facts upload using the test procedure from BZ1458857 comment #19 (original of this bug). I used about 40 docker containers registered to Sat 6.3 Snap 21, and started a "subscription-manager facts --update" loop, running on each container in a random interval between 1 and 60 seconds, to spread the load a bit.

Looking at the passenger-status output, Satellite was able to keep up with requests, spawning foreman processes as needed, distributing requests between them. 

Running "subscription manager facts --update" when there is 40 concurrent hosts running gives the following output:

=============================================================================================================
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/8945a4ba-b568-4733-8182-f0533edfa289
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/f760a5dd-6dfe-4e15-8ac3-8c59d11a8c16
Successfully updated the system facts.
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/bdb198a6-1128-4f78-9b9a-30e3dd8e4d45
Successfully updated the system facts.
Successfully updated the system facts.
Successfully updated the system facts.
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/0f440440-f43c-466b-bff9-d2c3d9012d37
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/4621e0e7-2171-416f-974b-5c4e1e048b30
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/4621e0e7-2171-416f-974b-5c4e1e048b30
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/4621e0e7-2171-416f-974b-5c4e1e048b30
=============================================================================================================

There seems to be a problem with many concurrent host updates. Output of "hammer -u admin -p changeme task list --page 1 --per-page 30" is following:
=============================================================================================================
-------------------------------------|------|---------------|---------------------|---------------------|-----------|---------|------------------------------|---------------------------------------------------------------------------------
ID                                   | NAME | OWNER         | STARTED AT          | ENDED AT            | STATE     | RESULT  | TASK ACTION                  | TASK ERRORS
-------------------------------------|------|---------------|---------------------|---------------------|-----------|---------|------------------------------|---------------------------------------------------------------------------------
b09878a1-5bd1-48c1-be5e-cb5d5250abe2 |      |               |                     |                     | scheduled | pending | Insights Email Notifications | N/A
6de7596f-52dc-48ab-a586-d2a89e1a0f4d |      | foreman_admin | 2017/11/08 09:59:17 | 2017/11/08 09:59:17 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
a16f6d93-7703-4be2-9f7b-2e74971bd479 |      | foreman_admin | 2017/11/08 09:59:17 |                     | planned   | pending | Update for host 53f42b1581bf |
890b6bc1-8805-4e9e-af54-81acaea4b98c |      | foreman_admin | 2017/11/08 09:59:16 |                     | running   | pending | Update for host c9bff051760c |
86685be1-fe31-49db-87ae-c43bdec5deb0 |      | foreman_admin | 2017/11/08 09:59:15 | 2017/11/08 09:59:16 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
d18abbed-15b2-49c3-9e7b-520af84b0bf6 |      | foreman_admin | 2017/11/08 09:59:15 |                     | running   | pending | Update for host d6e9fdc32d11 |
e3186e3e-3d13-4f62-a523-a0cb28b7aaa0 |      | foreman_admin | 2017/11/08 09:59:15 |                     | running   | pending | Update for host b25d04b85791 |
5c39b09f-b3ed-40b3-81f8-a9026d4aa082 |      | foreman_admin | 2017/11/08 09:59:14 |                     | running   | pending | Update for host 974dda635c99 |
f785acfa-e327-49d3-9008-fcaf51ac6478 |      | foreman_admin | 2017/11/08 09:59:14 | 2017/11/08 09:59:14 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
2d698fd6-f055-47c9-95ff-2234b0b407c4 |      | foreman_admin | 2017/11/08 09:59:13 |                     | running   | pending | Update for host eda34869a3c2 |
80ec45c0-f7cd-4706-869e-b323625257e5 |      | foreman_admin | 2017/11/08 09:59:13 | 2017/11/08 09:59:13 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
508c952d-5c89-4eec-b2d4-32c0cd9b34dd |      | foreman_admin | 2017/11/08 09:59:11 |                     | running   | pending | Update for host 3e1edd87dc41 |
01497bf9-00f2-4e69-a08b-ca7b4715ce8d |      | foreman_admin | 2017/11/08 09:59:10 |                     | running   | pending | Update for host 8498b3257141 |
a70ea292-0d9f-4917-8679-93920751e7f2 |      | foreman_admin | 2017/11/08 09:59:10 | 2017/11/08 09:59:10 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
bb881a9c-397d-4121-b661-ad08e7315259 |      | foreman_admin | 2017/11/08 09:59:10 | 2017/11/08 09:59:10 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
f3ba9151-c66e-4e6c-ae8e-40998f228f23 |      | foreman_admin | 2017/11/08 09:59:10 |                     | running   | pending | Update for host 0d0a865933d0 |
59ca9c7f-5405-4acd-91a5-af7509da4370 |      | foreman_admin | 2017/11/08 09:59:09 | 2017/11/08 09:59:09 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
d0745f9c-f2e1-4de2-9e3f-769d1f5ca553 |      | foreman_admin | 2017/11/08 09:59:08 | 2017/11/08 09:59:08 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
3e1b49e9-20e8-454e-8161-a9f14c227722 |      | foreman_admin | 2017/11/08 09:59:07 | 2017/11/08 09:59:08 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
ade4971b-8abd-482f-af75-df1ce2722f53 |      | foreman_admin | 2017/11/08 09:59:07 | 2017/11/08 09:59:07 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
dcdc6701-be33-43d3-87b4-174ff55999cc |      | foreman_admin | 2017/11/08 09:59:07 | 2017/11/08 09:59:07 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
0e75e243-9b30-4b86-883f-17e76763e713 |      | foreman_admin | 2017/11/08 09:59:04 | 2017/11/08 09:59:04 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
1ee58000-bdd8-498e-aa3e-f495e599887d |      | foreman_admin | 2017/11/08 09:59:04 | 2017/11/08 09:59:04 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
364f2836-6f3c-4948-990c-82a9e40947f1 |      | foreman_admin | 2017/11/08 09:59:03 | 2017/11/08 09:59:03 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
3a9c4f6d-8a5f-4da8-aaa1-aea0061c2c65 |      | foreman_admin | 2017/11/08 09:59:03 | 2017/11/08 09:59:03 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
b09e8c43-fa43-4d42-a1eb-16f4d08ac7dc |      | foreman_admin | 2017/11/08 09:59:03 | 2017/11/08 09:59:04 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
2d8dbb83-657e-4bcc-b334-df4269902ae3 |      | foreman_admin | 2017/11/08 09:59:01 |                     | running   | pending | Update for host ea223e62eaab |
39d3be60-746d-49bc-8dea-51e5ed9ed6c3 |      | foreman_admin | 2017/11/08 09:59:01 |                     | running   | pending | Update for host 512b1576dfad |
47713fe3-f8e3-4c15-b6fe-c840a7ae5e1b |      | foreman_admin | 2017/11/08 09:59:01 |                     | running   | pending | Update for host 3f411c7e967f |
8d85a963-ef6e-4b3e-9ff7-93f4033c5489 |      | foreman_admin | 2017/11/08 09:59:01 | 2017/11/08 09:59:01 | stopped   | error   | Update for host              | Required lock is already taken by other running tasks.
Please inspect their st...
-------------------------------------|------|---------------|---------------------|---------------------|-----------|---------|------------------------------|------------------------------------------------------
Comment 3 pm-sat@redhat.com 2017-11-08 10:05:11 EST
Upstream bug assigned to sshtein@redhat.com
Comment 4 Ivan Necas 2017-11-09 11:50:09 EST
Can you provide more details on the tasks that were causing the locks errors? Ideally a task export would be helpful.

Note You need to log in before you can comment on or make changes to this bug.