I have tested facts upload using the test procedure from BZ1458857 comment #19 (original of this bug). I used about 40 docker containers registered to Sat 6.3 Snap 21, and started a "subscription-manager facts --update" loop, running on each container in a random interval between 1 and 60 seconds, to spread the load a bit.
Looking at the passenger-status output, Satellite was able to keep up with requests, spawning foreman processes as needed, distributing requests between them.
Running "subscription manager facts --update" when there is 40 concurrent hosts running gives the following output:
=============================================================================================================
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/8945a4ba-b568-4733-8182-f0533edfa289
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/f760a5dd-6dfe-4e15-8ac3-8c59d11a8c16
Successfully updated the system facts.
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/bdb198a6-1128-4f78-9b9a-30e3dd8e4d45
Successfully updated the system facts.
Successfully updated the system facts.
Successfully updated the system facts.
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/0f440440-f43c-466b-bff9-d2c3d9012d37
Successfully updated the system facts.
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/4621e0e7-2171-416f-974b-5c4e1e048b30
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/4621e0e7-2171-416f-974b-5c4e1e048b30
Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: update
Conflicts with tasks:
- https://dell-per415-02.dsal.lab.eng.bos.redhat.com/foreman_tasks/tasks/4621e0e7-2171-416f-974b-5c4e1e048b30
=============================================================================================================
There seems to be a problem with many concurrent host updates. Output of "hammer -u admin -p changeme task list --page 1 --per-page 30" is following:
=============================================================================================================
-------------------------------------|------|---------------|---------------------|---------------------|-----------|---------|------------------------------|---------------------------------------------------------------------------------
ID | NAME | OWNER | STARTED AT | ENDED AT | STATE | RESULT | TASK ACTION | TASK ERRORS
-------------------------------------|------|---------------|---------------------|---------------------|-----------|---------|------------------------------|---------------------------------------------------------------------------------
b09878a1-5bd1-48c1-be5e-cb5d5250abe2 | | | | | scheduled | pending | Insights Email Notifications | N/A
6de7596f-52dc-48ab-a586-d2a89e1a0f4d | | foreman_admin | 2017/11/08 09:59:17 | 2017/11/08 09:59:17 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
a16f6d93-7703-4be2-9f7b-2e74971bd479 | | foreman_admin | 2017/11/08 09:59:17 | | planned | pending | Update for host 53f42b1581bf |
890b6bc1-8805-4e9e-af54-81acaea4b98c | | foreman_admin | 2017/11/08 09:59:16 | | running | pending | Update for host c9bff051760c |
86685be1-fe31-49db-87ae-c43bdec5deb0 | | foreman_admin | 2017/11/08 09:59:15 | 2017/11/08 09:59:16 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
d18abbed-15b2-49c3-9e7b-520af84b0bf6 | | foreman_admin | 2017/11/08 09:59:15 | | running | pending | Update for host d6e9fdc32d11 |
e3186e3e-3d13-4f62-a523-a0cb28b7aaa0 | | foreman_admin | 2017/11/08 09:59:15 | | running | pending | Update for host b25d04b85791 |
5c39b09f-b3ed-40b3-81f8-a9026d4aa082 | | foreman_admin | 2017/11/08 09:59:14 | | running | pending | Update for host 974dda635c99 |
f785acfa-e327-49d3-9008-fcaf51ac6478 | | foreman_admin | 2017/11/08 09:59:14 | 2017/11/08 09:59:14 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
2d698fd6-f055-47c9-95ff-2234b0b407c4 | | foreman_admin | 2017/11/08 09:59:13 | | running | pending | Update for host eda34869a3c2 |
80ec45c0-f7cd-4706-869e-b323625257e5 | | foreman_admin | 2017/11/08 09:59:13 | 2017/11/08 09:59:13 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
508c952d-5c89-4eec-b2d4-32c0cd9b34dd | | foreman_admin | 2017/11/08 09:59:11 | | running | pending | Update for host 3e1edd87dc41 |
01497bf9-00f2-4e69-a08b-ca7b4715ce8d | | foreman_admin | 2017/11/08 09:59:10 | | running | pending | Update for host 8498b3257141 |
a70ea292-0d9f-4917-8679-93920751e7f2 | | foreman_admin | 2017/11/08 09:59:10 | 2017/11/08 09:59:10 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
bb881a9c-397d-4121-b661-ad08e7315259 | | foreman_admin | 2017/11/08 09:59:10 | 2017/11/08 09:59:10 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
f3ba9151-c66e-4e6c-ae8e-40998f228f23 | | foreman_admin | 2017/11/08 09:59:10 | | running | pending | Update for host 0d0a865933d0 |
59ca9c7f-5405-4acd-91a5-af7509da4370 | | foreman_admin | 2017/11/08 09:59:09 | 2017/11/08 09:59:09 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
d0745f9c-f2e1-4de2-9e3f-769d1f5ca553 | | foreman_admin | 2017/11/08 09:59:08 | 2017/11/08 09:59:08 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
3e1b49e9-20e8-454e-8161-a9f14c227722 | | foreman_admin | 2017/11/08 09:59:07 | 2017/11/08 09:59:08 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
ade4971b-8abd-482f-af75-df1ce2722f53 | | foreman_admin | 2017/11/08 09:59:07 | 2017/11/08 09:59:07 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
dcdc6701-be33-43d3-87b4-174ff55999cc | | foreman_admin | 2017/11/08 09:59:07 | 2017/11/08 09:59:07 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
0e75e243-9b30-4b86-883f-17e76763e713 | | foreman_admin | 2017/11/08 09:59:04 | 2017/11/08 09:59:04 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
1ee58000-bdd8-498e-aa3e-f495e599887d | | foreman_admin | 2017/11/08 09:59:04 | 2017/11/08 09:59:04 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
364f2836-6f3c-4948-990c-82a9e40947f1 | | foreman_admin | 2017/11/08 09:59:03 | 2017/11/08 09:59:03 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
3a9c4f6d-8a5f-4da8-aaa1-aea0061c2c65 | | foreman_admin | 2017/11/08 09:59:03 | 2017/11/08 09:59:03 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
b09e8c43-fa43-4d42-a1eb-16f4d08ac7dc | | foreman_admin | 2017/11/08 09:59:03 | 2017/11/08 09:59:04 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
2d8dbb83-657e-4bcc-b334-df4269902ae3 | | foreman_admin | 2017/11/08 09:59:01 | | running | pending | Update for host ea223e62eaab |
39d3be60-746d-49bc-8dea-51e5ed9ed6c3 | | foreman_admin | 2017/11/08 09:59:01 | | running | pending | Update for host 512b1576dfad |
47713fe3-f8e3-4c15-b6fe-c840a7ae5e1b | | foreman_admin | 2017/11/08 09:59:01 | | running | pending | Update for host 3f411c7e967f |
8d85a963-ef6e-4b3e-9ff7-93f4033c5489 | | foreman_admin | 2017/11/08 09:59:01 | 2017/11/08 09:59:01 | stopped | error | Update for host | Required lock is already taken by other running tasks.
Please inspect their st...
-------------------------------------|------|---------------|---------------------|---------------------|-----------|---------|------------------------------|------------------------------------------------------
Comment 3Satellite Program
2017-11-08 15:05:11 UTC