Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1789522 - On unhealthy Satellite, dynflow_envelopes table might grow indefinitely
Summary: On unhealthy Satellite, dynflow_envelopes table might grow indefinitely
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Tasks Plugin
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.8.0
Assignee: Adam Ruzicka
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-09 18:24 UTC by Adam Ruzicka
Modified: 2023-03-24 16:40 UTC (History)
4 users (show)

Fixed In Version: tfm-rubygem-dynflow-1.4.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 12:59:52 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 29419 0 Normal Closed On unhealthy Satellite, dynflow_envelopes table might grow indefinitely 2020-10-28 17:22:12 UTC
Red Hat Product Errata RHSA-2020:4366 0 None None None 2020-10-27 13:00:06 UTC

Description Adam Ruzicka 2020-01-09 18:24:25 UTC 
Description of problem:
The dynflow_envelopes table is used internally by dynflow for communication between worlds (which roughly map to processes or threads). Basically we use the table as a message bus. That means every row being added to the table should be deleted eventually. However, if recipient of an envelope is not present anymore the envelope will stay there forever.

Having lots of stale envelopes may have negative impact on Satellite's performance.

Version-Release number of selected component (if applicable):


How reproducible:
A bit tricky


Steps to Reproduce:
1. Trigger bunch of tasks
2. Check contents of dynflow_envelopes table
3. systemctl kill -s 9 dynflowd
4. (optional) Go to /foreman_tasks/dynflow > status, click load execution items count (this makes the client world send out some envelopes to executors)
5. Check contents of dynflow_envelopes table

Actual results:
A new record appears in dynflow_envelopes and is never removed from there.

Expected results:
Ideally the record should not appear at all. There are some edge cases where this cannot be achieved. If it does appear, then we should guarrantee that it will eventually be removed from there.

Additional info:
We should probably clean undeliverable envelopes when running task cleanup and during world invalidation. Also we should be able to reduce the overall number of envelopes being sent around.

Maybe adding a check to foreman-maintain and having the number of rows in that table collected by sosreport/foreman-debug would be good idea too.

To see how many undeliverable envelopes there are on the system, run the following query in psql

SELECT COUNT(*) FROM dynflow_envelopes WHERE receiver_id NOT IN (SELECT id FROM dynflow_coordinator_records);

In general it should be safe to replace "SELECT COUNT(*)" with "DELETE" to get things back in order.
Comment 3 Pavel Moravec 2020-01-10 08:25:37 UTC 
Tricky reproducer:

1) Have 3 Content Hosts that trigger Host::Update tasks - each do so 10 times in parallel, just to simulate multiple Content Hosts. So run on each of the 3 systems:

while true; do date; for i in $(seq 1 10); do /usr/libexec/rhsmcertd-worker & done; time wait; sleep 1; done

2) On Satellite, regularly check number of dynflow_envelopes records - all and orphaned ones:

su - postgres -c "psql foreman -c \"copy (select count(*) from dynflow_envelopes) to stdout;\"";
su - postgres -c "psql foreman -c \"copy (select count(*) from dynflow_envelopes where receiver_id NOT IN (SELECT id FROM dynflow_coordinator_records)) to stdout;\""

3) Restart there dynflowd:

service dynflowd restart

4) still monitor dynflow_envelopes size


Observations:
- the dynflow_envelopes grows after restart (as tasks get accumulated), with all the records being valid / receiver_id having a valid record in dynflow_coordinator_records - BUT just for some time.
- within a minute or two, the extra records in dynflow_envelopes become orphaned


Anyway, this is just one possible reproducer, there must be another one _without_ dynflowd restart.
Comment 4 Pavel Moravec 2020-01-10 08:42:16 UTC 
Forgot to mention: "Concurrent Actions::Katello::Host::Update allowed" must be set to Yes (in Administer -> Settings -> Content)  -  or use 30+ systems that run the rhsmcertd-worker sequentially
Comment 5 Pavel Moravec 2020-01-10 09:26:13 UTC 
Oh, the reproducer without dynflowd restart is trivial: just leave the test running, i.e. just trigger up to 10 concurrent Actions::Katello::Host::Update tasks from each of 3 hosts.

Number of planned tasks will grow, as well as number of orphaned dynflow envelopes.
Comment 6 Pavel Moravec 2020-01-10 09:29:12 UTC 
.. and this reproducer is even more evident, when I _disable_ "Concurrent Actions::Katello::Host::Update allowed" (i.e. put to default).
Comment 7 Brad Buckingham 2020-01-10 21:42:43 UTC 
Hi Pavel,

For awareness, we have a change coming for Host::Update that will move the Katello::Host::Update action out of dynflow.  It is being tracked with bug 1771719.

That said, it is possible that the scenario described could be encountered for other actions or scenarios.
Comment 8 Adam Ruzicka 2020-01-21 12:28:37 UTC 
This can indeed be encountered with any other action and in several different scenarios. This one was just the easiest to trigger
Comment 9 Adam Ruzicka 2020-03-26 13:18:08 UTC 
Created redmine issue https://projects.theforeman.org/issues/29419 from this bug
Comment 10 Bryan Kearney 2020-03-26 14:05:21 UTC 
Upstream bug assigned to aruzicka
Comment 11 Bryan Kearney 2020-03-26 14:05:23 UTC 
Upstream bug assigned to aruzicka
Comment 12 Adam Ruzicka 2020-05-06 20:30:38 UTC 
Dynflow 1.4.4 containing fix for this BZ was released in upstream, moving to MODIFIED and setting flags.
Comment 13 Peter Ondrejka 2020-05-25 13:22:39 UTC 
Verified on Sat 6.8 snap 1, using the scenario from description and comment3, seen no rise in dynflow_envelopes count
Comment 16 errata-xmlrpc 2020-10-27 12:59:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366
Note You need to log in before you can comment on or make changes to this bug.