Bug 1789522

Summary:	On unhealthy Satellite, dynflow_envelopes table might grow indefinitely
Product:	Red Hat Satellite	Reporter:	Adam Ruzicka <aruzicka>
Component:	Tasks Plugin	Assignee:	Adam Ruzicka <aruzicka>
Status:	CLOSED ERRATA	QA Contact:	Peter Ondrejka <pondrejk>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.6.0	CC:	aruzicka, bbuckingham, egolov, pmoravec
Target Milestone:	6.8.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-dynflow-1.4.4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 12:59:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Adam Ruzicka 2020-01-09 18:24:25 UTC

Description of problem:
The dynflow_envelopes table is used internally by dynflow for communication between worlds (which roughly map to processes or threads). Basically we use the table as a message bus. That means every row being added to the table should be deleted eventually. However, if recipient of an envelope is not present anymore the envelope will stay there forever.

Having lots of stale envelopes may have negative impact on Satellite's performance.

Version-Release number of selected component (if applicable):


How reproducible:
A bit tricky


Steps to Reproduce:
1. Trigger bunch of tasks
2. Check contents of dynflow_envelopes table
3. systemctl kill -s 9 dynflowd
4. (optional) Go to /foreman_tasks/dynflow > status, click load execution items count (this makes the client world send out some envelopes to executors)
5. Check contents of dynflow_envelopes table

Actual results:
A new record appears in dynflow_envelopes and is never removed from there.

Expected results:
Ideally the record should not appear at all. There are some edge cases where this cannot be achieved. If it does appear, then we should guarrantee that it will eventually be removed from there.

Additional info:
We should probably clean undeliverable envelopes when running task cleanup and during world invalidation. Also we should be able to reduce the overall number of envelopes being sent around.

Maybe adding a check to foreman-maintain and having the number of rows in that table collected by sosreport/foreman-debug would be good idea too.

To see how many undeliverable envelopes there are on the system, run the following query in psql

SELECT COUNT(*) FROM dynflow_envelopes WHERE receiver_id NOT IN (SELECT id FROM dynflow_coordinator_records);

In general it should be safe to replace "SELECT COUNT(*)" with "DELETE" to get things back in order.

Comment 3 Pavel Moravec 2020-01-10 08:25:37 UTC

Tricky reproducer:

1) Have 3 Content Hosts that trigger Host::Update tasks - each do so 10 times in parallel, just to simulate multiple Content Hosts. So run on each of the 3 systems:

while true; do date; for i in $(seq 1 10); do /usr/libexec/rhsmcertd-worker & done; time wait; sleep 1; done

2) On Satellite, regularly check number of dynflow_envelopes records - all and orphaned ones:

su - postgres -c "psql foreman -c \"copy (select count(*) from dynflow_envelopes) to stdout;\"";
su - postgres -c "psql foreman -c \"copy (select count(*) from dynflow_envelopes where receiver_id NOT IN (SELECT id FROM dynflow_coordinator_records)) to stdout;\""

3) Restart there dynflowd:

service dynflowd restart

4) still monitor dynflow_envelopes size


Observations:
- the dynflow_envelopes grows after restart (as tasks get accumulated), with all the records being valid / receiver_id having a valid record in dynflow_coordinator_records - BUT just for some time.
- within a minute or two, the extra records in dynflow_envelopes become orphaned


Anyway, this is just one possible reproducer, there must be another one _without_ dynflowd restart.

Comment 4 Pavel Moravec 2020-01-10 08:42:16 UTC

Forgot to mention: "Concurrent Actions::Katello::Host::Update allowed" must be set to Yes (in Administer -> Settings -> Content)  -  or use 30+ systems that run the rhsmcertd-worker sequentially

Comment 5 Pavel Moravec 2020-01-10 09:26:13 UTC

Oh, the reproducer without dynflowd restart is trivial: just leave the test running, i.e. just trigger up to 10 concurrent Actions::Katello::Host::Update tasks from each of 3 hosts.

Number of planned tasks will grow, as well as number of orphaned dynflow envelopes.

Comment 6 Pavel Moravec 2020-01-10 09:29:12 UTC

.. and this reproducer is even more evident, when I _disable_ "Concurrent Actions::Katello::Host::Update allowed" (i.e. put to default).

Comment 7 Brad Buckingham 2020-01-10 21:42:43 UTC

Hi Pavel,

For awareness, we have a change coming for Host::Update that will move the Katello::Host::Update action out of dynflow.  It is being tracked with bug 1771719.

That said, it is possible that the scenario described could be encountered for other actions or scenarios.

Comment 8 Adam Ruzicka 2020-01-21 12:28:37 UTC

This can indeed be encountered with any other action and in several different scenarios. This one was just the easiest to trigger

Comment 9 Adam Ruzicka 2020-03-26 13:18:08 UTC

Created redmine issue https://projects.theforeman.org/issues/29419 from this bug

Comment 10 Bryan Kearney 2020-03-26 14:05:21 UTC

Upstream bug assigned to aruzicka

Comment 11 Bryan Kearney 2020-03-26 14:05:23 UTC

Upstream bug assigned to aruzicka

Comment 12 Adam Ruzicka 2020-05-06 20:30:38 UTC

Dynflow 1.4.4 containing fix for this BZ was released in upstream, moving to MODIFIED and setting flags.

Comment 13 Peter Ondrejka 2020-05-25 13:22:39 UTC

Verified on Sat 6.8 snap 1, using the scenario from description and comment3, seen no rise in dynflow_envelopes count

Comment 16 errata-xmlrpc 2020-10-27 12:59:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366