1410735 – dynflow_executor memory increase triggered by capsule sync

Bug 1410735 - dynflow_executor memory increase triggered by capsule sync

Summary: dynflow_executor memory increase triggered by capsule sync

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Tasks Plugin
Sub Component:
Version:	6.2.5
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	Unspecified
Assignee:	satellite6-bugs
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-01-06 09:42 UTC by Pavel Moravec
Modified:	2021-08-30 13:39 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-01-10 09:49:57 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1362168	0	high	CLOSED	Memory leak in several processes (mainly dynflow_executor) when (un)installing a package in loop	2024-03-25 14:56:55 UTC
Red Hat Knowledge Base (Solution)	2852821	0	None	None	None	2017-01-10 09:49:56 UTC

Internal Links: 1362168

Description Pavel Moravec 2017-01-06 09:42:06 UTC

Description of problem:
Running (external) Capsule sync repeatedly, dynflow_executor process consumes more and more memory.

Maybe the cause is again in ruby garbage collector like in https://bugzilla.redhat.com/show_bug.cgi?id=1362168 


Version-Release number of selected component (if applicable):
Sat 6.2.4
tfm-rubygem-foreman-tasks-0.7.14.9-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Have some external Capsule
2. Have some synced repos from CDN (published in a lifecycle env. associated to the Capsule, such that the repos are to be synced to the Capsule)
3. Repeatedly sync the Capsule (even though each sync wont update anything since everything is already synced):

while true; do time hammer -u admin -p redhat capsule content synchronize --id=2; sleep 10; done

4. monitor "ps aux" output regularly


Actual results:
4. shows "dynflow_executor" process memory growth. Particular data from my system where one capsule sync took approx. 1.5 minutes and "ps aux" was taken every 5 minutes:

# grep dynflow_executor$ monitor_ps.txt.capsuleSync 
foreman   2991  0.8  5.8 1790400 709912 ?      Sl   Jan02  37:38 dynflow_executor
foreman   2991  0.8  5.8 1790400 708916 ?      Sl   Jan02  37:49 dynflow_executor
foreman   2991  0.8  5.8 1790400 710964 ?      Sl   Jan02  38:00 dynflow_executor
foreman   2991  0.8  5.8 1790400 711172 ?      Sl   Jan02  38:11 dynflow_executor
foreman   2991  0.8  5.8 1790400 710972 ?      Sl   Jan02  38:21 dynflow_executor
foreman   2991  0.8  5.8 1790400 713704 ?      Sl   Jan02  38:34 dynflow_executor
foreman   2991  0.8  5.8 1790400 713312 ?      Sl   Jan02  38:42 dynflow_executor
foreman   2991  0.8  5.8 1790400 716400 ?      Sl   Jan02  38:55 dynflow_executor
foreman   2991  0.8  5.8 1790400 716416 ?      Sl   Jan02  39:05 dynflow_executor
foreman   2991  0.8  5.8 1790400 718200 ?      Sl   Jan02  39:16 dynflow_executor
foreman   2991  0.8  5.9 1790400 720728 ?      Sl   Jan02  39:26 dynflow_executor
foreman   2991  0.8  5.9 1790400 722304 ?      Sl   Jan02  39:39 dynflow_executor
foreman   2991  0.9  5.8 1790400 712220 ?      Sl   Jan02  39:51 dynflow_executor
foreman   2991  0.9  5.8 1790400 714136 ?      Sl   Jan02  40:00 dynflow_executor
foreman   2991  0.9  5.8 1790400 716872 ?      Sl   Jan02  40:11 dynflow_executor
foreman   2991  0.9  5.8 1790400 715332 ?      Sl   Jan02  40:21 dynflow_executor
foreman   2991  0.9  5.8 1790812 715736 ?      Sl   Jan02  40:30 dynflow_executor
foreman   2991  0.9  5.8 1791488 716372 ?      Sl   Jan02  40:43 dynflow_executor
foreman   2991  0.9  5.8 1791488 716376 ?      Sl   Jan02  40:54 dynflow_executor
foreman   2991  0.9  5.9 1791488 722348 ?      Sl   Jan02  41:04 dynflow_executor
foreman   2991  0.9  5.9 1791488 722380 ?      Sl   Jan02  41:15 dynflow_executor
foreman   2991  0.9  5.9 1791488 722336 ?      Sl   Jan02  41:25 dynflow_executor
foreman   2991  0.9  5.9 1791488 722308 ?      Sl   Jan02  41:32 dynflow_executor
foreman   2991  0.9  5.9 1791488 722344 ?      Sl   Jan02  41:45 dynflow_executor
foreman   2991  0.9  5.9 1791488 722344 ?      Sl   Jan02  41:54 dynflow_executor
foreman   2991  0.9  5.9 1791488 722348 ?      Sl   Jan02  42:05 dynflow_executor
foreman   2991  0.9  5.9 1791488 721384 ?      Sl   Jan02  42:15 dynflow_executor
foreman   2991  0.9  5.9 1791488 721384 ?      Sl   Jan02  42:24 dynflow_executor
foreman   2991  0.9  5.9 1791488 721420 ?      Sl   Jan02  42:36 dynflow_executor
foreman   2991  0.9  5.9 1791488 722528 ?      Sl   Jan02  42:46 dynflow_executor
foreman   2991  0.9  5.9 1791488 725832 ?      Sl   Jan02  42:58 dynflow_executor
foreman   2991  0.9  5.9 1791488 726208 ?      Sl   Jan02  43:07 dynflow_executor
foreman   2991  0.9  5.9 1791488 726248 ?      Sl   Jan02  43:19 dynflow_executor
foreman   2991  0.9  5.9 1791488 726232 ?      Sl   Jan02  43:29 dynflow_executor
foreman   2991  0.9  5.9 1791488 726208 ?      Sl   Jan02  43:40 dynflow_executor
foreman   2991  0.9  5.9 1791488 726272 ?      Sl   Jan02  43:52 dynflow_executor
foreman   2991  0.9  5.9 1791488 726208 ?      Sl   Jan02  44:05 dynflow_executor
foreman   2991  0.9  5.9 1791488 731140 ?      Sl   Jan02  44:15 dynflow_executor
foreman   2991  0.9  5.9 1791488 731180 ?      Sl   Jan02  44:26 dynflow_executor
foreman   2991  0.9  5.9 1791488 731184 ?      Sl   Jan02  44:37 dynflow_executor
foreman   2991  0.9  5.9 1791488 731144 ?      Sl   Jan02  44:47 dynflow_executor
foreman   2991  0.9  5.9 1791488 731592 ?      Sl   Jan02  44:57 dynflow_executor
foreman   2991  0.9  5.9 1791488 731548 ?      Sl   Jan02  45:11 dynflow_executor
foreman   2991  0.9  5.9 1791488 731556 ?      Sl   Jan02  45:20 dynflow_executor
foreman   2991  0.9  5.9 1788656 723564 ?      Sl   Jan02  45:33 dynflow_executor
foreman   2991  0.9  5.9 1788656 723564 ?      Sl   Jan02  45:43 dynflow_executor
foreman   2991  0.9  5.9 1788656 725676 ?      Sl   Jan02  45:53 dynflow_executor
foreman   2991  1.0  5.9 1788656 725676 ?      Sl   Jan02  46:04 dynflow_executor
foreman   2991  1.0  5.9 1788656 725680 ?      Sl   Jan02  46:15 dynflow_executor
foreman   2991  1.0  5.9 1788656 725708 ?      Sl   Jan02  46:25 dynflow_executor
foreman   2991  1.0  5.9 1788656 725796 ?      Sl   Jan02  46:37 dynflow_executor
foreman   2991  1.0  5.9 1788656 729160 ?      Sl   Jan02  46:48 dynflow_executor
foreman   2991  1.0  6.0 1788656 733084 ?      Sl   Jan02  46:58 dynflow_executor
foreman   2991  1.0  6.0 1788656 738204 ?      Sl   Jan02  47:09 dynflow_executor
foreman   2991  1.0  6.0 1788656 738176 ?      Sl   Jan02  47:22 dynflow_executor
foreman   2991  1.0  6.0 1788656 738168 ?      Sl   Jan02  47:32 dynflow_executor
foreman   2991  1.0  6.0 1788656 738176 ?      Sl   Jan02  47:43 dynflow_executor
foreman   2991  1.0  6.0 1788656 738204 ?      Sl   Jan02  47:55 dynflow_executor
foreman   2991  1.0  6.0 1788656 738208 ?      Sl   Jan02  48:06 dynflow_executor
foreman   2991  1.0  6.0 1788656 738204 ?      Sl   Jan02  48:18 dynflow_executor
foreman   2991  1.0  6.0 1788656 740724 ?      Sl   Jan02  48:29 dynflow_executor
foreman   2991  1.0  6.0 1788656 740716 ?      Sl   Jan02  48:39 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  48:51 dynflow_executor
foreman   2991  1.0  6.0 1788656 740752 ?      Sl   Jan02  49:00 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  49:11 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  49:22 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  49:31 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  49:40 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  49:53 dynflow_executor
foreman   2991  1.0  6.0 1788656 740820 ?      Sl   Jan02  50:03 dynflow_executor
foreman   2991  1.0  6.0 1788656 740820 ?      Sl   Jan02  50:13 dynflow_executor
foreman   2991  1.0  6.0 1788656 740776 ?      Sl   Jan02  50:26 dynflow_executor
foreman   2991  1.0  6.0 1788656 740784 ?      Sl   Jan02  50:38 dynflow_executor
foreman   2991  1.0  6.0 1788656 741744 ?      Sl   Jan02  50:47 dynflow_executor
foreman   2991  1.0  6.0 1789236 741984 ?      Sl   Jan02  51:00 dynflow_executor
foreman   2991  1.0  6.0 1789236 733240 ?      Sl   Jan02  51:12 dynflow_executor
foreman   2991  1.0  6.0 1789236 732208 ?      Sl   Jan02  51:22 dynflow_executor
foreman   2991  1.0  6.0 1789236 732376 ?      Sl   Jan02  51:33 dynflow_executor
foreman   2991  1.0  6.0 1789236 732340 ?      Sl   Jan02  51:44 dynflow_executor
foreman   2991  1.0  6.0 1789236 732372 ?      Sl   Jan02  51:52 dynflow_executor
foreman   2991  1.0  6.0 1789236 732336 ?      Sl   Jan02  52:05 dynflow_executor
foreman   2991  1.0  6.0 1789236 732324 ?      Sl   Jan02  52:14 dynflow_executor
foreman   2991  1.0  6.0 1789236 732332 ?      Sl   Jan02  52:26 dynflow_executor
foreman   2991  1.1  6.0 1789236 736244 ?      Sl   Jan02  52:38 dynflow_executor
foreman   2991  1.1  6.0 1789236 740908 ?      Sl   Jan02  52:49 dynflow_executor
foreman   2991  1.1  6.1 1789236 744500 ?      Sl   Jan02  52:58 dynflow_executor
foreman   2991  1.1  6.0 1789236 743532 ?      Sl   Jan02  53:09 dynflow_executor
foreman   2991  1.1  6.0 1789236 743796 ?      Sl   Jan02  53:21 dynflow_executor
foreman   2991  1.1  6.0 1789236 743796 ?      Sl   Jan02  53:32 dynflow_executor
foreman   2991  1.1  6.0 1789236 743796 ?      Sl   Jan02  53:44 dynflow_executor
foreman   2991  1.1  6.0 1789236 743832 ?      Sl   Jan02  53:56 dynflow_executor
foreman   2991  1.1  6.0 1789236 740844 ?      Sl   Jan02  54:05 dynflow_executor
foreman   2991  1.1  6.0 1789236 740888 ?      Sl   Jan02  54:16 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  54:28 dynflow_executor
foreman   2991  1.1  6.0 1789236 742824 ?      Sl   Jan02  54:38 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  54:49 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  55:01 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  55:13 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  55:26 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  55:38 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  55:50 dynflow_executor
foreman   2991  1.1  6.0 1789236 742764 ?      Sl   Jan02  55:59 dynflow_executor
foreman   2991  1.1  6.0 1789236 742800 ?      Sl   Jan02  56:12 dynflow_executor
foreman   2991  1.1  6.0 1789236 743760 ?      Sl   Jan02  56:23 dynflow_executor
foreman   2991  1.1  6.0 1789236 743760 ?      Sl   Jan02  56:35 dynflow_executor
foreman   2991  1.1  6.0 1789236 743764 ?      Sl   Jan02  56:45 dynflow_executor
foreman   2991  1.1  6.0 1789236 740356 ?      Sl   Jan02  56:57 dynflow_executor
foreman   2991  1.1  6.0 1789236 740388 ?      Sl   Jan02  57:06 dynflow_executor
foreman   2991  1.1  6.0 1789236 740352 ?      Sl   Jan02  57:17 dynflow_executor
foreman   2991  1.1  6.0 1789236 740388 ?      Sl   Jan02  57:29 dynflow_executor
foreman   2991  1.1  6.1 1789236 747164 ?      Sl   Jan02  57:42 dynflow_executor
foreman   2991  1.1  6.1 1789236 747172 ?      Sl   Jan02  57:51 dynflow_executor
foreman   2991  1.1  6.1 1789236 747208 ?      Sl   Jan02  58:03 dynflow_executor
foreman   2991  1.1  6.1 1789236 747168 ?      Sl   Jan02  58:12 dynflow_executor
foreman   2991  1.1  6.1 1789236 751148 ?      Sl   Jan02  58:24 dynflow_executor
foreman   2991  1.1  6.1 1789236 750184 ?      Sl   Jan02  58:34 dynflow_executor
foreman   2991  1.1  6.1 1789236 750184 ?      Sl   Jan02  58:46 dynflow_executor
foreman   2991  1.1  6.1 1789236 750208 ?      Sl   Jan02  58:57 dynflow_executor
foreman   2991  1.1  6.1 1789236 750148 ?      Sl   Jan02  59:08 dynflow_executor
foreman   2991  1.1  6.1 1789236 750128 ?      Sl   Jan02  59:17 dynflow_executor
foreman   2991  1.1  6.1 1789236 750236 ?      Sl   Jan02  59:29 dynflow_executor
foreman   2991  1.2  6.1 1789236 750164 ?      Sl   Jan02  59:41 dynflow_executor
foreman   2991  1.2  6.1 1789236 750092 ?      Sl   Jan02  59:49 dynflow_executor
foreman   2991  1.2  6.1 1789236 751456 ?      Sl   Jan02  59:59 dynflow_executor
foreman   2991  1.2  6.1 1789236 751560 ?      Sl   Jan02  60:10 dynflow_executor
foreman   2991  1.2  6.1 1789236 752608 ?      Sl   Jan02  60:21 dynflow_executor
foreman   2991  1.2  6.1 1789236 752628 ?      Sl   Jan02  60:32 dynflow_executor
foreman   2991  1.2  6.1 1789236 751608 ?      Sl   Jan02  60:43 dynflow_executor
foreman   2991  1.2  6.1 1789236 751684 ?      Sl   Jan02  60:55 dynflow_executor
foreman   2991  1.2  6.1 1789236 751612 ?      Sl   Jan02  61:04 dynflow_executor
foreman   2991  1.2  6.1 1789236 751608 ?      Sl   Jan02  61:16 dynflow_executor
foreman   2991  1.2  6.1 1789236 751648 ?      Sl   Jan02  61:29 dynflow_executor
foreman   2991  1.2  6.1 1789236 751500 ?      Sl   Jan02  61:34 dynflow_executor
foreman   2991  1.2  6.1 1789236 751320 ?      Sl   Jan02  61:35 dynflow_executor
foreman   2991  1.2  6.1 1789236 751284 ?      Sl   Jan02  61:35 dynflow_executor
foreman   2991  1.2  6.1 1789236 752280 ?      Sl   Jan02  61:36 dynflow_executor
#


Expected results:
No such memory growth.


Additional info:

Comment 1 Ivan Necas 2017-01-06 14:00:55 UTC

There were some bugs fixed in 6.2.6 that potentially might influence this behaviour https://bugzilla.redhat.com/show_bug.cgi?id=1391704, https://bugzilla.redhat.com/show_bug.cgi?id=1398438 and https://bugzilla.redhat.com/show_bug.cgi?id=1399294

They all have been fixed in 6.2.6. Could you test this with 6.2.6.

Also, I think this comment applies here as well https://bugzilla.redhat.com/show_bug.cgi?id=1362168#c20. I the small-growths, that might be just caused by the inefficiencies of Ruby memmory defragrmentation, we don't have much possibilities other than setting the top limit where we preffer restart the process to start from scratch.

Comment 2 Pavel Moravec 2017-01-10 09:49:57 UTC

(In reply to Ivan Necas from comment #1)
> There were some bugs fixed in 6.2.6 that potentially might influence this
> behaviour https://bugzilla.redhat.com/show_bug.cgi?id=1391704,
> https://bugzilla.redhat.com/show_bug.cgi?id=1398438 and
> https://bugzilla.redhat.com/show_bug.cgi?id=1399294
> 
> They all have been fixed in 6.2.6. Could you test this with 6.2.6.

Longer test with 6.2.6, memory stabilized after many hours/capsule synces on some fairly low value. Therefore this is fixed in 6.2.6, thanks!

Closing this BZ as resolved by:

https://access.redhat.com/errata/RHBA-2016:2958

Note You need to log in before you can comment on or make changes to this bug.